kafak管理常用的工具是kafka-manager和kakfa-eagle.这两个工具在分区管理上都是只管加不管减,副本调整的话,也不支持,只能自己手工操作.本文的操作也是去年双十一的沉淀,没想到今年又用了一次,索性记录下来.
修改
这里需要一个json文件,示例如下
bash-4.3# cat increase-replication-factor.json
{
"version":1,
"partitions":[
{
"topic":"test-topic",
"partition":0,
"replicas":[
1
]
},
{
"topic":"test-topic",
"partition":1,
"replicas":[
2
]
},
{
"topic":"test-topic",
"partition":2,
"replicas":[
3
]
}
]
}需要增加分区,就增加下面的块
{
"topic":"test-topic",
"partition":4,
"replicas":[
3
]
}其中partition要不重复,replicas要根据节点数循环.
需要增加副本,就修改
"replicas":[
3,2
]两个副本就有两位,三个副本就有三位.
执行修改
会返回当前的配置,记得保存一下,万一搞错了,恢复时用
bash-4.3# kafka-reassign-partitions.sh \
--zookeeper 192.168.0.1:2181 \
--reassignment-json-file increase-test-topic.json \
--execute
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test-topic","partition":0,"replicas":[3],"log_dirs":["any"]},{"topic":"test-topic","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"test-topic","partition":1,"replicas":[1],"log_dirs":["any"]}]}
Save this to use as the --reassignment-json-file option during rollback验证修改
需要清楚,无论分区的增加还是副本的调整都有一个过程,会有数据的移动,再进行后面的操作前要等上一个操作完成
bash-4.3# kafka-reassign-partitions.sh \
--zookeeper 192.168.0.1:2181 \
--reassignment-json-file increase-test-topic.json \
--verify
Status of partition reassignment:
Reassignment of partition test-topic-0 completed successfully
Reassignment of partition test-topic-1 completed successfully
Reassignment of partition test-topic-2 completed successfully错误一则
Partitions reassignment failed due to Partition reassignment data file is empty
kafka.common.AdminCommandFailedException: Partition reassignment data file is empty
at kafka.admin.ReassignPartitionsCommand$.parseAndValidate(ReassignPartitionsCommand.scala:266)
at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:194)
at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:190)
at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61)
at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)Partitions reassignment failed due to Partition reassignment data file is empty
kafka.common.AdminCommandFailedException: Partition reassignment data file is empty
at kafka.admin.ReassignPartitionsCommand$.parseAndValidate(ReassignPartitionsCommand.scala:266)
at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:194)
at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:190)
at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61)
at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)解决:
一般是json文件格式不对,少逗号或多逗号
{
"version":1,
"partitions":[
{"topic":"test_topic","partition":0,"replicas":[1,2,3]},
{"topic":"test_topic","partition":1,"replicas":[1,2,3]},
{"topic":"test_topic","partition":2,"replicas":[1,2,3]}
{"topic":"test_topic","partition":3,"replicas":[1,2,3]}
]
}弄了一个微信群欢迎正常学习elk的朋友扫码交流.
转载请注明: 转自Rainbird的个人博客 本文链接: ELK|kafka增加分区或调整副本数
