ELK|kafka增加分区或调整副本数

Sommaire de cet article :
  1. 修改
  2. 错误一则


kafak管理常用的工具是kafka-manager和kakfa-eagle.这两个工具在分区管理上都是只管加不管减,副本调整的话,也不支持,只能自己手工操作.本文的操作也是去年双十一的沉淀,没想到今年又用了一次,索性记录下来.

修改

这里需要一个json文件,示例如下

bash-4.3# cat increase-replication-factor.json 
{
    "version":1,
    "partitions":[
        {
            "topic":"test-topic",
            "partition":0,
            "replicas":[
                1
            ]
        },
        {
            "topic":"test-topic",
            "partition":1,
            "replicas":[
                2
            ]
        },
        {
            "topic":"test-topic",
            "partition":2,
            "replicas":[
                3
            ]
        }
    ]
}

需要增加分区,就增加下面的块

{
    "topic":"test-topic",
    "partition":4,
    "replicas":[
        3
    ]
}

其中partition要不重复,replicas要根据节点数循环.

需要增加副本,就修改

"replicas":[
    3,2
]

两个副本就有两位,三个副本就有三位.

执行修改
会返回当前的配置,记得保存一下,万一搞错了,恢复时用

bash-4.3# kafka-reassign-partitions.sh \
--zookeeper 192.168.0.1:2181 \
--reassignment-json-file increase-test-topic.json \
--execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"test-topic","partition":0,"replicas":[3],"log_dirs":["any"]},{"topic":"test-topic","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"test-topic","partition":1,"replicas":[1],"log_dirs":["any"]}]}

Save this to use as the --reassignment-json-file option during rollback

验证修改
需要清楚,无论分区的增加还是副本的调整都有一个过程,会有数据的移动,再进行后面的操作前要等上一个操作完成

bash-4.3# kafka-reassign-partitions.sh \
--zookeeper 192.168.0.1:2181 \
--reassignment-json-file increase-test-topic.json \
--verify

Status of partition reassignment: 
Reassignment of partition test-topic-0 completed successfully
Reassignment of partition test-topic-1 completed successfully
Reassignment of partition test-topic-2 completed successfully

错误一则

Partitions reassignment failed due to Partition reassignment data file is empty
kafka.common.AdminCommandFailedException: Partition reassignment data file is empty
        at kafka.admin.ReassignPartitionsCommand$.parseAndValidate(ReassignPartitionsCommand.scala:266)
        at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:194)
        at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:190)
        at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61)
        at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)Partitions reassignment failed due to Partition reassignment data file is empty
kafka.common.AdminCommandFailedException: Partition reassignment data file is empty
        at kafka.admin.ReassignPartitionsCommand$.parseAndValidate(ReassignPartitionsCommand.scala:266)
        at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:194)
        at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:190)
        at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61)
        at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)

解决:

一般是json文件格式不对,少逗号或多逗号

{
    "version":1,
    "partitions":[
        {"topic":"test_topic","partition":0,"replicas":[1,2,3]},
        {"topic":"test_topic","partition":1,"replicas":[1,2,3]},
        {"topic":"test_topic","partition":2,"replicas":[1,2,3]}
        {"topic":"test_topic","partition":3,"replicas":[1,2,3]}
    ]
}

弄了一个微信群欢迎正常学习elk的朋友扫码交流.
elk


相关博文

About rainbird

IOS攻城狮
This entry was posted in ELK and tagged , , , , . Bookmark the permalink.

发表评论