kafkaOffset相关命令总结
⼀、查询topic的offset的范围
1、查询某个topic的offset的最⼩值
[root@Hexindai-C11-71 ~]# kafka-run-class.ls.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-kafka --time -2
rabin-kafka:8:0
rabin-kafka:2:0
rabin-kafka:5:0
rabin-kafka:4:0
rabin-kafka:7:0
rabin-kafka:1:0
rabin-kafka:9:0
rabin-kafka:3:0
rabin-kafka:6:0
rabin-kafka:0:0
[root@Hexindai-C11-71 ~]# kafka-run-class.ls.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-kafka --time -2|wc -l
10
[root@Hexindai-C11-71 ~]#
[root@Hexindai-C11-71 ~]# kafka-run-class.ls.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-kafka --time -2
[root@Hexindai-C11-71 ~]# kafka-run-class.ls.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-channel --time -2
rabin-channel:2:0
rabin-channel:1:0
rabin-channel:0:0
[root@Hexindai-C11-71 ~]# kafka-run-class.ls.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-channel --time -2|wc -l
3
[root@Hexindai-C11-71 ~]#
[root@Hexindai-C11-71 ~]# kafka-run-class.ls.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-channel --time -2
　#很显然，第⼀列为topic名称，第⼆列为partition名称，第三列则是offset。为估计仔细的你应该已经发现了最⼩的offset为0，分区数为30个
2、查询某个topic的offset的最⼤值
[root@Hexindai-C11-71 ~]# kafka-run-class.ls.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-kafka --time -1
rabin-kafka:8:1
rabin-kafka:2:1
rabin-kafka:5:1
rabin-kafka:4:0
rabin-kafka:7:0
rabin-kafka:1:0
rabin-kafka:9:0
rabin-kafka:3:1
rabin-kafka:6:1
rabin-kafka:0:1
[root@Hexindai-C11-71 ~]#
[root@Hexindai-C11-71 ~]# kafka-run-class.ls.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-kafka --time -1|wc -l
10
[root@Hexindai-C11-71 ~]#
[root@Hexindai-C11-71 ~]# kafka-run-class.ls.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-kafka --time -1
[root@Hexindai-C11-71 ~]# kafka-run-class.ls.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-channel --time -1
rabin-channel:2:0
rabin-channel:1:0
rabin-channel:0:0
[root@Hexindai-C11-71 ~]# kafka-run-class.ls.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-channel --time -1|wc -l
3
[root@Hexindai-C11-71 ~]#
[root@Hexindai-C11-71 ~]# kafka-run-class.ls.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-channel --time -1
⼆、查看consumer group列表，使⽤ --list参数
  查看consumser group列表有新，旧两种命令，分别查看新版（信息保存在broker中）consumer列表和⽼版（信息保存在zookeeper中）consumer列表，因⽽需要区分指定bootstrap-server和zookeeper参数：（该博客测试的kafka版本为0.10.2.1）
[root@Hexindai-C11-71 ~]# kafka-consumer-groups.sh --zookeeper 172.20.11.74:2181/kafka --list
Note: This will only show information about consumers that use ZooKeeper (not those using the Java consumer API).
[root@Hexindai-C11-71 ~]#
##很显然，为这⾥使⽤的是kafka 0.10.2.1版本，我们是查询不到对应的信息的！
#⽼版本的consumer信息都保存在zookeeper中，很显然，0.10.2.1这个版本已经将consumer信息⾃⼰保存啦。
[root@Hexindai-C11-71 ~]# kafka-consumer-groups.sh --zookeeper 172.20.11.74:2181/kafka --list
[root@Hexindai-C11-71 ~]# kafka-consumer-groups.sh --bootstrap-server 172.20.11.74:9092 --list
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
console-consumer-61283
console-consumer-83065
console-consumer-4362
[root@Hexindai-C11-71 ~]#
#查看新版consumer列表，很显然，我们可以查询到对应的数据
[root@Hexindai-C11-71 ~]# kafka-consumer-groups.sh --bootstrap-server 172.20.11.74:9092 --list
三、查看指定group.id的消费者消费情况（kafka-consumer-groups.sh)
[root@Hexindai-C11-71 ~]# kafka-consumer-groups.sh  --new-consumer --bootstrap-server 172.20.11.74:9092 --group rabin-kafka --describe
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
Error: Consumer group 'rabin-kafka' does not exist.
没有组，需要查看是什么原因，是需要创建？
四、查询__consumer_offsets topic所有内容
1、运⾏下⾯命令前先要在consumer.properties中设置pics=false
[root@Hexindai-C11-71 ~]# cat /export/servers/kafka_2.11-0.10.2.1/config/consumer.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    /licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# sumer.ConsumerConfig for more details
# Zookeeper connection string
# comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002"
# timeout in ms for connecting to zookeeper
#consumer group id
group.id=test-consumer-group
#consumer timeout
#consumer.timeout.ms=5000
pics=false
[root@Hexindai-C11-71 ~]#
[root@Hexindai-C11-71 ~]# cat /export/servers/kafka_2.11-0.10.2.1/config/consumer.properties
2、0.11.0.0之前版本
[root@Hexindai-C11-71 ~]#  kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server 172.20.11.73:9092 --formatter "dinator.GroupMetadataManager\$OffsetsMessageFormatter" --fig /export/servers/kafk [2019-07-26 15:22:57,224] WARN The configuration 't' was supplied but isn't a known config. (org.apache.sumer.ConsumerConfig)
[2019-07-26 15:22:57,224] WARN The configuration 'tion.timeout.ms' was supplied but isn't a known config. (org.apache.sumer.ConsumerConfig)
[test-consumer-group,__consumer_offsets,22]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,30]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,8]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,21]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,4]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,27]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,7]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,9]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,46]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,25]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,35]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,41]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,33]::[OffsetMetadata[712,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,23]::[OffsetMetadata[93,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,49]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,47]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,16]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,28]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,31]::[OffsetMetadata[1244,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
kafka命令[test-consumer-group,__consumer_offsets,36]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,42]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,3]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,18]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,37]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,15]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,24]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,38]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,17]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,48]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,19]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,11]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,13]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,2]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,43]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,6]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,14]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,20]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,0]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,44]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,39]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,12]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,45]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,1]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,5]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,26]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,29]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,34]::[OffsetMetadata[93,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,10]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,32]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,40]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
.........
[root@Hexindai-C11-71 ~]# kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server 172.20.11.73:9092 --formatter
"dinator.GroupMetadataManager\$OffsetsMessageFormatter" --fig /export/servers/kafka_2.11-0.10.2.1/config/consumer.properties --from-beginning
3、0.11.0.0之后版本（含）
[root@Hexindai-C11-71 ~]# kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server 172.20.11.74:9092 --formatter "up.GroupMetadataManager\$OffsetsMessageFormatter"
--fig /export/servers/kafka_2.11-0.10.2.1/config/consumer.properties --from-beginning

版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。