Kafka学习笔记之kafka常见报错及解决⽅法（topic类、⽣产消费类、启动类）0x01 启动报错
1.1 第⼀种错误
2017-02-17 17:25:29,224] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafkamon.KafkaException: Failed to acquire lock on file .lock in /var/log/kafka-logs. A Kafka instance in another process or thread is using this directory.
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:100)
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:97)
llection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
llection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
llection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
llection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
llection.TraversableLike$class.map(TraversableLike.scala:234)
llection.AbstractTraversable.map(Traversable.scala:104)
at kafka.log.LogManager.lockLogDirs(LogManager.scala:97)
at kafka.log.LogManager.<init>(LogManager.scala:59)
at kafka.ateLogManager(KafkaServer.scala:609)
at kafka.server.KafkaServer.startup(KafkaServer.scala:183)
ics.SupportedServerStartable.startup(SupportedServerStartable.java:100)
ics.SupportedKafka.main(SupportedKafka.java:49)
解决⽅法：Failed to acquire lock on file .lock in /var/log/kafka-logs.--问题原因是有其他的进程在使⽤kafka，ps -ef|grep kafka，杀掉使⽤该⽬录的进程即可；
1.2 第⼆种错误：对index⽂件⽆权限
把⽂件的权限更改为正确的⽤户名和⽤户组即可；
⽬录/var/log/kafka-logs/，其中__consumer_offsets-29是偏移量；
1.3 第三种⽣产消费报错:jaas连接有问题
kafka_f⽂件配置有问题
16环境上
/opt/dataload/filesource_wangjuan/conf下kafka_f
KafkaClient {
com.sun.dule.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/home/client/keytabs/client.keytab"
serviceName="kafka"
principal="client/dcp@DCP.COM";
};
0x02 ⽣产者报错
2.1 第⼀种：⽣产者向topic发送消息失败
[2017-03-09 09:16:00,982] [ERROR] [startJob_Worker-10] [DCPKafkaProducer.java line:62] produceR向topicdf02211发送信息出现异常
org.apache.kafkamon.KafkaException: Failed to construct kafka producer
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:335)
原因是配置⽂件：kafka_f中配置有问题，keyTab的路径不对，导致的；
2.2 第⼆种：⽣产消费报错： Failed to construct kafka producer
报错关键信息：Failed to construct kafka producer
解决⽅法：配置⽂件问题：KafkaClient中serviceName应该是kafka，之前配置成了zookeeper；重启后，就好了；
配置⽂件如下：
KafkaServer {
com.sun.dule.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
serviceName=kafka
keyTab="/etc/security/keytabs/kafka.service.keytab"
principal="kafka/dcp16@DCP.COM";
};
KafkaClient {
com.sun.dule.Krb5LoginModule required
useKeyTab=true
storeKey=true
serviceName=kafka
keyTab="/etc/security/keytabs/kafka.service.keytab"
principal="kafka/dcp16@DCP.COM";
};
Client {
com.sun.dule.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
serviceName=zookeeper
keyTab="/etc/security/keytabs/kafka.service.keytab"
principal="kafka/dcp16@DCP.COM";
};
问题描述：
[kafka@DCP16 bin]$ ./kafka-console-producer --broker-list DCP16:9092 --topic topicin050511 --fig ../etc/kafka/producer.properties
org.apache.kafkamon.KafkaException: Failed to construct kafka producer
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:335)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:188)
at kafka.producer.NewShinyProducer.<init>(BaseProducer.scala:40)
ls.ConsoleProducer$.main(ConsoleProducer.scala:45)
ls.ConsoleProducer.main(ConsoleProducer.scala)
Caused by: org.apache.kafkamon.KafkaException: java.lang.IllegalArgumentException: Conflicting serviceName values found in JAAS and Kafka configs value in JAAS file zookeeper, value in Kafka config kafka
at org.apache.figure(SaslChannelBuilder.java:86)
at org.apache.ate(ChannelBuilders.java:70)
at org.apache.kafka.ateChannelBuilder(ClientUtils.java:83)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:277)
... 4 more
Caused by: java.lang.IllegalArgumentException: Conflicting serviceName values found in JAAS and K
afka configs value in JAAS file zookeeper, value in Kafka config kafka
at org.apache.kafkamon.security.ServiceName(KerberosLogin.java:305)
at org.apache.kafkamon.security.figure(KerberosLogin.java:103)
at org.apache.kafkamon.security.authenticator.LoginManager.<init>(LoginManager.java:45)
at org.apache.kafkamon.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)
at org.apache.figure(SaslChannelBuilder.java:78)
... 7 more
[kafka@DCP16 bin]$ ./kafka-console-producer --broker-list DCP16:9092 --topic topicin050511 --fig ../etc/kafka/producer.properties
2.3 消费时报错: ERROR Unknown error when running consumer: (ls.ConsoleConsumer$)
[root@DCP16 bin]# ./kafka-console-consumer --zookeeper dcp18:2181,dcp16:2181,dcp19:2181/kafkakerberos --from-beginning --topic topicout050511 --new-consumer --fig ../etc/kafka/consumer.properties --bootstrap-server DCP16 [2017-05-07 22:24:37,479] ERROR Unknown error when running consumer: (ls.ConsoleConsumer$)
org.apache.kafkamon.KafkaException: Failed to construct kafka consumer
at org.apache.sumer.KafkaConsumer.<init>(KafkaConsumer.java:702)
at org.apache.sumer.KafkaConsumer.<init>(KafkaConsumer.java:587)
at org.apache.sumer.KafkaConsumer.<init>(KafkaConsumer.java:569)
sumer.NewShinyConsumer.<init>(BaseConsumer.scala:53)
ls.ConsoleConsumer$.run(ConsoleConsumer.scala:64)
ls.ConsoleConsumer$.main(ConsoleConsumer.scala:51)
ls.ConsoleConsumer.main(ConsoleConsumer.scala)
Caused by: org.apache.kafkamon.KafkaException: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user at org.apache.figure(SaslChannelBuilder.java:86)
at org.apache.ate(ChannelBuilders.java:70)
at org.apache.kafka.ateChannelBuilder(ClientUtils.java:83)
at org.apache.sumer.KafkaConsumer.<init>(KafkaConsumer.java:623)
... 6 more
Caused by: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication inform at com.sun.dule.Krb5LoginModule.promptForPass(Krb5LoginModule.java:899)
at com.sun.dule.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
at com.sun.dule.Krb5LoginModule.login(Krb5LoginModule.java:584)
flect.NativeMethodAccessorImpl.invoke0(Native Method)
flect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)kafka为什么那么快
flect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at flect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
at org.apache.kafkamon.security.authenticator.AbstractLogin.login(AbstractLogin.java:69)
at org.apache.kafkamon.security.kerberos.KerberosLogin.login(KerberosLogin.java:110)
at org.apache.kafkamon.security.authenticator.LoginManager.<init>(LoginManager.java:46)
at org.apache.kafkamon.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)
at org.apache.figure(SaslChannelBuilder.java:78)
衍⽣问题:
kafka⽣产消息就会报错：
[2017-05-07 23:17:16,240] ERROR Error when sending message to topic topicin050511 with key: null, value: 0 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.s.TimeoutException: Failed to update metadata after 60000 ms.
把KafkaClient更改为如下的配置，就可以了：
KafkaClient {
com.sun.dule.Krb5LoginModule required
useTicketCache=true;
};
0x03消费者报错
3.1 第⼀种错误：replication factor: 1 larger than available brokers: 0
消费时报错：Error while executing topic command : replication factor: 1 larger than available brokers: 0
解决办法：/confluent-3.0.0/bin 下重启daemon
./kafka-server-stop -daemon ../etc/kafka/server.properties
./kafka-server-start -daemon ../etc/kafka/server.properties
然后zk重启;sh zkCli.sh -server ai186;
/usr/hdp/2.4.2.0-258/zookeeper/bin/zkCli.sh --脚本的⽬录
如果还报错，可以查看配置⽂件中下⾯的配置：
3.2 第⼆种错误：TOPIC_AUTHORIZATION_FAILED
./bin/kafka-console-consumer --zookeeper DCP185:2181,DCP186:2181,DCP187:2181/kafka --from-beginning --topic wangjuan_topic1 --new-consumer --fig ./etc/kafka/consumer.properties --bootstrap-server DCP187:9092 [2017-03-02 13:44:38,398] WARN The tion.timeout.ms = 6000 was supplied but isn't a known config. (org.apache.sumer.ConsumerConfig)
[2017-03-02 13:44:38,575] WARN Error while fetching metadata with correlation id 1 : {wangjuan_topic1=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2017-03-02 13:44:38,677] WARN Error while fetching metadata with correlation id 2 : {wangjuan_topic1=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2017-03-02 13:44:38,780] WARN Error while fetching metadata with correlation id 3 : {wangjuan_topic1=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
解决⽅法：配置⽂件中下⾯的参数中的User的U必须是⼤写；
super.users=User:kafka
或者有可能是server.properties中的adver.listen的IP是不对的，有可能是代码中写死的IP；
3.3 第三种错误的可能的解决⽅法：
⽆法消费，则查看kafka的启动⽇志中的报错信息：⽇志⽂件的所属组不对，应该是hadoop；
或者，查看kafka对应的zookeeper的配置后缀，是否已经更改，如果更改了，则topic需要重新⽣成才⾏；
3.4 第四种错误：消费的tomcat报错
[2017-04-01 06:37:21,823] [INFO] [Thread-5] [AbstractCoordinator.java line:542] Marking the coordinator DCP187:9092 (id: 2147483647 rack: null) dead for group test-consumer-group
[2017-04-01 06:37:21,825] [WARN] [Thread-5] [ConsumerCoordinator.java line:476] Auto offset commit failed for group test-consumer-group: Commit offsets failed with retriable exception. You should retry committing offsets.
更改代码中，tomcat的⼼跳超时时间如下：
没有改之前的:;
./webapps/web/WEB-INF/classes/com/ai/bdx/dcp/hadoop/service/impl/DCPKafkaConsumer.class;
重启后，⽇志中显⽰：
[2017-04-01 10:14:56,167] [INFO] [Thread-5] [AbstractCoordinator.java line:542] Marking the coordinator DCP187:9092 (id: 2147483647 rack: null) dead for group test-consumer-group
[2017-04-01 10:14:56,286] [INFO] [Thread-5] [AbstractCoordinator.java line:505] Discovered coordinator DCP187:9092 (id: 2147483647 rack: null) for group test-consumer-group.
0x04 创建topic时错误
创建topic时报错：
[2017-04-10 10:32:23,776] WARN SASL configuration failed: javax.security.auth.login.LoginException: Checksum failed Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (ke Exception in thread "main" org.ption.ZkAuthFailedException: Authentication failure
at org.I0Itec.zkclient.ZkClient.waitForKeeperState(ZkClient.java:946)
at org.I0Itec.zkclient.ZkClient.waitUntilConnected(ZkClient.java:923)
at org.I0Itec.t(ZkClient.java:1230)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:75)
at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:54)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
问题定位：是jaas⽂件有问题：
解决⽅法：server.properties⽂件中的super.user要和jaas⽂件中的keytab的principle⼀致；
server.properties:super.users=User:client
kafka_f⽂件改为：
KafkaServer {
com.sun.dule.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
serviceName=kafka
keyTab="/data/data1/confluent-3.0.0/kafka.keytab"
principal="kafka@DCP.COM";
};
KafkaClient {
com.sun.dule.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/home/client/client.keytab"
principal="client/DCP187@DCP.COM";
};
Client {
com.sun.dule.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
serviceName=zookeeper
keyTab="/home/client/client.keytab"
principal="client/DCP187@DCP.COM";
};

版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。