2
0
0
Kafka 日志分析报告20260721
2026-07-21
文章摘要
|
Kafka 日志分析报告
背景
运行了一周的kafka,突然没有日志了,于是看到这些错误:
kafka | at java.base/java.lang.Thread.run(Unknown Source)
kafka | [2026-07-20 10:36:16,017] WARN [SocketServer listenerType=ZK_BROKER, nodeId=1001] Unexpected error from /45.33.12.122; closing connection (org.apache.kafka.common.network.Selector)
kafka | org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 302055471 larger than 104857600)
kafka | at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
kafka | at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
kafka | at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
kafka | at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
kafka | at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
kafka | at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
kafka | at kafka.network.Processor.poll(SocketServer.scala:989)
kafka | at kafka.network.Processor.run(SocketServer.scala:892)
kafka | at java.base/java.lang.Thread.run(Unknown Source)
kafka | [2026-07-20 10:36:21,281] ERROR Closing socket for 172.22.6.12:9092-45.33.12.122:55361-6659 because of error (kafka.network.Processor)
kafka | org.apache.kafka.common.errors.InvalidRequestException: Error parsing request header. Our best guess of the apiKey is: -173
kafka | Caused by: java.lang.IllegalArgumentException: Unexpected api key: -173
kafka | at org.apache.kafka.common.protocol.ApiKeys.forId(ApiKeys.java:183)
kafka | at org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:93)
kafka | at kafka.network.Processor.parseRequestHeader(SocketServer.scala:999)
kafka | at kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:1012)
kafka | at java.base/java.util.LinkedHashMap$LinkedValues.forEach(Unknown Source)
kafka | at kafka.network.Processor.processCompletedReceives(SocketServer.scala:1008)
kafka | at kafka.network.Processor.run(SocketServer.scala:893)
kafka | at java.base/java.lang.Thread.run(Unknown Source)
kafka | [2026-07-20 10:36:21,281] ERROR Exception while processing request from 172.22.6.12:9092-45.33.12.122:55361-6659 (kafka.network.Processor)
kafka | org.apache.kafka.common.errors.InvalidRequestException: Error parsing request header. Our best guess of the apiKey is: -173
kafka | Caused by: java.lang.IllegalArgumentException: Unexpected api key: -173
kafka | at org.apache.kafka.common.protocol.ApiKeys.forId(ApiKeys.java:183)
kafka | at org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:93)
kafka | at kafka.network.Processor.parseRequestHeader(SocketServer.scala:999)
kafka | at kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:1012)
kafka | at java.base/java.util.LinkedHashMap$LinkedValues.forEach(Unknown Source)
kafka | at kafka.network.Processor.processCompletedReceives(SocketServer.scala:1008)
kafka | at kafka.network.Processor.run(SocketServer.scala:893)
kafka | at java.base/java.lang.Thread.run(Unknown Source)
kafka | [2026-07-20 10:37:14,260] WARN [SocketServer listenerType=ZK_BROKER, nodeId=1001] Unexpected error from /45.33.12.122; closing connection (org.apache.kafka.common.network.Selector)
kafka | org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1937006964 larger than 104857600)
kafka | at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
kafka | at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
kafka | at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
kafka | at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
kafka | at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
kafka | at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
kafka | at kafka.network.Processor.poll(SocketServer.scala:989)
kafka | at kafka.network.Processor.run(SocketServer.scala:892)
kafka | at java.base/java.lang.Thread.run(Unknown Source)
kafka | [2026-07-20 10:37:15,096] WARN [SocketServer listenerType=ZK_BROKER, nodeId=1001] Unexpected error from /45.33.12.122; closing connection (org.apache.kafka.common.network.Selector)
kafka | org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1330664521 larger than 104857600)
kafka | at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
kafka | at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
kafka | at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
kafka | at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
kafka | at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
kafka | at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
kafka | at kafka.network.Processor.poll(SocketServer.scala:989)
kafka | at kafka.network.Processor.run(SocketServer.scala:892)
kafka | at java.base/java.lang.Thread.run(Unknown Source)
kafka | [2026-07-20 17:03:19,962] INFO [ProducerStateManager partition=LOG_APP-0] Writing producer snapshot at offset 16732667 (kafka.log.ProducerStateManager)
kafka | [2026-07-20 17:03:19,965] INFO [Log partition=LOG_APP-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Rolled new log segment at offset 16732667 in 4 ms. (kafka.log.Log)
kafka | [2026-07-20 17:03:37,795] INFO [ProducerStateManager partition=LOG_INF-0] Writing producer snapshot at offset 68253900 (kafka.log.ProducerStateManager)
kafka | [2026-07-20 17:03:37,800] INFO [Log partition=LOG_INF-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Rolled new log segment at offset 68253900 in 6 ms. (kafka.log.Log)
kafka | [2026-07-20 17:08:35,091] INFO [Log partition=LOG_INF-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Deleting segment LogSegment(baseOffset=56231686, size=1073502484, lastModifiedTime=1783962500085, largestRecordTimestamp=Some(1783962500071)) due to retention time 604800000ms breach based on the largest record timestamp in the segment (kafka.log.Log)
kafka | [2026-07-20 17:08:35,091] INFO [Log partition=LOG_INF-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Incremented log start offset to 57563616 due to segment deletion (kafka.log.Log)
kafka | [2026-07-20 17:09:35,092] INFO [Log partition=LOG_INF-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Deleting segment files LogSegment(baseOffset=56231686, size=1073502484, lastModifiedTime=1783962500085, largestRecordTimestamp=Some(1783962500071)) (kafka.log.Log)
kafka | [2026-07-20 17:09:35,136] INFO Deleted log /kafka/kafka-logs-5cb6ac8d09b8/LOG_INF-0/00000000000056231686.log.deleted. (kafka.log.LogSegment)
kafka | [2026-07-20 17:09:35,137] INFO Deleted offset index /kafka/kafka-logs-5cb6ac8d09b8/LOG_INF-0/00000000000056231686.index.deleted. (kafka.log.LogSegment)
kafka | [2026-07-20 17:09:35,137] INFO Deleted time index /kafka/kafka-logs-5cb6ac8d09b8/LOG_INF-0/00000000000056231686.timeindex.deleted. (kafka.log.LogSegment)
kafka | [2026-07-20 17:18:35,091] INFO [Log partition=LOG_APP-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Deleting segment LogSegment(baseOffset=13328159, size=1073733722, lastModifiedTime=1783962834936, largestRecordTimestamp=Some(1783962834913)) due to retention time 604800000ms breach based on the largest record timestamp in the segment (kafka.log.Log)
kafka | [2026-07-20 17:18:35,091] INFO [Log partition=LOG_APP-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Incremented log start offset to 13825650 due to segment deletion (kafka.log.Log)
kafka | [2026-07-20 17:19:35,091] INFO [Log partition=LOG_APP-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Deleting segment files LogSegment(baseOffset=13328159, size=1073733722, lastModifiedTime=1783962834936, largestRecordTimestamp=Some(1783962834913)) (kafka.log.Log)
kafka | [2026-07-20 17:19:35,124] INFO Deleted log /kafka/kafka-logs-5cb6ac8d09b8/LOG_APP-0/00000000000013328159.log.deleted. (kafka.log.LogSegment)
kafka | [2026-07-20 17:19:35,125] INFO Deleted offset index /kafka/kafka-logs-5cb6ac8d09b8/LOG_APP-0/00000000000013328159.index.deleted. (kafka.log.LogSegment)
kafka | [2026-07-20 17:19:35,125] INFO Deleted time index /kafka/kafka-logs-5cb6ac8d09b8/LOG_APP-0/00000000000013328159.timeindex.deleted. (kafka.log.LogSegment)
kafka | [2026-07-20 19:15:06,619] WARN [SocketServer listenerType=ZK_BROKER, nodeId=1001] Unexpected error from /205.210.31.73; closing connection (org.apache.kafka.common.network.Selector)
kafka | org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)
kafka | at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
kafka | at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
kafka | at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
kafka | at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
kafka | at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
kafka | at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
kafka | at kafka.network.Processor.poll(SocketServer.scala:989)
kafka | at kafka.network.Processor.run(SocketServer.scala:892)
kafka | at java.base/java.lang.Thread.run(Unknown Source)
kafka | [2026-07-21 07:33:35,091] INFO [Log partition=LOG_INF-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Deleting segment LogSegment(baseOffset=57563616, size=1073656676, lastModifiedTime=1784014379578, largestRecordTimestamp=Some(1784014378578)) due to retention time 604800000ms breach based on the largest record timestamp in the segment (kafka.log.Log)
kafka | [2026-07-21 07:33:35,092] INFO [Log partition=LOG_INF-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Incremented log start offset to 59201750 due to segment deletion (kafka.log.Log)
kafka | [2026-07-21 07:34:35,092] INFO [Log partition=LOG_INF-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Deleting segment files LogSegment(baseOffset=57563616, size=1073656676, lastModifiedTime=1784014379578, largestRecordTimestamp=Some(1784014378578)) (kafka.log.Log)
kafka | [2026-07-21 07:34:35,141] INFO Deleted log /kafka/kafka-logs-5cb6ac8d09b8/LOG_INF-0/00000000000057563616.log.deleted. (kafka.log.LogSegment)
kafka | [2026-07-21 07:34:35,141] INFO Deleted offset index /kafka/kafka-logs-5cb6ac8d09b8/LOG_INF-0/00000000000057563616.index.deleted. (kafka.log.LogSegment)
kafka | [2026-07-21 07:34:35,141] INFO Deleted time index /kafka/kafka-logs-5cb6ac8d09b8/LOG_INF-0/00000000000057563616.timeindex.deleted. (kafka.log.LogSegment)
kafka | [2026-07-21 08:15:11,592] WARN [SocketServer listenerType=ZK_BROKER, nodeId=1001] Unexpected error from /89.37.172.140; closing connection (org.apache.kafka.common.network.Selector)
kafka | org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 218762506 larger than 104857600)
kafka | at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
kafka | at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
kafka | at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
kafka | at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
kafka | at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
kafka | at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
kafka | at kafka.network.Processor.poll(SocketServer.scala:989)
kafka | at kafka.network.Processor.run(SocketServer.scala:892)
kafka | at java.base/java.lang.Thread.run(Unknown Source)
一、核心问题:外部恶意扫描/探测攻击 ⚠️
你的 Kafka 端口(9092)暴露在了公网上,正在遭受来自外部 IP 的扫描探测。以下是涉及的可疑 IP:
| IP 地址 | 时间 | 错误类型 | 说明 |
|---|---|---|---|
45.33.12.122 | 07-20 10:36 | InvalidReceiveException + InvalidRequestException | 多次尝试,发送畸形数据 |
205.210.31.73 | 07-20 19:15 | InvalidReceiveException | 单次探测 |
89.37.172.140 | 07-21 08:15 | InvalidReceiveException | 单次探测 |
具体分析
1. InvalidReceiveException: Invalid receive (size = XXX larger than 104857600)
攻击者向 Kafka 端口发送了非 Kafka 协议的数据(如 HTTP 请求、SSH 探测等)。Kafka 将收到的前 4 字节解读为消息长度,得到荒谬的超大值(如 1937006964、302055471),远超 socket.request.max.bytes(默认 100MB),因此主动断开连接。
2. InvalidRequestException: Unexpected api key: -173
45.33.12.122 发来的数据被 Kafka 勉强尝试解析,但 API key 为 -173(非法值),说明这根本不是 Kafka 协议请求,而是攻击者在尝试识别服务类型(端口扫描指纹识别)。
🚨 风险等级:高
- 你的 Kafka Broker(nodeId=1001)直接暴露在公网
- 虽然 Kafka 自身的协议校验阻止了这些请求,但这会:
- 消耗网络/线程资源
- 日志被大量垃圾信息污染
- 一旦存在已知漏洞(如旧版本 CVE),可能被攻破
🔧 建议措施
# 1. 最重要:不要将 Kafka 端口暴露到公网!
# 使用防火墙限制 9092 端口只允许内网 IP 访问
# iptables 示例(仅允许内网)
iptables -A INPUT -p tcp --dport 9092 -s 172.22.0.0/16 -j ACCEPT
iptables -A INPUT -p tcp --dport 9092 -j DROP
# 2. 如果使用 Docker,绑定端口到内网接口
# 修改 docker-compose.yml 中的端口映射
ports:
- "172.22.6.12:9092:9092" # 只绑定内网 IP,而不是 0.0.0.0
# 3. 如果必须公网访问,启用 SASL/SSL 认证
# 在 server.properties 中配置:
# listeners=SASL_SSL://0.0.0.0:9093
# sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256
二、正常操作:日志清理(无需关注)✅
以下日志是 Kafka 正常的日志保留策略执行,retention.ms=604800000(7 天):
| Topic | 操作 | 详情 |
|---|---|---|
LOG_INF-0 | 滚动新段 | offset 68253900 |
LOG_INF-0 | 删除过期段 | baseOffset=56231686, 57563616 |
LOG_APP-0 | 滚动新段 | offset 16732667 |
LOG_APP-0 | 删除过期段 | baseOffset=13328159 |
这些是正常的日志生命周期管理,无需处理。
三、总结
| 项目 | 状态 | 优先级 |
|---|---|---|
| 公网扫描/探测 | 🚨 需立即处理 | P0 |
| 日志清理 | ✅ 正常 | 无需操作 |
最关键的行动:立即通过防火墙或 Docker 端口绑定,将 Kafka 9092 端口限制为仅内网可访问。如果你使用的是云服务商,优先配置安全组规则。
Kafka 日志分析报告20260721
/archives/kafka-ri-zhi-fen-xi-bao-gao-20260721