20250716-es重启之后报错处理

作者：Administrator 发布时间： 2025-07-16 阅读量：6 评论数：0
es
#es

20250716-es重启之后报错处理

问题

重启docker更新了一下配置文件，发现es那边报错分片失败，查看日志如下：

elasticsearch  | {"type": "server", "timestamp": "2025-07-16T02:40:13,268Z", "level": "WARN", "component": "o.e.x.t.t.TransformIndexer", "cluster.name": "elasticsearch", "node.name": "ce5c96dde3a9", "message": "[endpoint.metadata_united-default-1.3.0] failed to detect changes for transform. Skipping update till next check.", "cluster.uuid": "WmFtbNipQY2fRKzZ53aVKw", "node.id": "BW6QmHNdSqibU2pNCTtrbQ" , 
elasticsearch  | "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: ",
elasticsearch  | "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:713) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:459) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:199) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1048) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:763) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:399) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:487) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:285) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:190) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:577) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:26) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:100) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:103) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:26) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:87) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:128) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:571) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:556) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$11(AuthorizationService.java:450) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:967) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:931) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:352) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:41) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1015) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:343) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:443) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:371) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:256) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:138) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:173) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:279) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:144) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:127) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:258) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:186) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:182) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124) [x-pack-security-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:57) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.client.ParentTaskAssigningClient.doExecute(ParentTaskAssigningClient.java:55) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:251) [x-pack-core-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:211) [x-pack-core-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.transform.checkpoint.TimeBasedCheckpointProvider.sourceHasChanged(TimeBasedCheckpointProvider.java:80) [transform-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.transform.transforms.TransformIndexer.sourceHasChanged(TransformIndexer.java:1008) [transform-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.transform.transforms.TransformIndexer.onStart(TransformIndexer.java:366) [transform-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.xpack.core.indexing.AsyncTwoPhaseIndexer.lambda$maybeTriggerAsyncJob$5(AsyncTwoPhaseIndexer.java:219) [x-pack-core-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
elasticsearch  | "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
elasticsearch  | "at java.lang.Thread.run(Thread.java:833) [?:?]",
elasticsearch  | "Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[metrics-endpoint.metadata_current_default][0]]. Consider using `allow_partial_search_results` setting to bypass this error.",
elasticsearch  | "at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:227) ~[elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454) [elasticsearch-7.17.3.jar:7.17.3]",
elasticsearch  | "... 90 more"] }

查找原因

📝 日志内容简要

日志显示的是一个 Elasticsearch 7.17.3 实例的警告信息，具体如下：

[endpoint.metadata_united-default-1.3.0] failed to detect changes for transform. Skipping update till next check.

这表示 Elasticsearch 的 transform（数据转换）任务 在尝试检测源索引中的变更时失败了。因此，它跳过了这次更新，直到下一次检查。

⚠️ 异常信息

从日志中可以看到抛出了如下异常：

org.elasticsearch.action.search.SearchPhaseExecutionException

这个异常通常表示在执行搜索请求时出现了问题，而 Elasticsearch 的 transform 任务依赖于定期对源索引进行搜索来检测变更。

结合堆栈信息来看，问题可能与权限有关，因为日志中涉及了很多 x-pack-security 模块的类，例如：

  • AuthorizationService
  • RBACEngine
  • FieldAndDocumentLevelSecurityRequestInterceptor

这表明 transform 任务使用的用户可能没有足够的权限访问源索引或执行搜索。

✅ 建议排查步骤

1. 查看 transform 的状态

使用如下命令查看 transform 的运行状态和详细信息：

GET _transform/endpoint.metadata_united-default-1.3.0/_stats
GET _transform/endpoint.metadata_united-default-1.3.0

查看返回结果中是否有错误信息（reason 或 error 字段）。

2. 检查用户权限

确认运行该 transform 的用户是否有权限访问源索引。可以使用如下命令查看角色权限：

GET /_security/role/<role-name>

确保角色包含如下权限：

"indices": [
  {
    "names": [ "源索引名称" ],
    "privileges": ["read"]
  }
]

3. 检查字段级/文档级安全设置

如果你的集群启用了 字段级安全（Field-Level Security） 或 文档级安全（Document-Level Security），请确认 transform 使用的用户能够访问所有必要的字段和文档。

4. 查看相关时间点的日志

检查 2025-07-16T02:39:33 UTC 前后是否有其他错误日志，可能有助于定位问题根源。

5. 重启或重置 transform（谨慎操作）

如果问题持续存在，可以尝试停止并重新启动 transform：

POST _transform/endpoint.metadata_united-default-1.3.0/_stop
POST _transform/endpoint.metadata_united-default-1.3.0/_start

或者重置 transform（会删除目标索引和状态）：

POST _transform/endpoint.metadata_united-default-1.3.0/_reset

⚠️ 注意：重置操作是破坏性的，请确保你了解其影响后再执行。

📌 总结

这个警告可能是由于权限问题导致 transform 无法读取源索引数据。虽然下一次检查可能会自动恢复，但如果频繁出现，建议检查以下内容：

  • 用户对源索引是否有 read 权限
  • 是否启用了字段或文档级安全策略
  • 集群节点之间的通信是否正常
  • transform 配置是否正确

处理之后就查看日志已经没有相关报错了

上一篇：英文例句第一部分
下一篇：arm架构容器无法在amd64上运行

评论