前往小程序，Get更优阅读体验！
立即前往
文档建议反馈控制台
首页
学习
活动
专区
工具
TVP
最新优惠活动
文章/答案/技术大牛
发布
首页
学习
活动
专区
工具
TVP最新优惠活动
返回腾讯云官网
社区首页 >专栏 >ES6.8.2 kibana的monitoring访问报错We couldn't activate monitoring处理过程

ES6.8.2 kibana的monitoring访问报错We couldn't activate monitoring处理过程

原创
作者头像
zjiekou
发布于 2023-09-28 18:32:31
6460
发布于 2023-09-28 18:32:31
举报
文章被收录于专栏：大数据zjiekou大数据zjiekou

一、前言

某客户反馈最近kibana的监控都看不了了

ES版本：6.8.2

kibana版本：6.8.2

二、排查过程

这种问题通常就是系统索引.monitoring-es和.monitoring-kibana索引有问题，首先需要检查系统自监控参数xpack.monitoring.collection.enabled是否设置的是true，通过GET _cluster/settings?include_defaults&flat_settings查看xpack.monitoring.collection.enabled参数设置的是true没有问题

接下来排查ES日志发下日志中有报错集群索引存在多Type的问题，日志如下

代码语言：javascript
复制
[o.e.x.m.e.l.LocalExporter] [1692069051003930432] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: java.lang.IllegalArgumentException: Rejecting mapping update to [.monitoring-kibana-6-2023.09.28] as the final mapping would have more than 1 type: [_doc, doc]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:125) ~[x-pack-monitoring-6.8.2.jar:6.8.2]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) ~[?:?]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:126) [x-pack-monitoring-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:108) [x-pack-monitoring-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkRequestModifier.lambda$wrapActionListenerIfNeeded$0(TransportBulkAction.java:681) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:486) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:467) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:456) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishOnSuccess(TransportReplicationAction.java:981) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:900) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:884) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1133) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.transport.TcpTransport$1.doRun(TcpTransport.java:1015) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:1007) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:969) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53) [transport-netty4-client-6.8.2.jar:6.8.2]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.IllegalArgumentException: Rejecting mapping update to [.monitoring-kibana-6-2023.09.28] as the final mapping would have more than 1 type: [_doc, doc]
	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:479) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:419) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:354) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:513) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:482) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:235) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:179) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:161) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:87) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1062) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1040) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:435) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:376) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$14(IndexShard.java:2641) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2616) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:999) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:372) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:327) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:314) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:299) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:345) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.8.2.jar:6.8.2]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-6.8.2.jar:6.8.2]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.8.2.jar:6.8.2]
	at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:340) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.8.2.jar:6.8.2]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-6.8.2.jar:6.8.2]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.8.2.jar:6.8.2]
	at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:340) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.8.2.jar:6.8.2]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-6.8.2.jar:6.8.2]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.8.2.jar:6.8.2]
	at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:340) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.8.2.jar:6.8.2]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-6.8.2.jar:6.8.2]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.8.2.jar:6.8.2]
	at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:340) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.8.2.jar:6.8.2]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-6.8.2.jar:6.8.2]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.8.2.jar:6.8.2]
	at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:346) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:323) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:264) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:605) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:580) ~[?:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:514) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:282) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:642) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:551) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:252) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:642) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:313) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:248) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:640) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:251) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:640) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:274) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:298) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:271) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:262) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:228) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:194) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:114) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:334) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:120) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:108) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:196) ~[?:?]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:150) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:246) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:306) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:317) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:244) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139) ~[?:?]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:133) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:306) ~[?:?]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:705) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.2.jar:6.8.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]

这个错误是由于Elasticsearch 6.x版本开始，一个索引只能有一个类型，但是在你的监控索引.monitoring-kibana-6-2023.09.28中，却试图创建两个类型_doc和doc，这是不被允许的。

系统索引怎么会出现这个异常呢，肯定是客户设置了模板mapping为_doc，模板中index_patterns匹配为*导致系统索引的创建匹配到这个模板了。经过排查发现果然是用户的template有问题导致系统索引的创建出来了两个type。

系统索引创建日志：

代码语言：javascript
复制
[o.e.c.m.MetaDataCreateIndexService] [1692069051003930532] [.monitoring-kibana-6-2023.09.28] creating index, cause [auto(bulk api)], templates [temp1, scene@template, default@template, .monitoring-kibana], shards [6]/[0], mappings [_doc]

.monitoring-kibana-6-2023.09.2索引mapping如下

客户的temp1 template如下

代码语言：javascript
复制
{
  "order": 10,
  "index_patterns": [
    "*"
  ],
  "settings": {
    "index": {
      "number_of_shards": "6",
      "number_of_replicas": "0"
    }
  },
  "mappings": {
    "_doc": {}
  },
  "aliases": {}
}

三、解决方案

经过上述排查过程已确认问题，接下来的解决过程如下

1、先把temp1 template中的index_patterns调整为x，这样索引创建就不会走这个模板了

调整命令如下：

代码语言：javascript
复制
PUT _template/temp1
{
  "order": 10,
  "index_patterns": [
    "x"
  ],
  "settings": {
    "index": {
      "number_of_shards": "6",
      "number_of_replicas": "0"
    }
  },
  "mappings": {
    "_doc": {}
  },
  "aliases": {}
}

2、删除所有的.monitoring-es-6-和.monitoring-kibana-6-索引

代码语言：javascript
复制
DELETE .monitoring-es-6-*
DELETE .monitoring-kibana-6-*

3、验证结果

系统索引有数据正常写入了

查看索引mapping也正常了，没有_doc的type了

4、验证monitoring界面是否已恢复

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

kibana

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

kibana
#kibana
#monitoring
评论
登录后参与评论
0 条评论
热度
最新
登录 后参与评论
推荐阅读
LV.
文章
0
获赞
0
目录
  • 一、前言
  • 二、排查过程
    • 这种问题通常就是系统索引.monitoring-es和.monitoring-kibana索引有问题，首先需要检查系统自监控参数xpack.monitoring.collection.enabled是否设置的是true，通过GET _cluster/settings?include_defaults&flat_settings查看xpack.monitoring.collection.enabled参数设置的是true没有问题
    • 三、解决方案
      • 经过上述排查过程已确认问题，接下来的解决过程如下
      相关产品与服务
      Elasticsearch Service
      腾讯云 Elasticsearch Service（ES）是云端全托管海量数据检索分析服务，拥有高性能自研内核，集成X-Pack。ES 支持通过自治索引、存算分离、集群巡检等特性轻松管理集群，也支持免运维、自动弹性、按需使用的 Serverless 模式。使用 ES 您可以高效构建信息检索、日志分析、运维监控等服务，它独特的向量检索还可助您构建基于语义、图像的AI深度应用。
      免费体验产品介绍产品文档
      ES特惠专场，首月1折秒杀，新客首购6折起！
      领券

      腾讯云开发者

      扫码关注腾讯云开发者

      扫码关注腾讯云开发者

      领取腾讯云代金券

      热门产品

      热门推荐

      更多推荐

      Copyright ? 2013 - 2024 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有?

      深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059?深公网安备号 44030502008569

      腾讯云计算（北京）有限责任公司 京ICP证150476号 | ?京ICP备11018762号 | 京公网安备号11010802020287

      问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档

      Copyright ? 2013 - 2024 Tencent Cloud.

      All Rights Reserved. 腾讯云 版权所有

      登录 后参与评论
      http://www.vxiaotou.com