Skip to content

multiline logs being cutoff randomly #3901

New issue
New issue
Closed as not planned
Closed as not planned
multiline logs being cutoff randomly#3901
Labels
Stale
@liyuntao

Description

@liyuntao
liyuntao
opened on Aug 3, 2021

Bug Report

Describe the bug
Same input (Java stack trace) may cutoff or not randomly after multiline parser.

The demo below shows a piece of Java stack trace, which I want to match the first line(start with iso8601 date-fmt e.g. \d{4}-\d{2}) and collect every "cont" line not start with a date, into a whole single record.

To Reproduce

input text

Expected to be cut into two records.

2021-08-03 19:45:07,670 INFO  [qtp771832737-18] [] com.example.project.controller.debug.DebugController: >>>>>>>> 手动触发 fluent bit err 收集测试 >>>>>>>>
2021-08-03 19:45:07,671 WARN  [qtp771832737-18] [] org.eclipse.jetty.server.HttpChannel: /esop-trade/debug/trigger_err_log
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.NullPointerException: Cannot invoke "String.getBytes()" because "i" is null
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
	at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:920)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1617)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:121)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at com.example.project.common.auth.EsopJwtFilter.doFilterInternal(EsopJwtFilter.java:58)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1592)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1592)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1592)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1592)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1592)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:506)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:580)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1571)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1373)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:463)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1544)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1295)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.Server.handle(Server.java:562)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:399)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:656)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:391)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:319)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
	at org.eclipse.jetty.io.SocketChannelEndPoint$1.run(SocketChannelEndPoint.java:101)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:378)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:894)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1038)
	at java.base/java.lang.Thread.run(Thread.java:831)
Caused by: java.lang.NullPointerException: Cannot invoke "String.getBytes()" because "i" is null
	at com.example.project.controller.debug.DebugController.triggerErrLog(DebugController.java:67)
	at jdk.internal.reflect.GeneratedMethodAccessor129.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1064)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	... 91 more

normal logs output of fluent-bit (trace level)

8月 03 19:47:46 node11 td-agent-bit[1957]: [2021/08/03 19:47:46] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:47:46 node11 td-agent-bit[1957]: [2021/08/03 19:47:46] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:47:46 node11 td-agent-bit[1957]: [2021/08/03 19:47:46] [trace] [task 0x7f844ce37960] created (id=0)
8月 03 19:47:46 node11 td-agent-bit[1957]: [2021/08/03 19:47:46] [debug] [task] created task=0x7f844ce37960 id=0 OK
8月 03 19:47:46 node11 td-agent-bit[1957]: [0] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627991266.555125140, {"log"=>"2021-08-03 19:47:46,554 INFO  [qtp771832737-21] [] com.example.project.controller.debug.DebugController: >>>>>>>> 手动触发 fluent bit err 收集测试 >>>>>>>>
8月 03 19:47:46 node11 td-agent-bit[1957]: ", "stream"=>"stdout", "time"=>"2021-08-03T11:47:46.55512514Z"}]
8月 03 19:47:46 node11 td-agent-bit[1957]: [1] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627991266.555963477, {"log"=>"2021-08-03 19:47:46,555 WARN  [qtp771832737-21] [] org.eclipse.jetty.server.HttpChannel: /esop-trade/debug/trigger_err_log
8月 03 19:47:46 node11 td-agent-bit[1957]: org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.NullPointerException: Cannot invoke "String.getBytes()" because "i" is null
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:920)
8月 03 19:47:46 node11 td-agent-bit[1957]: at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
8月 03 19:47:46 node11 td-agent-bit[1957]: at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1617)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:121)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:47:46 node11 td-agent-bit[1957]: at com.example.project.common.auth.EsopJwtFilter.doFilterInternal(EsopJwtFilter.java:58)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1592)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1592)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1592)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1592)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1592)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:506)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:580)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1571)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1373)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:463)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1544)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1295)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.Server.handle(Server.java:562)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:399)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:656)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:391)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:319)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.io.SocketChannelEndPoint$1.run(SocketChannelEndPoint.java:101)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:378)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:894)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1038)
8月 03 19:47:46 node11 td-agent-bit[1957]: at java.base/java.lang.Thread.run(Thread.java:831)
8月 03 19:47:46 node11 td-agent-bit[1957]: Caused by: java.lang.NullPointerException: Cannot invoke "String.getBytes()" because "i" is null
8月 03 19:47:46 node11 td-agent-bit[1957]: at com.example.project.controller.debug.DebugController.triggerErrLog(DebugController.java:67)
8月 03 19:47:46 node11 td-agent-bit[1957]: at jdk.internal.reflect.GeneratedMethodAccessor129.invoke(Unknown Source)
8月 03 19:47:46 node11 td-agent-bit[1957]: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
8月 03 19:47:46 node11 td-agent-bit[1957]: at java.base/java.lang.reflect.Method.invoke(Method.java:567)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1064)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
8月 03 19:47:46 node11 td-agent-bit[1957]: at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
8月 03 19:47:46 node11 td-agent-bit[1957]: ... 91 more
8月 03 19:47:46 node11 td-agent-bit[1957]: ", "stream"=>"stdout", "time"=>"2021-08-03T11:47:46.555963477Z"}]
8月 03 19:47:46 node11 td-agent-bit[1957]: [2021/08/03 19:47:46] [trace] [engine] [task event] task_id=0 out_id=0 return=OK
8月 03 19:47:46 node11 td-agent-bit[1957]: [2021/08/03 19:47:46] [debug] [out coro] cb_destroy coro_id=10
8月 03 19:47:46 node11 td-agent-bit[1957]: [2021/08/03 19:47:46] [trace] [coro] destroy coroutine=0x7f844ce0a6a8 data=0x7f844ce0a6c0
8月 03 19:47:46 node11 td-agent-bit[1957]: [2021/08/03 19:47:46] [debug] [task] destroy task=0x7f844ce37960 (task_id=0)
8月 03 19:47:53 node11 td-agent-bit[1957]: [2021/08/03 19:47:53] [debug] [input:tail:tail.0] scanning path /data/docker/containers/5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112/*.log

one of abnormal logs output of fluent-bit(trace level)

8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [input:tail:tail.0] inode=27970462 events: IN_MODIFY
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [trace] [task 0x7f844ce37960] created (id=0)
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [task] created task=0x7f844ce37960 id=0 OK
8月 03 19:41:20 node11 td-agent-bit[1957]: [0] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.087841191, {"log"=>"2021-08-03 19:41:20,087 INFO  [qtp771832737-15] [] com.example.project.controller.debug.DebugController: >>>>>>>> 手动触发 fluent bit err 收集测试 >>>>>>>>
8月 03 19:41:20 node11 td-agent-bit[1957]: ", "stream"=>"stdout", "time"=>"2021-08-03T11:41:20.087841191Z"}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [1] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"2021-08-03 19:41:20,088 WARN  [qtp771832737-15] [] org.eclipse.jetty.server.HttpChannel: /esop-trade/debug/trigger_err_log
8月 03 19:41:20 node11 td-agent-bit[1957]: org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.NullPointerException: Cannot invoke "String.getBytes()" because "i" is null
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:920)
8月 03 19:41:20 node11 td-agent-bit[1957]: at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
8月 03 19:41:20 node11 td-agent-bit[1957]: at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
8月 03 19:41:20 node11 td-agent-bit[1957]: ", "stream"=>"stdout", "time"=>"2021-08-03T11:41:20.089463255Z"}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [2] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1617)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [3] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [4] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [5] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [6] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [7] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:121)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [8] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [9] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [10] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [11] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [12] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [13] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [14] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [15] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [16] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [17] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [18] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [19] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at com.example.project.common.auth.EsopJwtFilter.doFilterInternal(EsopJwtFilter.java:58)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [20] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [21] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [22] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [23] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [24] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [25] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [26] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [27] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [28] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [29] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [30] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [31] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [32] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [33] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [34] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [35] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [36] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [37] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [38] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [39] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [40] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1592)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [41] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [42] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [43] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [44] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1592)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [45] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [46] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [47] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [48] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1592)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [49] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [50] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [51] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [52] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1592)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [53] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [54] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [55] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [56] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1592)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [57] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:506)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [58] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [59] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:580)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [60] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [61] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1571)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [62] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [63] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1373)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [64] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [65] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:463)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [66] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1544)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [67] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [68] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1295)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [69] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [70] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [71] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.server.Server.handle(Server.java:562)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [72] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:399)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [73] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:656)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [74] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:391)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [75] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [76] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:319)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [77] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [78] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.io.SocketChannelEndPoint$1.run(SocketChannelEndPoint.java:101)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [79] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [80] kube.data.docker.containers.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112.5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112-json.log: [1627990880.089463255, {"log"=>"        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:378)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:894)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1038)
8月 03 19:41:20 node11 td-agent-bit[1957]: at java.base/java.lang.Thread.run(Thread.java:831)
8月 03 19:41:20 node11 td-agent-bit[1957]: Caused by: java.lang.NullPointerException: Cannot invoke "String.getBytes()" because "i" is null
8月 03 19:41:20 node11 td-agent-bit[1957]: at com.example.project.controller.debug.DebugController.triggerErrLog(DebugController.java:67)
8月 03 19:41:20 node11 td-agent-bit[1957]: at jdk.internal.reflect.GeneratedMethodAccessor129.invoke(Unknown Source)
8月 03 19:41:20 node11 td-agent-bit[1957]: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
8月 03 19:41:20 node11 td-agent-bit[1957]: at java.base/java.lang.reflect.Method.invoke(Method.java:567)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1064)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
8月 03 19:41:20 node11 td-agent-bit[1957]: at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
8月 03 19:41:20 node11 td-agent-bit[1957]: ... 91 more
8月 03 19:41:20 node11 td-agent-bit[1957]: "}]
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [trace] [engine] [task event] task_id=0 out_id=0 return=OK
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [out coro] cb_destroy coro_id=7
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [trace] [coro] destroy coroutine=0x7f844ce0a6a8 data=0x7f844ce0a6c0
8月 03 19:41:20 node11 td-agent-bit[1957]: [2021/08/03 19:41:20] [debug] [task] destroy task=0x7f844ce37960 (task_id=0)

Your Environment

  • Version used: 1.8.3
  • config below
[INPUT]
    name tail
    tag  kube.*
    Path              /data/docker/containers/5843e3a3b5da7e41264e090d406199cf39f6e431219fea951348969507c7d112/*.log
    Parser            docker
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     10MB
    Buffer_Chunk_Size 256K
    Buffer_Max_Size   512K
    Skip_Empty_Lines  On
    Skip_Long_Lines   On
    Refresh_Interval  10

[FILTER]
    name                  multiline
    match                 kube.*
    multiline.key_content log
    multiline.parser      multiline-regex-v1

[OUTPUT]
    name  stdout
    match *

[MULTILINE_PARSER]
        name                  multiline-regex-v1
        type                  regex
        multiline.parser      docker
        multiline.key_content log
        flush_timeout         500
        #
        # Regex rules for multiline parsing
        # ---------------------------------
        #
        # configuration hints:
        #
        #  - first state always has the name: start_state
        #  - every field in the rule must be inside double quotes
        #
        # rules |   state name  | regex pattern                  | next state
        # ------|---------------|--------------------------------------------
        rule      "start_state"   "/\[?\d{4}-\d{2}.*/"              "cont"
        rule      "cont"          "/^\s+at.*/"                      "cont"
        rule      "cont"          "/^[^0-9\[]{3}.*/"                "cont"

Metadata

Metadata

Assignees

No one assigned

    Labels

    Stale

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions