Adding more format?

[LiamKarlMitchell]

Is it possible to add more
For instance logs in some software im working on are using json logs from spring boot with logback.

The fields/keys to pull message or error or timestamp from may have different names/paths.

This may very well be a custom format not sure. But a way to configure the relevant keys per project to use including a custom path would be great to have...

[orangain]

@LiamKarlMitchell

Is it possible to add more
For instance logs in some software im working on are using json logs from spring boot with logback.

The fields/keys to pull message or error or timestamp from may have different names/paths.

Thank you for the suggestion! I would like the plugin to support various JSON log formats commonly used in the real world out of the box.

If you have any example of JSON logs that are not supported by the plugin yet, please share them with me. I will try to add support for them. It would be helpful if you could also provide the information about the log library and its configuration.

This may very well be a custom format not sure. But a way to configure the relevant keys per project to use including a custom path would be great to have...

Yes, custom format configurations would be useful. However, I believe that zero configuration is an important feature of the plugin. I would like to support as many log formats as possible before implementing custom configurations.

[top1436]

Hello!

Sorry to interrupt, I have a same opinion as reporter. I'am using logback with customizing field names. please refer to below my case!

message > log_message
timestamp > logdate

If user can customise the field in your plugin it would be very thankful for our team

Thank you!

[orangain]

@top1436 Thank you for the feedback! I would consider adding a feature to customize field names. However, I would like to better understand the use case. What does your team customize field names for? If you have software that requires such keys, I would like to support them out of the box.

[LiamKarlMitchell]

Maybe its just what the org names things because other things are configured to extract specific keys or is adhoc e.g. what they want to type in. You probably cant support them all as there is no standard being followed as inhouse or manually configured json structures could be used, as I found there was a conflict between message as a string and message as an object to have a message.description I had to put it first. Adding a custom key path config may help to resolve this, rather than hard coding the options make an array or split string by commas or just an override that could be set for project and/or user config? I was able to modify and compile to get it working by editing the code/definitions. However, not all projects are configured the same. Having a way to explicitly specify the keys would be helpful and probably faster than searching a larger list of unused keys per message as this could get more expensive the more combinations added.

…

On Wed, 21 Aug 2024, 10:48 pm orangain, ***@***.***> wrote: @top1436 <https://github.com/top1436> Thank you for the feedback! I would consider adding a feature to customize field names. However, I would like to better understand the use case. What does your team customize field names for? If you have software that requires such keys, I would like to support them out of the box. — Reply to this email directly, view it on GitHub <#57 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPAP3O7RXLDRJUKL6ABVJDZSRWA5AVCNFSM6AAAAABMPN4L7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBRG42TEMZYGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[orangain]

@LiamKarlMitchell I understood the situation. Even if I were to implement a customizable configuration, without knowing what kind of logs you want to parse, I cannot determine how much flexibility to give the configuration. It would be very helpful if you could give me an example of the actual logs you would like to parse.

[RovoMe]

It would be super cool if the plugin could also support logfmt formatted log lines as well where the log basically consists of

key1=value1 key2=value2 key3="value 3" ...

lines.

A very simple RL example i.e. looks like this:

...
time="2024-09-05T15:51:28.425" level=debug msg="Running with Spring Boot v3.2.6, Spring v6.1.8" package=org.springframework.boot thread=main
time="2024-09-05T15:51:28.425" level=info msg="The following 1 profile is active: \\\"docker\\\"" package=org.springframework.boot thread=main
...
time="2024-09-05T15:51:33.612" level=info msg="\\n\\nError starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled." package=org.springframework.boot.autoconfigure.logging thread=main
time="2024-09-05T15:51:33.621" level=error msg="\\n\\n***************************\\nAPPLICATION FAILED TO START\\n***************************\\n\\nDescription:\\n\\nBinding to target org.acme.Main failed:\\n\\n    Property: some.property\\n    Value: \\\"null\\\"\\n    Reason: must not be null\\n\\n\\nAction:\\n\\nUpdate your application's configuration\\n" package=org.springframework.boot.diagnostics thread=main

Here folding could also be indicated with either {...} like in the JSON case or with [...] to indicate a list of properties which upon expanding would just log each key-value pair in a new line, as you do already for JSON expanded entries. In the collapse form, the log line should probably show the timestamp, the log level and the actual message, usually indicated by msg and hide everything else (unless maybe configured) within the bracket symbols.

It would be nice if the plugin could also replace \\n and \n characters with new lines and accordingly any escaped characters in the value part as well

[orangain]

@RovoMe That's an interesting idea! Although the plugin is named as Pretty JSON Log, it might be reasonable to support other structured log formats.

Since the topic of this issue is custom JSON format, could you please create a new issue for the logfmt support? I would like to consider details there.

[extempl]

Hi @orangain. We're using LogstashEncoder, and the lib doesn't parse exceptions

{"@timestamp":"2025-03-07T12:18:14.547Z","@version":"1","message":"Handled controller exception (/api/1.0/companies/testcompany13/files/636575988/content/realMax800x801).","logger_name":"n.c.t.a.errors.ApiExceptionHandling","thread_name":"http-nio-8080-exec-3","level":"DEBUG","level_value":10000,"stack_trace":"no.cloudware.tibet.model.errors.ResourceNotFoundException: No CloudFile with id 636575988\n\tat no.cloudware.tibet.services.CloudFilePreviewService.getOrCreateCloudFilePreview(CloudFilePreviewService.java:273)\n\tat no.cloudware.tibet.services.CloudFilePreviewService.getOrCreateCloudFilePreview(CloudFilePreviewService.java:281)\n\tat no.cloudware.tibet.services.CloudFilePreviewService$$FastClassBySpringCGLIB$$90355549.invoke(<generated>)\n\tat org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)\n\tat org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)\n\tat org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)\n\tat org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)\n\tat org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)\n\tat no.cloudware.tibet.services.CloudFilePreviewService$$EnhancerBySpringCGLIB$$16bb7fd7.getOrCreateCloudFilePreview(<generated>)\n\tat no.cloudware.tibet.api.CloudFileController.getFileContent(CloudFileController.java:306)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072)\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965)\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)\n\tat org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:635)\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:742)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)\n\tat org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)\n\tat org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat no.cloudware.tibet.security.authentication.LoginTokenAuthenticationFilter.doFilter(LoginTokenAuthenticationFilter.java:99)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:158)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:155)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat no.cloudware.tibet.security.xss.XSSFilter.doFilter(XSSFilter.java:50)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat org.springframework.security.web.authentication.ui.DefaultLogoutPageGeneratingFilter.doFilterInternal(DefaultLogoutPageGeneratingFilter.java:52)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:216)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:92)\n\tat org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:77)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat no.cloudware.tibet.security.MDCFilter.doFilter(MDCFilter.java:52)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n\tat org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)\n\tat org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)\n\tat org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)\n\tat org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:186)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)\n\tat org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)\n\tat org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)\n\tat org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:389)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n","ipAddress":"127.0.0.1","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36","company":"testcompany13","companyId":"1045271","requestURL":"http://localhost:8080/api/1.0/companies/testcompany13/files/636575988/content/realMax800x801 (GET)"}

I don't really care about other stuff, but having exception in the console normally logged is essential.
It seems to normally parse other logs, so it might be something in the exception itself.

Also the problem is when it doesn't parse the log, it just shows {...} which is not explicit enough, that there is some log that couldn't be parsed. Should probably show first 100 symbols, or something like that.