Repository navigation
Pipeline compilation significantly increased in 7.7.0 in comparison to 7.6.2 #12031
Description
Activity
- changed the title
[-]Regression in pipeline compilation [/-][+]Pipeline compilation significantly increased in `7.7.0` in comparison to `7.6.2`[/+]on Jun 17, 2020 - changed the title
[-]Pipeline compilation significantly increased in `7.7.0` in comparison to `7.6.2`[/-][+]Pipeline compilation significantly increased in 7.7.0 in comparison to 7.6.2[/+]on Jun 17, 2020 Here's a minimal code change that, when applied to v7.6.2, triggers this issue when starting two large pipelines.
Testing with two 3k line pipelines, goes from 3 min startup to > 10 minutes.diff --git a/logstash-core/src/main/java/org/logstash/config/ir/compiler/DatasetCompiler.java b/logstash-core/src/main/java/org/logstash/config/ir/compiler/DatasetCompiler.java index 785c09160..50cf6fc1e 100644 --- a/logstash-core/src/main/java/org/logstash/config/ir/compiler/DatasetCompiler.java +++ b/logstash-core/src/main/java/org/logstash/config/ir/compiler/DatasetCompiler.java @@ -196,7 +196,7 @@ public final class DatasetCompiler { final ValueSyntaxElement inputBuffer, final ClassFields fields, final AbstractFilterDelegatorExt plugin) { final ValueSyntaxElement filterField = fields.add(plugin); - final Closure body = Closure.wrap( + final Closure body = Closure.wrap(setPluginIdForLog4j(plugin), buffer(outputBuffer, filterField.call("multiFilter", inputBuffer)) ); if (plugin.hasFlush()) { @@ -303,6 +303,11 @@ public final class DatasetCompiler { ); } + private static MethodLevelSyntaxElement setPluginIdForLog4j(final AbstractFilterDelegatorExt filterPlugin) { + final IRubyObject pluginId = filterPlugin.getId(); + return () -> "org.apache.logging.log4j.ThreadContext.put(\"plugin.id\", \"" + pluginId + "\")"; + } + private static MethodLevelSyntaxElement clear(final ValueSyntaxElement field) { return field.call("clear"); }
Creating a method for each plugin id seems to blow up compilation.
Fix in #12038
Reacted by weijh and ryan4yin | 二花- added a commit that references this issue
on Jun 25, 2020 - added 3 commits that reference this issue
on Jun 25, 2020
7.7.0,7.7.1,7.8.0Starting from Logstash
7.7was released a feature to put the in logs also theplugin.idto help debugging of problems with pipelines (PR #11078 and #11593) that changed required to "decorate" each plugin invocation with code to set/unset the ThreadLocal withplugin.id(7a22220#diff-aa66540c4668f70bc7568433f4dac1efR155 and 7a22220#diff-aa66540c4668f70bc7568433f4dac1efR172). This introduction of more SythaxElements is causing slow downs in pipeline compilations, introducing a potential regression.To test it is sufficient to create a pipeline with 500 mutate filters (same filter replicated 500 times) and the pipeline bring more time to be compile in version
7.7then in version7.6.2In some test pipelines we have spot this increment of pipeline compilation times:
7.6.27.7.1