add a few additional optimizations to the callback receiver - #8193
Conversation
|
Build succeeded.
|
| def emit_event_detail(event): | ||
| if ( | ||
| settings.UI_LIVE_UPDATES_ENABLED is False and | ||
| event.event not in ('playbook_on_stats', 'EOF') |
There was a problem hiding this comment.
I'd like to maybe make this be not startswith("runner_"), but I'm having some trouble getting the UI to behave in a reasonable way cc @jakemcdermott is taking a look, I think.
|
How hard is a placeholder in the text window in this case? |
56e3470 to
9d23f64
Compare
|
Build succeeded.
|
9d23f64 to
b22de28
Compare
|
Build succeeded.
|
|
@wenottingham see updated behavior in GIFs above. |
b22de28 to
b6afc08
Compare
|
Build succeeded.
|
|
Build succeeded.
|
|
Here are some before/after numbers of this PR processing a large backlog of events on a single node before (devel)after (
|
|
Here is the same comparison again, but with logging configured and pointed at a local UDP port via rsyslogd: before (devel)after (UI_LIVE_UPDATES_ENABLED = False) |
|
With |
| 'Event data saved.', | ||
| extra=dict(python_objects=dict(job_event=self)) | ||
| ) | ||
| if settings.LOG_AGGREGATOR_ENABLED: |
There was a problem hiding this comment.
Our custom filter technically checks this flag, too, but it does so after making the logging record and looping through all the handlers (which is actually pretty expensive). If logging is disabled, it's cheaper to just not call this .info() function at all.
| def SETTINGS_MODULE(self): | ||
| return self._get_default('SETTINGS_MODULE') | ||
|
|
||
| @cachetools.cached(cache=cachetools.TTLCache(maxsize=2048, ttl=SETTING_MEMORY_TTL)) |
There was a problem hiding this comment.
In my profiling for high workloads, the cache hits for redis actually add up and end up being quite expensive in bulk; even though it's just a round trip to a domain socket, it's still 10+ round trips per emitted event (because we access a handful of distinct settings to determine how to log) and that has a real cost when you multiply it by millions of events
For the purposes of the callback receiver, it's probably not imperative that we have real-time correctness for these setting values (caching for a few seconds locally in memory makes a notable performance improvement).
| return dict(host_status_counts) | ||
|
|
||
|
|
||
| MINIMAL_EVENTS = set([ |
There was a problem hiding this comment.
I would think that with a set this small that the hash calculation would overwhelm the big-O lookup advantage you gain by using a set instead of a tuple.
|
Build succeeded (gate pipeline).
|
|
Hello @ryanpetrello, @kdelee CC: @psuriset, We deployed patch on 3 node cluster and all the 3 nodes are virtual machines (created using libvirt) with RHEL 7.8 installed. Each vm's configured with 4vcpus, 23G RAM, and 75G of shared disk space. We can see 33.8% increase in events processing rate after changing Please let me know if there is anything else that i need to test it. |
|
This looks great - thank you @mcharanrm! |
cc: @gamuniz
Many events, few tasks:
Many events, many tasks:
