Execute triggers in parallel batches#701
Conversation
2a84640 to
1929629
Compare
|
@joebonrichie Have you experimented with progess tick values of 4-10ms yet? |
aea2783 to
eba25f7
Compare
eba25f7 to
40ab22b
Compare
b42cb85 to
a01d13d
Compare
tarkah
left a comment
There was a problem hiding this comment.
LGTM! Thanks @joebonrichie
|
How do we know that two seemingly unrelated triggers won't each write to the same file, meaning the trigger outcome is not deterministic (because the output might be different than if we knew that one would always come after the other)? |
In a nutshell, we don't. We have no mechanism for recording what files each trigger writes to |
In the context of tx triggers, we arguably should, as this is stuff that is under our control and will become a part of a tx via the postblit stuff. BTW, this could in fact be one of the reasons that Ikey chose not to parallelize tx triggers initially. In fact, doing the tx triggers serially would allow us to create a "before" list of files and an "after" list of files for each trigger. In the context of system triggers, this seems a bit pointless, as the tools we call should handle not writing to stuff they have no business writing to. |
|
Definitely a valid concern, but surely outside the scope of the this PR unless there are tx triggers writing to the same file without holding a lock on the file. |
Well, the solution we have now is arguably the more correct one, while this PR potentially sacrifices that correctness on the altar of speed. Hence the question. |
|
Converting this to draft until we have a better handle on the correctness story. |
Use the batched_topo() method to return a vec of vecs and execute each batch of triggers in parallel with rayon Refactor so we only enter the isolation container once for each scope of triggers executed (if neccessary) Add progress callbacks so we can continue to present a smooth progress bar to the user despite the batched execution On a VM with 4 vCPUs on an NVMe drive formatted with ext4, total wall time for installing a simple package is reduced from ~21secs to ~13secs.
a01d13d to
d7eac73
Compare
No description provided.