Skip to content

Execute triggers in parallel batches#701

Draft
joebonrichie wants to merge 1 commit into
mainfrom
parallelized-triggers
Draft

Execute triggers in parallel batches#701
joebonrichie wants to merge 1 commit into
mainfrom
parallelized-triggers

Conversation

@joebonrichie
Copy link
Copy Markdown
Contributor

No description provided.

@joebonrichie joebonrichie force-pushed the parallelized-triggers branch 4 times, most recently from 2a84640 to 1929629 Compare February 18, 2026 13:22
@joebonrichie joebonrichie marked this pull request as ready for review February 18, 2026 13:53
@ermo
Copy link
Copy Markdown
Member

ermo commented Feb 18, 2026

@joebonrichie Have you experimented with progess tick values of 4-10ms yet?

@joebonrichie joebonrichie force-pushed the parallelized-triggers branch 6 times, most recently from aea2783 to eba25f7 Compare February 18, 2026 21:51
Comment thread crates/dag/src/lib.rs Outdated
Comment thread crates/dag/src/lib.rs
Comment thread moss/src/client/postblit.rs
Comment thread moss/src/client/postblit.rs Outdated
Comment thread moss/src/client/postblit.rs Outdated
@ermo ermo requested a review from tarkah February 19, 2026 09:49
@joebonrichie joebonrichie force-pushed the parallelized-triggers branch 2 times, most recently from b42cb85 to a01d13d Compare February 19, 2026 18:39
Copy link
Copy Markdown
Contributor

@tarkah tarkah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks @joebonrichie

@ermo
Copy link
Copy Markdown
Member

ermo commented Feb 20, 2026

How do we know that two seemingly unrelated triggers won't each write to the same file, meaning the trigger outcome is not deterministic (because the output might be different than if we knew that one would always come after the other)?

@joebonrichie
Copy link
Copy Markdown
Contributor Author

How do we know that two seemingly unrelated triggers won't each write to the same file, meaning the trigger outcome is not deterministic (because the output might be different than if we knew that one would always come after the other)?

In a nutshell, we don't. We have no mechanism for recording what files each trigger writes to

@ermo
Copy link
Copy Markdown
Member

ermo commented Feb 20, 2026

How do we know that two seemingly unrelated triggers won't each write to the same file, meaning the trigger outcome is not deterministic (because the output might be different than if we knew that one would always come after the other)?

In a nutshell, we don't. We have no mechanism for recording what files each trigger writes to.

In the context of tx triggers, we arguably should, as this is stuff that is under our control and will become a part of a tx via the postblit stuff. BTW, this could in fact be one of the reasons that Ikey chose not to parallelize tx triggers initially.

In fact, doing the tx triggers serially would allow us to create a "before" list of files and an "after" list of files for each trigger.

In the context of system triggers, this seems a bit pointless, as the tools we call should handle not writing to stuff they have no business writing to.

@joebonrichie
Copy link
Copy Markdown
Contributor Author

Definitely a valid concern, but surely outside the scope of the this PR unless there are tx triggers writing to the same file without holding a lock on the file.

@ermo
Copy link
Copy Markdown
Member

ermo commented Feb 20, 2026

Definitely a valid concern, but surely outside the scope of the this PR unless there are tx triggers writing to the same file without holding a lock on the file.

Well, the solution we have now is arguably the more correct one, while this PR potentially sacrifices that correctness on the altar of speed. Hence the question.

@ermo ermo marked this pull request as draft February 25, 2026 22:15
@ermo
Copy link
Copy Markdown
Member

ermo commented Feb 25, 2026

Converting this to draft until we have a better handle on the correctness story.

Use the batched_topo() method to return a vec of vecs and execute
each batch of triggers in parallel with rayon

Refactor so we only enter the isolation container once for each scope of
triggers executed (if neccessary)

Add progress callbacks so we can continue to present a smooth progress
bar to the user despite the batched execution

On a VM with 4 vCPUs on an NVMe drive formatted with ext4, total wall
time for installing a simple package is reduced from ~21secs to ~13secs.
@joebonrichie joebonrichie force-pushed the parallelized-triggers branch from a01d13d to d7eac73 Compare April 11, 2026 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants