Skip to content

Run jobs that take no environment from any runner - #23

Open
sgrif wants to merge 1 commit into
masterfrom
sg-always-run-envless-jobs
Open

Run jobs that take no environment from any runner#23
sgrif wants to merge 1 commit into
masterfrom
sg-always-run-envless-jobs

Conversation

@sgrif

@sgrif sgrif commented Apr 16, 2020

Copy link
Copy Markdown
Owner

Attempting to update crates.io to the most recent version of swirl
caused jobs to fail to run. The cause of this was that one of our jobs
only needed a DB connection, not the environment. This caused the job's
environment type to be (), meaning our runner would not find it in the
registry (since the environment type didn't match).

The fix on the crates.io side would be to add an unused argument with
the environment, but this feels like such a major pitfall that I want to
avoid it in Swirl. Having a few jobs that need a shared environment, and
many that only need a database connection (or neither) seems common
enough to warrant special casing jobs with no environment.

This was relatively easy to track down on the crates.io side since the
jobs were failing to run (and thus logging their failure). However, this
does beg the question of whether having jobs with different environments
in the same database is going to be a use case we need to support, in
which case we probably want the runner to ignore job types it doesn't
know about.

Attempting to update crates.io to the most recent version of swirl
caused jobs to fail to run. The cause of this was that one of our jobs
only needed a DB connection, not the environment. This caused the job's
environment type to be `()`, meaning our runner would not find it in the
registry (since the environment type didn't match).

The fix on the crates.io side would be to add an unused argument with
the environment, but this feels like such a major pitfall that I want to
avoid it in Swirl. Having a few jobs that need a shared environment, and
many that only need a database connection (or neither) seems common
enough to warrant special casing jobs with no environment.

This was relatively easy to track down on the crates.io side since the
jobs were failing to run (and thus logging their failure). However, this
does beg the question of whether having jobs with different environments
in the same database is going to be a use case we need to support, in
which case we probably want the runner to ignore job types it doesn't
know about.
@sgrif

sgrif commented Apr 16, 2020

Copy link
Copy Markdown
Owner Author

@jtgeibel If you have a few to review this, it will fix the issue that rust-lang/crates.io#2269 uncovered

sgrif added a commit to sgrif/crates.io that referenced this pull request Apr 16, 2020
This is a second attempt at rust-lang#2269, which had to be reverted shortly
after being deployed, as it caused jobs to fail to run in production.
This commit only differs from that PR in the revision of swirl used.

The cause of the issue was the change in the signature of
`update_downloads`, which does not need anything from the job
environment, and so it only takes a connection. This changed the
environment type for that job from `background_jobs::Environment` to
`()`, so our runner no longer knew how to run that job.

The smallest fix in crates.io would have been to add an unused
environment argument to `update_downloads`. However, having some jobs
not require a shared environment felt common enough that I fixed this in
swirl instead. sgrif/swirl#23 changed the
behavior so jobs with no environment (a.k.a. the environment type is
`()`) are always run, regardless of the environment type of the runner.

Since `update_downloads` is essentially a cron job, and is not enqueued
from the web server directly, this was not caught by our integration
suite. This case was caught quickly after being deployed, and only
affected a non-critical part of the service, so I've opted not to figure
out how to add this to our integration suite.
bors added a commit to rust-lang/crates.io that referenced this pull request May 2, 2020
Update Swirl, take two

This is a second attempt at #2269, which had to be reverted shortly
after being deployed, as it caused jobs to fail to run in production.
This commit only differs from that PR in the revision of swirl used.

The cause of the issue was the change in the signature of
`update_downloads`, which does not need anything from the job
environment, and so it only takes a connection. This changed the
environment type for that job from `background_jobs::Environment` to
`()`, so our runner no longer knew how to run that job.

The smallest fix in crates.io would have been to add an unused
environment argument to `update_downloads`. However, having some jobs
not require a shared environment felt common enough that I fixed this in
swirl instead. sgrif/swirl#23 changed the
behavior so jobs with no environment (a.k.a. the environment type is
`()`) are always run, regardless of the environment type of the runner.

Since `update_downloads` is essentially a cron job, and is not enqueued
from the web server directly, this was not caught by our integration
suite. This case was caught quickly after being deployed, and only
affected a non-critical part of the service, so I've opted not to figure
out how to add this to our integration suite.

r? @jtgeibel

@jtgeibel jtgeibel left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. I've merged the crates.io PR and this change is currently deployed to staging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants