Run jobs that take no environment from any runner - #23
Open
sgrif wants to merge 1 commit into
Open
Conversation
Attempting to update crates.io to the most recent version of swirl caused jobs to fail to run. The cause of this was that one of our jobs only needed a DB connection, not the environment. This caused the job's environment type to be `()`, meaning our runner would not find it in the registry (since the environment type didn't match). The fix on the crates.io side would be to add an unused argument with the environment, but this feels like such a major pitfall that I want to avoid it in Swirl. Having a few jobs that need a shared environment, and many that only need a database connection (or neither) seems common enough to warrant special casing jobs with no environment. This was relatively easy to track down on the crates.io side since the jobs were failing to run (and thus logging their failure). However, this does beg the question of whether having jobs with different environments in the same database is going to be a use case we need to support, in which case we probably want the runner to ignore job types it doesn't know about.
Owner
Author
|
@jtgeibel If you have a few to review this, it will fix the issue that rust-lang/crates.io#2269 uncovered |
sgrif
added a commit
to sgrif/crates.io
that referenced
this pull request
Apr 16, 2020
This is a second attempt at rust-lang#2269, which had to be reverted shortly after being deployed, as it caused jobs to fail to run in production. This commit only differs from that PR in the revision of swirl used. The cause of the issue was the change in the signature of `update_downloads`, which does not need anything from the job environment, and so it only takes a connection. This changed the environment type for that job from `background_jobs::Environment` to `()`, so our runner no longer knew how to run that job. The smallest fix in crates.io would have been to add an unused environment argument to `update_downloads`. However, having some jobs not require a shared environment felt common enough that I fixed this in swirl instead. sgrif/swirl#23 changed the behavior so jobs with no environment (a.k.a. the environment type is `()`) are always run, regardless of the environment type of the runner. Since `update_downloads` is essentially a cron job, and is not enqueued from the web server directly, this was not caught by our integration suite. This case was caught quickly after being deployed, and only affected a non-critical part of the service, so I've opted not to figure out how to add this to our integration suite.
bors
added a commit
to rust-lang/crates.io
that referenced
this pull request
May 2, 2020
Update Swirl, take two This is a second attempt at #2269, which had to be reverted shortly after being deployed, as it caused jobs to fail to run in production. This commit only differs from that PR in the revision of swirl used. The cause of the issue was the change in the signature of `update_downloads`, which does not need anything from the job environment, and so it only takes a connection. This changed the environment type for that job from `background_jobs::Environment` to `()`, so our runner no longer knew how to run that job. The smallest fix in crates.io would have been to add an unused environment argument to `update_downloads`. However, having some jobs not require a shared environment felt common enough that I fixed this in swirl instead. sgrif/swirl#23 changed the behavior so jobs with no environment (a.k.a. the environment type is `()`) are always run, regardless of the environment type of the runner. Since `update_downloads` is essentially a cron job, and is not enqueued from the web server directly, this was not caught by our integration suite. This case was caught quickly after being deployed, and only affected a non-critical part of the service, so I've opted not to figure out how to add this to our integration suite. r? @jtgeibel
jtgeibel
approved these changes
May 2, 2020
jtgeibel
left a comment
There was a problem hiding this comment.
This looks good to me. I've merged the crates.io PR and this change is currently deployed to staging.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Attempting to update crates.io to the most recent version of swirl
caused jobs to fail to run. The cause of this was that one of our jobs
only needed a DB connection, not the environment. This caused the job's
environment type to be
(), meaning our runner would not find it in theregistry (since the environment type didn't match).
The fix on the crates.io side would be to add an unused argument with
the environment, but this feels like such a major pitfall that I want to
avoid it in Swirl. Having a few jobs that need a shared environment, and
many that only need a database connection (or neither) seems common
enough to warrant special casing jobs with no environment.
This was relatively easy to track down on the crates.io side since the
jobs were failing to run (and thus logging their failure). However, this
does beg the question of whether having jobs with different environments
in the same database is going to be a use case we need to support, in
which case we probably want the runner to ignore job types it doesn't
know about.