Add --pre-build-helper flag to run scripts before package build#2145
Add --pre-build-helper flag to run scripts before package build#2145NicholasBHubbard wants to merge 6 commits into
Conversation
6028a25 to
d630f6a
Compare
|
@jordansissel |
|
Howdy @jordansissel , I hope you are doing well sir! :-) This PR #2145 and our other open PR #2146 seem to be the last two big hurdles we need to overcome in order to move forward with our current tasks and goals. While they are not related from FPM's point of view, they are related from our Perl point of view and we believe they are both as non-intrusive to FPM as possible. Can you please take a few minutes now to review and approve these two PRs? |
jordansissel
left a comment
There was a problem hiding this comment.
I briefly read the PR and this looks fairly straightforward. I want to try it to get you some better feedback shortly; I’m busy with some family medical stuff these past weeks and haven’t had the headspace to move this forward. Hopefully soon!
Thanks for working on this!
|
@jordansissel |
|
Howdy @jordansissel , I hope all is well with your family! |
jordansissel
left a comment
There was a problem hiding this comment.
First, high level feedback: I support this proposal.
Naming change: I'd prefer to name this a 'hook' and not a 'helper', that is, "pre-build-hook"
User experience and compatibility concerns: This change allows greater experimentation, and I like that, but it exposes internal parts of fpm to users in a way that might easily be mistaken as a "public interface" and have some expectation of stability or compatibility across fpm releases. Specifically, the directory layout of the build and staging paths. Historically, iirc, these directories are not exposed to users and make no promise about the contents (especially across versions of fpm).
Will it be ok if fpm can't make promises about the contents of these directories (especially across fpm versions, including future releases) ?
I appreciate y'all already updated all the package types to support this feature :)
I added specific feedback as inline comments.
| "FPM_BUILD_PATH" => build_path, | ||
| "FPM_OUTPUT_TYPE" => type, | ||
| "FPM_PACKAGE_NAME" => name, | ||
| "FPM_PACKAGE_VERSION" => (version || ""), |
There was a problem hiding this comment.
Use version.to_s here.
The default version value is a Float 1.0 and this value causes Process.spawn to fail when fpm's default version is used:
% fpm --verbose -s empty -t deb -n example --pre-build-helper true
/home/jls/projects/fpm/lib/fpm/util.rb:146:in 'Process.spawn': no implicit conversion of Float into String (TypeError)
There was a problem hiding this comment.
(Out of scope: Maybe fpm's version attribute should always be a string, but that's an adventure for another day!)
| } | ||
| helpers.each do |helper| | ||
| logger.info("Running pre-build helper", :helper => helper) | ||
| safesystem(env, helper) |
There was a problem hiding this comment.
When a helper doesn't exist, fpm will error (good!) with a message that I think could be improved. Here's an example:
% fpm -s empty -t deb -n example --pre-build-helper asdf
Need executable 'asdf' to convert empty to deb {level: :error}
If possible, improving this to indicate that this was a requested pre-build step, something like:
A pre-build hook executable was not found: 'asdf'
Catching FPM::Util::ExecutableNotFound on the safesystem call will help this specific case.
begin
safesystem(env, helper)
rescue FPM::Util::ExecutableNotFound => e
logger.error("A pre-build hook was configured, but the executable is not found: #{e}")
endThere was a problem hiding this comment.
Fixed in commit 96129ab. Note that instead of logging here, I raised a ProcessFailed exception, because I think this case needs to crash fpm. If we just logged and moved on, problems could arise from multiple pre-build hook setups, where subsequent hooks expected prior hooks to have run successfully.
| input.replaces += replaces | ||
| input.config_files += config_files | ||
| input.directories += directories | ||
| input.attributes[:pre_build_helpers] = pre_build_helpers |
There was a problem hiding this comment.
It's been a while since fpm's added a global flag like this, so my memory is a bit fuzzy on this one; writing notes here partly for myself -
If we add additional hooks/helpers, it might be useful to group these into a single ivar like package scripts are (in FPM::Package's @scripts ivar).
(No change requested; just making notes for our future selves).
|
@jordansissel |
|
Hi @jordansissel! Thank you very much for the feedback. I fixed all of your requests. In regards to not being able to make promises about the potential instability of fpm internals that pre-build hooks expose, I agree that it is acceptable to not make promises about future compatibility. I updated the documentation of the One extra thing I did, in commit 96129ab, was I tightened up the enforcement that pre-build hooks must be executable files, and cannot be arbitrary shell commands. I then forced Let me know how everything looks! |
|
@jordansissel |
jordansissel
left a comment
There was a problem hiding this comment.
I found a bug quickly so I wanted to get you a fast review. I'll review the additional tests and such shortly.
| hooks.each do |hook| | ||
| logger.info("Running pre-build hook", :hook => hook) | ||
| begin | ||
| safesystem(env, hook, hook) # pass hook twice to force a direct exec |
There was a problem hiding this comment.
I agree with the comment on this line, but I believe the correct call is still safesystem(env, hook) after reviewing the safesystem() and execmd() methods.
First, I admit the execmd code is pretty awkward to read while trying to understand how it handles arguments.
A bug: this line safesystem(env, hook, hook) actually calls the hook with its own name as the first command line argument.
Demonstration: the using date command as the hook runs the equivalent shell of date date with this code:
% fpm -s empty -t deb -n example --pre-build-hook "date" --verbose
Running command {args: ["date", "date"], level: :info}
date: invalid date ‘date’ {level: :info}
Process failed: date failed (exit code 1). Full command was:["date", "date"] {level: :error}
Comparing date date on the shell:
% date date
date: invalid date ‘date’
Now, I agree with wanting to direct execution (avoiding sh -c ...), but safeystem({env hash}, "command") already appears to do this with no changes -
safesystem(one_arg) will invoke sh -c <one_arg> only if safesystem() is invoked with exactly one argument. That is,
safesystem("echo hello")effectively becomes:sh -c "echo hello"safesystem(env, "echo hello")is two arguments and results in direct execution, and fails because "echo hello" is not a program name.
Evidence with minimal reproduction:
Calling safesystem("one argument") invokes sh -c ... instead of a direct invocation:
% ruby -r./lib/fpm/util -rcabin -e 'class A; include FPM::Util; end; a = A.new; a.logger.subscribe($stdout); a.safesystem("echo hello from safesystem")'
Running command {args: ["/bin/zsh", "-c", "echo hello from safesystem"], level: :info}
hello from safesystem {level: :info}
Calling safesystem({}, "one argument") still does direct invocation:
% ruby -r./lib/fpm/util -rcabin -e 'class A; include FPM::Util; end; a = A.new; a.logger.subscribe($stdout); a.safesystem({}, "echo hello from safesystem")'
/home/jls/projects/fpm/lib/fpm/util.rb:136:in 'FPM::Util#execmd': echo hello from safesystem (FPM::Util::ExecutableNotFound)
There was a problem hiding this comment.
Thank you @jordansissel for reviewing this. You are right about the bug here: safesystem(env, hook, hook) is incorrect, because it passes the hook path as an extra argv entry. Your date example demonstrates that clearly, and I agree that part needs to be fixed. I had not tested against an executable that fails when given an argument.
I had tried to document around this by stating in the --pre-build-hook help text that the executable should not expect to be passed any arguments. I think that wording is still ambiguous and does not clearly address the case you raised.
However, I do not think that safesystem(env, hook) is sufficient to reject shell-command strings.
The subtle issue is that safesystem(env, hook) does avoid fpm’s own sh -c fallback, but it still ends up calling Process.spawn(env, hook, ...) with a single command string. Ruby can still treat that form as shell input when the string contains shell metacharacters. This is explained in this documentation: https://docs.ruby-lang.org/en/3.4/Process.html#module-Process-label-Argument+command_line+or+exe_path
This is why the new spec fails when I remove the second hook argument. The example in the spec is:
cmd = "echo first >> /tmp/fpm-pre-build-hook-output"That string contains shell redirection, so it bypasses fpm’s program_in_path? guard and reaches Process.spawn as a single string.
Here is a minimal reproduction outside of fpm:
ruby -e '
cmd = %q{echo first >> /tmp/fpm-pre-build-hook-output}
pid = Process.spawn({}, cmd)
Process.wait(pid)
puts "status=#{$?.exitstatus}"
puts File.read("/tmp/fpm-pre-build-hook-output")
File.delete("/tmp/fpm-pre-build-hook-output")
'On my side, that produces:
status=0
first
Further, in fpm, with just safesytem(env, hook), I reproduced the following behavior:
$ fpm -s empty -t deb -n example --pre-build-hook "echo foo >> /tmp/foo.txt" --verbose | grep foo
{timestamp: "2026-04-29T14:32:03.488055-0400", message: "Running pre-build hook", hook: "echo foo >> /tmp/foo.txt", level: :info}
{timestamp: "2026-04-29T14:32:03.488101-0400", message: "Running command", args: ["echo foo >> /tmp/foo.txt"], level: :info}
$ cat /tmp/foo.txt
foo
So Process.spawn({}, cmd) did execute the shell command successfully.
For comparison, this shows why the current hook, hook version rejects it:
ruby -e '
cmd = %q{echo first >> /tmp/fpm-pre-build-hook-output}
begin
pid = Process.spawn({}, cmd, cmd)
Process.wait(pid)
rescue => e
puts "#{e.class}: #{e.message}"
end
'That produces:
Errno::ENOENT: No such file or directory - echo first >> /tmp/fpm-pre-build-hook-output
So I agree the current safesystem(env, hook, hook) call is buggy, but I do not think replacing it with safesystem(env, hook) preserves the “reject shell command hook strings” behavior. I think we need a different fix that guarantees direct exec.
I'm thinking the fix here is to make hook execution explicit instead of relying on safesystem’s string behavior. We could resolve the hook as an executable first, and then invoke it using Process.spawn's direct-exec form, or an equivalent helper in fpm. That would avoid the extra-argv bug from hook, hook, while also preventing shell-command strings from being interpreted as a shell command line.
Thanks again for the feedback!
This PR implements a new feature to fpm that I refer to as "pre-build helpers". This adds a flag to fpm called "--pre-build-helper", that lets you register a program/script to be run just before a package is actually built, when the package metadata file (such as the
.specorcontrolfile) has been generated by fpm. This flag can be specified multiple times to run multiple scripts.All pre-build helper scripts are run with the following environment variables set:
Myself and @wbraswell want to do "strange" things to modify our package metadata files, that really don't fit nicely into the architecture (and sanity) of fpm. Instead of trying to add our own weird flags to do things only we want to do, we came up with the idea to add this generic concept of a pre-build helper, which may be useful to others. We will personally write our strange code as custom scripts that are invoked as pre-build helpers.
I added tests for this new feature for every output-capable package type that already had a test script (I did not create new test scripts for the package types that don't have one). These tests are strong since they test not only that the pre-build helpers are run, but that they are run at the correct time when the package metadata file exists.
It is important to note that I had to go in to every package
outputmethod, and insert a call torun_pre_build_helpersin the correct place such that they are invoked after the metadata files have been created. It would be nice if we only had to make this call once (fromcommand.rb'sexecutefunction). However fpm's architecture for theoutputmethod is not fine-grained enough for this to be possible. This actually reminds me a lot of our previous work on adding source-based packages (#1972, #1967, #1954), where we tried to split the functionality ofinputandoutputinto more functions to allow more fine-grained control over the package build process.Let me know your thoughts and how things look!