Skip to content

BeforeAll executes even if the example is not executed. #345

@jadutter

Description

@jadutter

BeforeAll appears to execute regardless whether the examples are going to be executed. This seems like a bug to me.
I also cannot seem to find an elegant way to conditionally execute an Include based on whether the example would run based on the CLI filter arguments.

The Problem

The script below demonstrates what I mean.

#! /bin/sh 
# shellcheck shell=sh

# define a temporary directory to write our sample project
dir=/tmp/hello
lib_dir="${dir}/lib"
spec_dir="${dir}/spec"

# define some shorthand utility functions
pad(){
  sed -E 's,^,    ,g;'
}
msg(){
  printf '\033[1;31m%s\033[0m\n' "$@" 
}
divider(){
  msg ''
}

# define a function 
# - to show the shellspec command we are running
#  - run the command, and indent the output
# - print a comment about the output result
run_cmd(){
  cmd="${1}"
  comment="${2}"
  msg "Running '${cmd}'"
  eval "${cmd}" | pad
  msg "${comment}" | pad
  divider
}

# setup the project directory, and shellspec config
setup_dir(){
  rm -rf "${dir}";
  mkdir -p "${lib_dir}" "${spec_dir}" && \
  cd "${dir}" && \
  shellspec --init && \
  cat<<'HERE'>"${dir}/.shellspec"
--format=t
--color
--require spec_helper
HERE

}

# create the spec files we want shellspec to run
create_specs(){
  # create a the tutorial spec
  cat<<'HERE'>"${spec_dir}/hello_spec.sh"
Describe 'hello.sh' hello
  Include lib/hello.sh
  It 'says hello'
    When call hello ShellSpec
    The output should equal 'Hello ShellSpec!'
  End
End
HERE

  # create a spec 
  cat<<'HERE'>"${spec_dir}/init_spec.sh"
match_any_value(){
  return 0
}

# check we are running in a supported shell before running the init tests
check_shell(){
  case "${SHELLSPEC_SHELL}" in
    *zsh|*bash)
      return 1
      ;;
    *)
      return 0
      ;;
  esac
}

Describe 'init'
  Skip if "skip because ${SHELLSPEC_SHELL} is unsupported" check_shell
  Describe 'setup'
    # before running the examples, define what we expect to find and import 
    # the correct init script based on the shell we are using
    BeforeAll setup
    setup(){
      printf "\033[1;34mSETUP INIT\033[0m\n"
      # the variables EXEC_BASH_PROFILE and EXEC_ZPROFILE should only be set
      # if their respective file executes
      expected_exc_bash_profile_status=1
      expected_exc_zprofile_status=1

      # only import the respective file if we are in the correct shell, and 
      # we have not already imported it
      case "${SHELLSPEC_SHELL}" in
        *bash)
          if [[ "${#EXEC_BASH_PROFILE}" -ne 1 || "${EXEC_BASH_PROFILE}" -ne 0 ]]; then 
            source lib/bash_profile
          fi
          expected_exc_bash_profile_status=0
          ;;
        *zsh)
          if [[ "${#EXEC_ZPROFILE}" -ne 1 || "${EXEC_ZPROFILE}" -ne 0 ]]; then 
            source lib/zprofile
          fi
          expected_exc_zprofile_status=0
          ;;
        *)
          :
          ;;
      esac
    }
    Describe 'bash' shell:bash
      It 'EXEC_BASH_PROFILE' custom_var:state
        When call printenv EXEC_BASH_PROFILE
        The stdout should satisfy match_any_value
        The status should equal "${expected_exc_bash_profile_status}"
      End
    End
    Describe 'zsh' shell:zsh
      It 'EXEC_ZPROFILE' custom_var:state
        When call printenv EXEC_ZPROFILE
        The stdout should satisfy match_any_value
        The status should equal "${expected_exc_zprofile_status}"
      End
    End
  End
End
HERE

}

# create the scripts we want shellspec to test
create_libs(){
    cat<<'HERE'>"${lib_dir}/hello.sh"
hello() {
  echo "Hello ${1}!"
}
HERE

    cat<<'HERE'>"${lib_dir}/bash_profile"
printf "\033[1;34mLOADED bash_profile\033[0m\n"
export EXEC_BASH_PROFILE=0
HERE

    cat<<'HERE'>"${lib_dir}/zprofile"
printf "\033[1;34mLOADED zprofile\033[0m\n"
export export EXEC_ZPROFILE=0
HERE

}

# run shellspec with different arguments, 
# commenting on the output
run_specs(){
  cd "${dir}" || return 1

  run_cmd \
    "shellspec" \
    "Automatically skipped the init tests because shellspec defaults to using /bin/sh,
    so the 'BeforeAll setup' correctly never executes." 

  run_cmd \
    "shellspec --shell bash" \
    "Runs all the tests in bash, and they all pass. 
    Correctly imports only the bash_profile when 'BeforeAll setup' executes.
    Using 'shellspec --shell zsh' does the same, but imports zprofile instead."    

  run_cmd \
    "shellspec -E hello.sh" \
    "Runs only the hello.sh example. 
    Correctly ignores the 'BeforeAll setup' in the init tests."

  run_cmd \
    "shellspec -E hello.sh --shell bash" \
    "Should run only the hello.sh example in bash. 
    Incorrectly executes the 'BeforeAll setup' and imports the bash_profile.
    Using 'shellspec -T hello --shell bash' does the same thing."

  run_cmd \
    "bash -c \". lib/bash_profile; shellspec -E hello.sh --shell bash\"" \
    "Explicitly running shellspec in a sub process where we source the init file
    saves us from needing to conditionally source the correct file in our init tests.
    Correctly runs only the hello.sh example in bash. 
    Incorrectly executes the 'BeforeAll setup', but does not bother to source the bash_profile again."

}

(
  # do it all in a subshell to prevent actually changing the directory
  setup_dir && \
  create_specs && \
  create_libs && \
  run_specs 
)

And when executed, I get the following output.

Image

"Solutions"

I've tried tackling this a couple of different ways.

Using Intercept to abort the init script

I've tried following he Intercept example to try to get the init script to stop executing, but I can't seem to find a way good way to tell if the init tests will be executed given the current cli parameters.

Include within the Describe for each shell

This runs into the same issue as BeforeAll setup, as it executes regardless of which examples will be targeted by the CLI arguments, but worse it imports both init scripts.

Using Include within an example also appears to be disallowed.

Sourcing the file from within the individual example

Doing something like this

When run ${SHELLSPEC_SHELL} -c ". ${init_file} ; printenv EXEC_BASH_PROFILE"

and using BeforeAll to set the init_file works, but adds a lot of time to the tests when the init file gets more complicated and runs longer.

Wrapping the shellspec command

Calling bash -c '. lib/bash_profile; shellspec --shell bash' seems like the best method given the current limitations.

Moving import logic to spec_helper

Instead of having BeforeAll setup conditionally import the init file, we can move the SHELLSPEC_SHELL check into spec_helper_configure in spec/spec_helper.sh.

This is the solution I'm going with for now, though I wish I had a better way to make sure the init file is loaded only if the init tests are going to be executed.

Conclusion

Running a BeforeAll hook for a Descibe block when the block won't be executed seems like a bug to me. If not, perhaps there's an opportunity here to add a new hook that will only execute if the block it is in will be executed.

Or perhaps provide a better way to conditionally Include

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions