Another big release that's been coming for quite some time (apologies to all who have been running with dev for weeks / months).

One or two major bugfixes, such as making nf-core download work again with the latest DSL2 formats. See below for the full list of changes in this release.

Full Changelog: 2.2...2.3

Template

• Removed mention of --singularity_pull_docker_container in pipeline README.md
• Replaced equals with ~ in nf-core headers, to stop false positive unresolved conflict errors when committing with VSCode.
• Add retry strategy for AWS megatests after releasing nf-core/tower-action v2.2
• Added .nf-core.yml file with repository_type: pipeline for modules commands
• Update igenomes path to the BWAIndex to fetch the whole version0.6.0 folder instead of only the genome.fa file
• Remove pinned Node version in the GitHub Actions workflows, to fix errors with markdownlint
• Bumped nf-core/tower-action to v3 and removed pipeline and revision from the AWS workflows, which were not needed
• Add yamllint GitHub Action.
• Add .yamllint.yml to avoid line length and document start errors (#1407)
• Add --publish_dir_mode back into the pipeline template (nf-core/rnaseq#752)
• Add optional loading of of pipeline-specific institutional configs to nextflow.config
• Make --outdir a mandatory parameter (nf-core/tools#1415)

General

• Updated nf-core download to work with latest DSL2 syntax for containers (#1379)
• Made nf-core modules create detect repository type with explicit .nf-core.yml instead of random readme stuff (#1391)
• Added a Gitpod environment and Dockerfile (#1384)
  • Adds conda, Nextflow, nf-core, pytest-workflow, mamba, and pip to base Gitpod Docker image.
  • Adds GH action to build and push Gitpod Docker image.
  • Adds Gitpod environment to template.
  • Adds Gitpod environment to tools with auto build of nf-core tool.
• Shiny new command-line help formatting (#1403)
• Call the command line help with -h as well as --help (was formerly just the latter) (#1404)
• Add .yamllint.yml config file to avoid line length and document start errors in the tools repo itself.
• Switch to yamllint-github-actionto be able to configure yaml lint exceptions (#1404)
• Prevent module linting KeyError edge case (#1321)
• Bump-versions: Don't trim the trailing newline on files, causes editorconfig linting to fail (#1265)
• Handle exception in nf-core list when a broken git repo is found (#1273)
• Updated URL for pipeline lint test docs (#1348)
• Updated nf-core create to tolerate failures and retry when fetching pipeline logos from the website (#1369)

Modules

• New command nf-core modules info that prints nice documentation about a module to the terminal ✨ (#1427)
• Linting a pipeline now fails instead of warning if a local copy of a module does not match the remote (#1313)
• Fixed linting bugs where warning was incorrectly generated for:
  • Module does not emit software version
  • Container versions do not match
  • input: / output: not being specified in module
  • Allow for containers from other biocontainers resource as defined here
• Fixed traceback when using stageAs syntax as defined here
• Added nf-core schema docs command to output pipline parameter documentation in Markdown format for inclusion in GitHub and other documentation systems (#741)
• Allow conditional process execution from the configuration file (#1393)
• Add linting for when condition(#1397)
• Added modules ignored table to nf-core modules bump-versions. (#1234)
• Added --conda-package-version flag for specifying version of conda package in nf-core modules create. (#1238)
• Add option of writing diffs to file in nf-core modules update using either interactive prompts or the new --diff-file flag.
• Fixed edge case where module names that were substrings of other modules caused both to be installed (#1380)
• Tweak handling of empty files when generating the test YAML (#1376)
  • Fail linting if a md5sum for an empty file is found (instead of a warning)
  • Don't skip the md5 when generating a test file if an empty file is found (so that linting fails and can be manually checked)
• Linting checks test files for TODO statements as well as the main module code (#1271)
• Handle error if manifest isn't set in nextflow.config (#1418)

Template

• Update repo logos to utilize GitHub's #gh-light/dark-mode-only, to switch between logos optimized for light or dark themes. The old repo logos have to be removed (in docs/images and assets/).
• Deal with authentication with private repositories
• Bump minimun Nextflow version to 21.10.3
• Convert pipeline template to updated Nextflow DSL2 syntax
• Solve circular import when importing nf_core.modules.lint
• Disable cache in nf_core.utils.fetch_wf_config while performing test_wf_use_local_configs.
• Modify software version channel handling to support multiple software version emissions (e.g. from mulled containers), and multiple software versions.
• Update dumpsoftwareversion module to correctly report versions with trailing zeros.
• Remove params.hostnames from the pipeline template (#1304)
• Update .gitattributes to mark installed modules and subworkflows as linguist-generated (#1311)
• Adding support for Julia package environments to nextflow.config(#1317)
• New YAML issue templates for pipeline bug reports and feature requests, with a much richer interface (#1165)
• Update AWS test GitHub Actions to use v2 of nf-core/tower-action
• Post linting comment even when linting.yml fails
• Update CONTRIBUTION.md bullets to remove points related to scrape_software_versions.py
• Update AWS test to set Nextflow version to 21.10.3

General

• Made lint check for parameters defaults stricter [#992]
  • Default values in nextflow.config must match the defaults given in the schema (anything with { in, or in main.nf is ignored)
  • Defaults in nextflow.config must now match the variable type specified in the schema
  • If you want the parameter to not have a default value, use null
  • Strings set to false or an empty string in nextflow.config will now fail linting
• Bump minimun Nextflow version to 21.10.3
• Changed questionary ask() to unsafe_ask() to not catch KeyboardInterupts (#1237)
• Fixed bug in nf-core launch due to revisions specified with -r not being added to nextflow command. (#1246)
• Update regex in readme test of nf-core lint to agree with the pipeline template (#1260)
• Update 'fix' message in nf-core lint to conform to the current command line options. (#1259)
• Fixed bug in nf-core list when NXF_HOME is set
• Run CI test used to create and lint/run the pipeline template with minimum and latest edge release of NF (#1304)
• New YAML issue templates for tools bug reports and feature requests, with a much richer interface (#1165)
• Handle synax errors in Nextflow config nicely when running nf-core schema build (#1267)
• Erase temporary files and folders while performing Python tests (pytest)
• Remove base Dockerfile used for DSL1 pipeline container builds
• Run tests with Python 3.10
• #1363 Fix tools CI workflow nextflow versions.

Modules

• Fixed typo in module_utils.py.
• Fixed failing lint test when process section was missing from module. Also added the local failing tests to the warned section of the output table. (#1235)
• Added --diff flag to nf-core modules update which shows the diff between the installed files and the versions
• Update nf-core modules create help texts which were not changed with the introduction of the --dir flag
• Check if README is from modules repo
• Update module template to DSL2 v2.0 (remove functions.nf from modules template and updating main.nf (#1289)
• Substitute get process/module name custom functions in module main.nf using template replacement (#1284)
• Linting now fails instead of warning if a local copy of a module does not match the remote (#1313)
• Check test YML file for md5sums corresponding to empty files (#1302)
• Exit with an error if empty files are found when generating the test YAML file (#1302)

Template

• Correct regex pattern for file names in nextflow_schema.json
• Remove . from nf-core/tools command examples
• Update Nextflow installation link in pipeline template (#1201)
• Command hostname is not portable [#1212]
• Changed how singularity and docker links are written in template to avoid duplicate links

General

• Changed names of some flags with -r as short options to make the flags more consistent between commands.

Modules

• Added consistency checks between installed modules and modules.json (#1199)
• Added support excluding or specifying version of modules in .nf-core.yml when updating with nf-core modules install --all (#1204)
• Created nf-core modules update and removed updating options from nf-core modules install
• Added missing function call to nf-core lint (#1198)
• Fix nf-core lint not filtering modules test when run with --key (#1203)
• Fixed nf-core modules install not working when installing from branch with -b (#1218)
• Added prompt to choose between updating all modules or named module in nf-core modules update
• Check if modules is installed before trying to update in nf-core modules update
• Verify that a commit SHA provided with --sha exists for install/update commands
• Add new-line to main.nf after bump-versions command to make ECLint happy

Template

• Critical tweak to add --dir declaration to nf-core lint GitHub Actions linting.yml workflow

General

• Add --dir declaration to nf-core sync GitHub Actions sync.yml workflow

v2.0 - Palladium Platypus - [2021-07-13]

⚠️ Major enhancements & breaking changes

This marks the first Nextflow DSL2-centric release of tools which means that some commands won't work in full with DSL1 pipelines anymore. Please use a v1.x version of tools for such pipelines or better yet join us to improve our DSL2 efforts! Here are the most important changes:

• The pipeline template has been completely re-written in DSL2
• A module template has been added to auto-create best-practice DSL2 modules to speed up development
• A whole suite of commands have been added to streamline the creation, installation, removal, linting and version bumping of DSL2 modules either installed within pipelines or the nf-core/modules repo

Template

• Move TODO item of contains: map in a YAML string [#1082]
• Trigger AWS tests via Tower API [#1160]

General

• Fixed a bug in the Docker image build for tools that failed due to an extra hyphen. [#1069]
• Regular release sync fix - this time it was to do with JSON serialisation [#1072]
• Fixed bug in schema validation that ignores upper/lower-case typos in parameters [#1087]
• Bugfix: Download should use path relative to workflow for configs
• Remove lint checks for files related to conda and docker as not needed anymore for DSL2
• Removed params_used lint check because of incompatibility with DSL2
• Addedmodules bump-versions command to README.md
• Update docs for v2.0 release

Modules

• Update comment style of modules functions.nf template file [#1076]
• Changed working directory to temporary directory for nf-core modules create-test-yml [#908]
• Use Biocontainers API instead of quayi.io API for nf-core modules create [#875]
• Update nf-core modules install to handle different versions of modules #1116
• Added nf-core modules bump-versions command to update all versions in the nf-core/modules repository [#1123]
• Updated nf-core modules lint to check whether a git_sha exists in the modules.json file or whether a new version is available [#1114]
• Refactored nf-core modules command into one file per command #1124
• Updated nf-core modules remove to also remove entry in modules.json file (#1115)
• Bugfix: Interactive prompt for nf-core modules install was receiving too few arguments
• Added progress bar to creation of 'modules.json'
• Updated nf-core modules list to show versions of local modules
• Improved exit behavior by replacing sys.exit with exceptions
• Updated nf-core modules remove to remove module entry in modules.json if module directory is missing
• Create extra tempdir as work directory for nf-core modules create-test-yml to avoid adding the temporary files to the test.yml
• Refactored passing of command line arguments to nf-core commands and subcommands (#1139, #1140)
• Check for modules.json for entries of modules that are not actually installed in the pipeline [#1141]
• Added <keywords> argument to nf-core modules list for filtering the listed modules. (#1139
• Added support for a bump-versions configuration file [#1142]
• Fixed nf-core modules create-test-yml so it doesn't break when the output directory is supplied [#1148]
• Updated nf-core modules lint to work with new directory structure [#1159]
• Updated nf-core modules install and modules.json to work with new directory structure (#1159)
• Updated nf-core modules remove to work with new directory structure [#1159]
• Restructured code and removed old table style in nf-core modules list
• Fixed bug causing modules.json creation to loop indefinitly
• Added --all flag to nf-core modules install
• Added remote and local subcommands to nf-core modules list
• Fix bug due to restructuring in modules template
• Added checks for verifying that the remote repository is well formed
• Added checks to ModulesCommand for verifying validity of remote repositories
• Misc. changes to modules install: check that module exist in remote, --all is has --latest by default.

Sync

• Don't set the default value to "null" when a parameter is initialised as null in the config [#1074]

Tests

• Added a test for the version_consistency lint check
• Refactored modules tests into separate files, and removed direct comparisons with number of tests in lint tests (#1158)

Template

• Fixed an issue regarding explicit disabling of unused container engines [#972]
• Removed trailing slash from params.igenomes_base to yield valid s3 paths (previous paths work with Nextflow but not aws cli)
• Added a timestamp to the trace + timetime + report + dag filenames to fix overwrite issue on AWS
• Rewrite the params_summary_log() function to properly ignore unset params and have nicer formatting [#971]
• Fix overly strict --max_time formatting regex in template schema [#973]
• Convert d to day in the cleanParameters function to make Duration objects like 2d pass the validation [#858]
• Added nextflow version to quick start section and adjusted nf-core bump-version [#1032]
• Use latest stable Nextflow version 21.04.0 for CI tests instead of the -edge release

Download

• Fix bug in nf-core download where image names were getting a hyphen in nf-core which was breaking things.
• Extensive new interactive prompts for all command line flags [#1027]
  • It is now recommended to run nf-core download without any cli options and follow prompts (though flags can be used to run non-interactively if you wish)
• New helper code to set $NXF_SINGULARITY_CACHEDIR and add to .bashrc if desired [#1027]

Launch

• Strip values from nf-core launch web response which are False and have no default in the schema [#976]
• Improve API caching code when polling the website, fixes noisy log message when waiting for a response [#1029]
• New interactive prompts for pipeline name [#1027]

Modules

• Added tool_name_underscore to the module template to allow TOOL_SUBTOOL in main.nf [#1011]
• Added --conda-name flag to nf-core modules create command to allow sidestepping questionary [#988]
• Extended nf-core modules lint functionality to check tags in test.yml and to look for a entry in the pytest_software.yml file
• Update modules commands to use new test tag format tool/subtool
• New modules lint test comparing the functions.nf file to the template version
• Modules installed from alternative sources are put in folders based on the name of the source repository

Linting

• Fix bug in nf-core lint config skipping for the nextflow_config test [#1019]
• New -k/--key cli option for nf-core lint to allow you to run only named lint tests, for faster local debugging
• Merge markers lint test - ignore binary files, allow config to ignore specific files [#1040]
• New lint test to check if all defined pipeline parameters are mentioned in main.nf [#1038]
• Added fix to remove warnings about params that get converted from camelCase to camel-case [#1035]
• Added pipeline schema lint checks for missing parameter description and parameters outside of groups [#1017]

General

• Try to fix the fix for the automated sync when we submit too many PRs at once [#970]
• Rewrite how the tools documentation is deployed to the website, to allow multiple versions
• Created new Docker image for the tools cli package - see installation docs for details [#917]
• Ignore permission errors for setting up requests cache directories to allow starting with an invalid or read-only HOME directory

This is a patch release fixing some nasty bugs that crept into the code base for the nf-core helper tools. There are no changes to the nf-core pipeline template, so there should be no automated pipeline synchronisation pull-requests for this release.

New features

• Running tests twice with nf-core modules create-test-yml to catch unreproducible md5 sums [#890]
• nf-core modules create command - if no bioconda package is found, prompt user for a different bioconda package name
• Updated module template main.nf with new test data paths

Bug fixes

• Fix sync error again where the Nextflow edge release needs to be used for some pipelines
• Fix bug with nf-core lint --release (NameError: name 'os' is not defined)
• Added linebreak to linting comment so that markdown header renders on PR comment properly

• Added better logging message if a user doesn't specificy the directory correctly with nf-core modules commands [#942]
• Fixed parameter validation bug caused by JSONObject [#937]
• Fixed template creation error regarding file permissions [#932]
• Split the create-lint-wf tests up into separate steps in GitHub Actions to make the CI results easier to read
• Added automated PR comments to the Markdown, YAML and Python lint CI tests to explain failures (tools and pipeline template)
• Make nf-core lint summary table borders coloured according to overall pass / fail status
• Attempted a fix for the automated sync when we submit too many PRs at once [#911]

Minor patch release to scoop up a small few errors discovered after the release.

• Fixed bug in pipeline linting markdown output that gets posted to PR comments [#914]
• Made text for the PR branch CI check less verbose with a TLDR in bold at the top
• A number of minor tweaks to the new nf-core modules lint code

When we do releases, we sometimes like to go big. This is one of those times.

This is a truly huge release of nf-core/tools with a massive amount of new functionality. Please be sure to read the changelog and updated readme carefully!

Note to developers - please run and push the results of nf-core lint --fix before attempting to merge the automated synchronisation PR. This should reduce the size of the diff and make merging a lot easier.

The template and associated updates in this release is for Nextflow DSL1 pipelines. This is likely to be one of the last releases before we officially switch to DSL2. For those of you already with DSL2 pipelines you may need to treat the automated merge PR very carefully. Ask for help on Slack if in any doubt.

Template

• Major new feature - Validation of pipeline parameters [#426]
  • The addition runs as soon as the pipeline launches and checks the pipeline input parameters two main things:
    • No parameters are supplied that share a name with core Nextflow options (eg. --resume instead of -resume)
    • Supplied parameters validate against the pipeline JSON schema (eg. correct variable types, required values)
  • If either parameter validation fails or the pipeline has errors, a warning is given about any unexpected parameters found which are not described in the pipeline schema.
  • This behaviour can be disabled by using --validate_params false
• Added profiles to support the Charliecloud and Shifter container engines [#824]
  • Note that Charliecloud requires Nextflow version v21.03.0-edge or later.
• Profiles for container engines now explicitly disable all other engines [#867]
• Fixed typo in nf-core-lint CI that prevented the markdown summary from being automatically posted on PRs as a comment.
• Changed default for --input from data/*{1,2}.fastq.gz to null, as this is now validated by the schema as a required value.
• Removed support for --name parameter for custom run names.
  • The same functionality for MultiQC still exists with the core Nextflow -name option.
• Added to template docs about how to identify process name for resource customisation
• The parameters --max_memory and --max_time are now validated against a regular expression [#793]
  • Must be written in the format 123.GB / 456.h with any of the prefixes listed in the Nextflow docs
  • Bare numbers no longer allowed, avoiding people from trying to specify GB and actually specifying bytes.
• Switched from cookiecutter to Jinja2 [#880]
• Finally dropped the wonderful cookiecutter library that was behind the first pipeline template that led to nf-core [#880]
  • Now rendering templates directly using Jinja, which is what cookiecutter was doing anyway

Modules

Initial addition of a number of new helper commands for working with DSL2 modules:

• modules list - List available modules
• modules install - Install a module from nf-core/modules
• modules remove - Remove a module from a pipeline
• modules create - Create a module from the template
• modules create-test-yml - Create the test.yml file for a module with md5 sums, tags, commands and names added
• modules lint - Check a module against nf-core guidelines

You can read more about each of these commands in the main tools documentation (see README.md or https://nf-co.re/tools)

Tools helper code

• Fixed some bugs in the command line interface for nf-core launch and improved formatting [#829]
• New functionality for nf-core download to make it compatible with DSL2 pipelines [#832]
  • Singularity images in module files are now discovered and fetched
  • Direct downloads of Singularity images in python allowed (much faster than running singularity pull)
  • Downloads now work with $NXF_SINGULARITY_CACHEDIR so that pipelines sharing containers have efficient downloads
• Changed behaviour of nf-core sync command [#787]
  • Instead of opening or updating a PR from TEMPLATE directly to dev, a new branch is now created from TEMPLATE and a PR opened from this to dev.
  • This is to make it easier to fix merge conflicts without accidentally bringing the entire pipeline history back into the TEMPLATE branch (which makes subsequent sync merges much more difficult)

Linting

• Major refactor and rewrite of pipieline linting code
  • Much better code organisation and maintainability
  • New automatically generated documentation using Sphinx
  • Numerous new tests and functions, removal of some unnecessary tests
• Added lint check for merge markers [#321]
• Added new option --fix to automatically correct some problems detected by linting
• Added validation of default params to nf-core schema lint [#823]
• Added schema validation of GitHub action workflows to lint function [#795]
• Fixed bug in schema title and description validation
• Added second progress bar for conda dependencies lint check, as it can be slow [#299]
• Added new lint test to check files that should be unchanged from the pipeline.
• Added the possibility to ignore lint tests using a nf-core-lint.yml config file [#809]