Document use of SquashFS image as gtdbtk database. by muniheart · Pull Request #793 · nf-core/mag

muniheart · 2025-04-18T20:27:45Z

PR checklist

This PR documents a feature of subworkflow GTDBTK. Users with limited storage resources can economize on inodes (The uncompressed database requires >200K inodes!) by providing the database for GTDB-Tk as a SquashFS image.
This feature is only available when using container engines singularity and apptainer.

This is a replacement of #785.

** Why is a PR necessary to use a SquashFS image? Why can't I just mount the image and pass its path?

Mounting a file image requires permission to mount a loop device. If you have sufficient privilege on your system to mount a loop device then you don't need this PR. Singularity and apptainer allow an unprivileged user to bind-mount a file system image to a container's file system. This PR documents the configuration of the bind-mount options.

** Why is a PR necessary to bind-mount an image? Why not just configure the parameters in containerOptions?

Setting containerOptions is sufficient. This PR provides usage details.

** Why not just pass the database path in input path(db)?

Prior to 788, the database was input to process GTDBTK_CLASSIFYWF as,

    input:
        path( 'database/*' )

This allowed for the configuration

process {
    withName: GTDBTK_CLASSIFYWF {
        containerOptions = "-B $params.gtdb_db:\$NXF_TASK_WORKDIR/database:image-src=/release220"
    }
}

This worked because shell function nxf_stage in .command.run script would create a directory named database and the image would be mounted there. With 788, nxf_stage creates a symbolic link to the database. The solution is to mount the image file system somewhere outside the workDir of the process and pass its location in path(db).

** Why not specify a distinguished absolute path at which to mount the image when the container runs and pass that as path(db)?

process {
    withName: GTDBTK_CLASSIFYWF {
        containerOptions = "-B $params.gtdb_db:/some/absolute/path:image-src=/release220"
    }
}

and call the process

    GTDBTK_CLASSIFYWF( [ 'gtdb', `/some/absolute/path` ] )

Yes! That does work. See notes on using SqashFS as GTDB-Tk database in docs/usage.md.

I used nf-core tools in docker container. $ nf-core modules update -f gtdbtk/classifywf I was user 'root' in container, so I had to change ownership and permissions of modified files afterward outside the container.

…base.

jfy133

Hi @muniheart ! Thanks for persisting with your experimentation on this one!

I've done a quick read through, but I will do a pass to update the text tomorrow (I think it's a bit too unnecessarily technical at points)

But so if I understand correctly in very simple terms:

No code changes are necessary
You generate your squashfs image
You need to make an 'empty' directory (so to say) somewhere on your filesystem (and you give this to --gtdb_db)
You then pass to the container options the squashfs image to then mount at the location of --gtdb_db (and the name of the top level directory)

Is that correct?

muniheart · 2025-04-24T22:35:48Z

Hi @muniheart ! Thanks for persisting with your experimentation on this one!

I've done a quick read through, but I will do a pass to update the text tomorrow (I think it's a bit too unnecessarily technical at points)

But so if I understand correctly in very simple terms:
* No code changes are necessary

* You generate your squashfs image

* You need to make an 'empty' directory (so to say) somewhere on your filesystem (and you give this to `--gtdb_db`)

* You then pass to the container options the squashfs image to then mount at the location of `--gtdb_db`  (and the name of the top level directory)
Is that correct?

Hi @jfy133. That's it! The top-level-directory bit can even be dropped if you like. It would be resolved by find.

 process {
      withName: GTDBTK_CLASSIFYWF {
              containerOptions = "-B /path/to/gtdb.squashfs:${params.gtdb_db}:image-src=/"
      }
  }

Thanks for your patience in considering this 'enhancement'.

jfy133 · 2025-04-25T08:06:07Z

@muniheart please have a look and check this looks OK - we can always roll back or take bits from your pverious commit if necessary

jfy133 · 2025-04-25T08:09:46Z

@nf-core-bot fix linting

muniheart · 2025-04-25T09:45:38Z

+```nextflow
+process {
+    withName: GTDBTK_CLASSIFYWF {
+            containerOptions = "-B /<path>/<to>/<empty_dir>/gtdbtk_r220.squashfs:${params.gtdb_db}:image-src=/<output_from_unsquashfs_ls>"
+    }
+}
+```


It looks good, @jfy133 . I have 2 comments:

The /<path>/<to>/<empty_dir>/ should be passed in params.gtdb_db. That is mentioned above.

It may look cleaner as

process { withName: GTDBTK_CLASSIFYWF { containerOptions = "-B <path_to_image>:${params.gtdb_db}:image-src=/" } }

Using <path_to_image> may be closer to the convention used elsewhere in the doc.

I prefer to use /<path>/<to> where possible as it's a more explicit example (and tells the user what to expect). In the case of <output_from_unsquas> bit I can't predict what this will look like so prefer to make it more generic, if that makes sense?

There is still a problem with arguments to -B, @jfy133.

It should be,

process { withName: GTDBTK_CLASSIFYWF { containerOptions = "-B /<path>/<to>/gtdbtk_r220.squashfs:${params.gtdb_db}:image-src=/" } }

The value /<path>/<to>/<empty_dir> is passed in params.gtdb_db.

Ah got, sorry my mistake!

Co-authored-by: muniheart <52059779+muniheart@users.noreply.github.com>

jfy133 · 2025-04-25T12:57:06Z

How about now @muniheart ?

muniheart · 2025-04-25T17:14:27Z

How about now @muniheart ?

Hi @jfy133. I suggested a change above.

jfy133 · 2025-04-28T07:12:56Z

+```nextflow
+process {
+    withName: GTDBTK_CLASSIFYWF {
+            containerOptions = "-B /<path>/<to>/<empty_dir>/gtdbtk_r220.squashfs:${params.gtdb_db}:image-src=/<output_from_unsquashfs_ls>"
+    }
+}
+```


Ah got, sorry my mistake!

jfy133 · 2025-04-28T07:14:43Z

Hopefully that's everything now @muniheart ?

jfy133 · 2025-04-28T07:52:10Z

@nf-core-bot fix linting

jfy133 · 2025-04-28T09:15:00Z

Ignore the failing tests for now @muniheart we can merge in if you're happy with the instructions :)

muniheart · 2025-04-28T11:38:28Z

LGTM, @jfy133. Thank you.

prototaxites · 2025-04-28T12:41:57Z

Hi @muniheart, no comments but just wanted to say thank you for persevering with this, despite the difficulties finding the best way to incorporate it! I think this makes a really nice addition to the documentation, and will definitely be helpful to other users of the pipeline.

jfy133 · 2025-04-28T12:50:52Z

I +1 that too, thank you @muniheart !

jfy133 · 2025-04-28T13:01:02Z

 ### `Added`

 - [#784](https://github.com/nf-core/mag/pull/784) - Added `--bin_min_size` and `--bin_max_size` parameters to filter out bins based on size (requested by @maxibor, @alexhbnr, added by @jfy133, @prototaxites).
+- [#793](https://github.com/nf-core/mag/pull/793) - Document use of a SquashFS image with `--gtdb_db` (by @muniheart).


praise:

Suggested change

- [#793](https://github.com/nf-core/mag/pull/793) - Document use of a SquashFS image with `--gtdb_db` (by @muniheart).

- [#793](https://github.com/nf-core/mag/pull/793) - Document use of a SquashFS image with `--gtdb_db`, useful for limited inode infrastructure (by @muniheart).

Document use of SquashFS image as gtdbtk database.

Updated nf-core module gtdbtk/classifywf.

897290b

I used nf-core tools in docker container. $ nf-core modules update -f gtdbtk/classifywf I was user 'root' in container, so I had to change ownership and permissions of modified files afterward outside the container.

muniheart changed the title ~~Allow squashfs image as gtdbtk database.~~ Document use of SquashFS image as gtdbtk database. Apr 20, 2025

Added: Added documentation on usage of SquashFS image as GTDB-Tk data…

6c7c6fc

…base.

muniheart force-pushed the dev branch from 906be36 to 6c7c6fc Compare April 20, 2025 19:14

jfy133 reviewed Apr 24, 2025

View reviewed changes

Simplify and rephrase

132a8a8

[automated] Fix code linting

6403aa0

muniheart commented Apr 25, 2025

View reviewed changes

Comment thread docs/usage.md Outdated

Update docs/usage.md

f6c5c10

Co-authored-by: muniheart <52059779+muniheart@users.noreply.github.com>

jfy133 reviewed Apr 25, 2025

View reviewed changes

Comment thread docs/usage.md

Update docs/usage.md

73559d6

jfy133 reviewed Apr 28, 2025

View reviewed changes

Apply suggestions from code review

d0790e0

[automated] Fix code linting

b950db3

jfy133 approved these changes Apr 28, 2025

View reviewed changes

Comment thread CHANGELOG.md Outdated

Update CHANGELOG.md

f4ca57c

jfy133 reviewed Apr 28, 2025

View reviewed changes

Comment thread CHANGELOG.md Outdated

jfy133 added 2 commits April 28, 2025 15:01

Update CHANGELOG.md

f12f79b

Update CHANGELOG.md

6b04b2e

jfy133 merged commit fd0f79c into nf-core:dev Apr 28, 2025

muabnezor pushed a commit to muabnezor/mag that referenced this pull request Apr 28, 2025

Merge pull request nf-core#793 from muniheart/dev

2fcfb81

Document use of SquashFS image as gtdbtk database.

prototaxites mentioned this pull request Nov 5, 2025

[FEATURE] Not use export GTDBTK_DATA_PATH="\$(find -L ${db} -name 'metadata' -type d -exec dirname {} \\;)" nf-core/modules#9310

Closed

	- [#793](https://github.com/nf-core/mag/pull/793) - Document use of a SquashFS image with `--gtdb_db` (by @muniheart).
	- [#793](https://github.com/nf-core/mag/pull/793) - Document use of a SquashFS image with `--gtdb_db`, useful for limited inode infrastructure (by @muniheart).

Conversation

muniheart commented Apr 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR checklist

Uh oh!

jfy133 left a comment

Choose a reason for hiding this comment

Uh oh!

muniheart commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jfy133 commented Apr 25, 2025

Uh oh!

jfy133 commented Apr 25, 2025

Uh oh!

muniheart Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jfy133 Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

muniheart Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

jfy133 Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jfy133 commented Apr 25, 2025

Uh oh!

muniheart commented Apr 25, 2025

Uh oh!

jfy133 Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jfy133 commented Apr 28, 2025

Uh oh!

jfy133 commented Apr 28, 2025

Uh oh!

jfy133 commented Apr 28, 2025

Uh oh!

muniheart commented Apr 28, 2025

Uh oh!

Uh oh!

prototaxites commented Apr 28, 2025

Uh oh!

jfy133 commented Apr 28, 2025

Uh oh!

jfy133 Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

muniheart commented Apr 18, 2025 •

edited

Loading

muniheart commented Apr 24, 2025 •

edited

Loading

muniheart Apr 25, 2025 •

edited

Loading