miniwdl

Workflow Description Language static analysis toolkit for Python 3.6+

miniwdl is a library for parsing WDL documents into a type-checked abstract syntax tree (AST), providing a foundation for new runtime systems, developer tooling, and language experimentation. It also includes a command-line tool which validates WDL documents and generates lint/style warnings.

This project in prealpha development; interfaces are liable to change substantially.

pip3 install miniwdl

This will also install the Lark parsing library automatically.

For development, git clone --recursive this repository and docker build -t miniwdl . to run miniwdl's test suite in a locally-built docker container. The Dockerfile illustrates how to configure another environment.

Once installed, miniwdl check /path/to/workflow.wdl loads the WDL document and shows a brief outline with any lint warnings. Add --path /path/to/tasks/ with a directory to search for imported documents (one or more times). Example with HumanCellAtlas/skylab:

$ git clone https://github.com/HumanCellAtlas/skylab.git
$ miniwdl check --path skylab/library/tasks/ \
    skylab/pipelines/smartseq2_single_sample/SmartSeq2SingleSample.wdl 

SmartSeq2SingleSample.wdl
    workflow SmartSeq2SingleCell
        (Ln 14, Col 3) UnusedDeclaration, nothing references File gtf_file
        call HISAT2.HISAT2PairedEnd
        call Picard.CollectMultipleMetrics
        call Picard.CollectRnaMetrics
        call Picard.CollectDuplicationMetrics
        call HISAT2.HISAT2RSEM as HISAT2Transcriptome
        call RSEM.RSEMExpression
        call GroupQCs.GroupQCOutputs
        call ZarrUtils.SmartSeq2ZarrConversion
    GroupQCs : GroupMetricsOutputs.wdl
        task GroupQCOutputs
            (Ln 10, Col 3) StringCoercion, String mem = :Int:
            (Ln 11, Col 3) StringCoercion, String cpu = :Int:
            (Ln 12, Col 3) StringCoercion, String disk_space = :Int:
    HISAT2 : HISAT2.wdl
        task HISAT2PairedEnd
        task HISAT2RSEM
        task HISAT2InspectIndex (not called)
        task HISAT2SingleEnd (not called)

    ...

If you haven't installed the PyPI package to get the miniwdl entry point, equivalently python3 -m /path/to/miniwdl/WDL check ....

The WDL package provides programmatic access to the WDL parser and AST. The following example prints all declarations in a workflow, descending into scatter and if stanzas as needed.

$ python3 -c "
import WDL

doc = WDL.load('skylab/pipelines/optimus/Optimus.wdl',
               path=['skylab/library/tasks/'])

def show(elements):
  for elt in elements:
    if isinstance(elt, WDL.Decl):
      print(str(elt.type) + ' ' + elt.name)
    elif isinstance(elt, WDL.Scatter) or isinstance(elt, WDL.Conditional):
      show(elt.elements)
show(doc.workflow.elements)
"

String version
Array[File] r1_fastq
Array[File] r2_fastq
Array[File] i1_fastq
String sample_id
File tar_star_reference
File annotations_gtf
File ref_genome_fasta
File whitelist
String fastq_suffix
Array[Int] indices
Array[File] non_optional_i1_fastq
File barcoded_bam

Online Python developer documentation for the WDL package:

(Read the Docs currently builds from the mlin/miniwdl fork of this repository.)

Locally, make doc triggers Sphinx to generate the docs under docs/_build/html/. Or, after building the docker image, copy them out with docker run --rm -v ~/Desktop:/io miniwdl cp -r /miniwdl/docs/_build/html /io/miniwdl_docs.

Feedback and contributions are welcome on this repository. Please:

1. Send pull requests from a dedicated branch without unrelated edits
2. Add appropriate tests to the automatic suite
3. Use make pretty to reformat the code with black
4. Ensure compatibility with this project's MIT license

The Project board is our up-to-date tracker.