Batch Arguments#

This guide covers what is common to every batch step, regardless of where its items come from: how a batch step expands into iterations, how to choose between the two batch argument types, and how to control the dependencies between batch steps with batch_align.

For the details of each source type, see batch_file_arguments (external sample sheets) and Batch List Arguments and Dynamic Batch Expansion (inline or dynamically generated items).

Batch Steps#

A step declared as batch_snippet or batch_pipeline runs its target once per batch item instead of once in total. The items come from exactly one batch argument in the step’s arguments block:

step_align:
  name: gatk_bwa_mem
  type: batch_snippet
  arguments:
    input_batch:
      value: "%(fastq_batch)s"
      type: batch_file_arg      # or batch_list_arg
    -r: "%(reference)s"

Whatever the source, the batch argument is resolved to the same internal shape, an ordered list of argument dictionaries, one per iteration:

[{"--f1": "lane1_R1", "--f2": "lane1_R2", "--out": "lane1.bam"},
 {"--f1": "lane2_R1", "--f2": "lane2_R2", "--out": "lane2.bam"}]

Non-batch arguments on the same step (-r above) are passed unchanged to every iteration. Each iteration is submitted to the queue as its own independent job.

Choosing a Batch Argument Type#

Feature	`batch_file_arg`	`batch_list_arg`
Source	External TSV file	Inline YAML or snippet results
Item count	Fixed (rows in the file)	Dynamic (from `composite_arg`)
Known at	Submission time	Runtime
Use case	Pre-defined sample sheets	Split-process-merge patterns
Item shaping	`columns` (filter/rename)	Zipping lists and scalars

Use batch_file_arg when the items are described by an external file a user maintains. Use batch_list_arg when the items are written into the pipeline itself, or when their number is only known after an upstream step has run.

Per-Iteration Dependencies with `batch_align`#

By default, when a batch step depends on another batch step, every iteration of the consumer waits for all iterations of the producer. That is often what you want, since a multiqc step really does need every sample’s QC output, but it needlessly serializes pipelines where the branches are independent.

Consider a per-donor pipeline that basecalls several tumor samples and then runs a per-sample copy-number step. Without batch_align the slowest basecaller holds up every downstream sample:

wakhan_hapcorrect_TUMOR2  <- dorado_TUMOR, dorado_TUMOR2, dorado_TUMOR3, dorado_TUMOR4

The optional step-level batch_align attribute narrows each iteration to just the producer iteration it actually consumes. It applies to any batch step, so it works the same for batch_file_arg and batch_list_arg items, because it matches on the resolved iterations rather than on where they came from. It is valid only on a batch_snippet or batch_pipeline, and every dependency it names must also appear in depends_on.

Dict form: match on a batch key#

This is the recommended form:

step_wakhan_hapcorrect:
  name: wakhan_hapcorrect
  type: batch_snippet
  depends_on:
    - step_pomfret                  # not aligned: every iteration waits for it
    - step_ont_tumors.step_dorado
  batch_align:
    step_ont_tumors.step_dorado: sample_name
  arguments:
    --input_batch:
      type: batch_list_arg
      value:
        --tumor_bam: "%(output_cram_tumor)s"
        --sample_name: "%(sample_name_tumor)s"
        --output_dir: "%(wakham_dir)s"

Iteration i of this step now depends only on the producer iteration whose --sample_name has the same value, giving:

wakhan_hapcorrect_TUMOR2  <- dorado_TUMOR2, pomfret_NORMAL

The key is written without dashes and must name a batch argument present on both sides. Short (-i) and long (--sample_name) flags are both resolved. With a batch_file_arg producer that means a column of the sample sheet; with columns renaming in play, use the renamed key, since matching happens after the batch items are resolved. Multiple matching producer iterations are unioned, so a consumer iteration may legitimately depend on a sub-batch.

The dependency may be a dotted reference into a batch_pipeline’s sub-step (step_ont_tumors.step_dorado above); alignment then pairs each of the consumer’s iterations with that sub-step’s job from the matching sub-pipeline run.

List form: positional alignment#

batch_align:
  - step_ont_tumors.step_dorado

Iteration i pairs with producer iteration i. Simpler, but it only makes sense when both batches are built from the same ordered source: the same rows of one batch file, or lists zipped in the same order. This is also the form that maps directly onto scheduler array-job semantics (SLURM --dependency=aftercorr, LSF element-wise done(job[*])).

Omitting batch_align keeps the legacy fan-in, unchanged. Nothing about existing pipelines changes unless the attribute is added.

Validation#

Both forms fail loudly rather than silently mis-wiring the DAG. At runtime, a positional alignment over batches of different lengths, or a keyed alignment that finds no matching producer iteration, raises a PipelineItemError naming the step, the dependency and the key:

Error in batch_snippet 'wakhan_hapcorrect': batch_align on
'step_ont_tumors.step_dorado' found no iteration with --sample_name 'TUMOR5'

Structural mistakes are caught earlier, at parse time, as a PipelineDependencyError, so pype validate and the LSP report them before anything is submitted:

batch_align on a step that is not a batch_snippet/batch_pipeline
a dependency listed in batch_align but missing from depends_on
aligning on a producer step that is not itself a batch step

Note that a partial key mismatch behaves like the analogous case for columns: only a complete absence of matching iterations is reported. If one consumer iteration matches and another does not, the failing iteration is the one that raises.