.. index:: Batch Arguments, Batch Steps, Batch Dependencies, batch_align

.. _batch_arguments:

Batch Arguments
===============

This guide covers what is common to every batch step, regardless of where its
items come from: how a batch step expands into iterations, how to choose
between the two batch argument types, and how to control the dependencies
between batch steps with ``batch_align``.

For the details of each source type, see :ref:`batch_file_arguments` (external
sample sheets) and :ref:`batch_list_arguments` (inline or dynamically generated
items).

Batch Steps
-----------

A step declared as ``batch_snippet`` or ``batch_pipeline`` runs its target once
per **batch item** instead of once in total. The items come from exactly one
batch argument in the step's ``arguments`` block:

.. code-block:: yaml

    step_align:
      name: gatk_bwa_mem
      type: batch_snippet
      arguments:
        input_batch:
          value: "%(fastq_batch)s"
          type: batch_file_arg      # or batch_list_arg
        -r: "%(reference)s"

Whatever the source, the batch argument is resolved to the same internal shape,
an ordered list of argument dictionaries, one per iteration:

.. code-block:: python

    [{"--f1": "lane1_R1", "--f2": "lane1_R2", "--out": "lane1.bam"},
     {"--f1": "lane2_R1", "--f2": "lane2_R2", "--out": "lane2.bam"}]

Non-batch arguments on the same step (``-r`` above) are passed unchanged to
every iteration. Each iteration is submitted to the queue as its own
independent job.

Choosing a Batch Argument Type
------------------------------

+------------------------+----------------------------------+----------------------------------+
| Feature                | ``batch_file_arg``               | ``batch_list_arg``               |
+========================+==================================+==================================+
| Source                 | External TSV file                | Inline YAML or snippet results   |
+------------------------+----------------------------------+----------------------------------+
| Item count             | Fixed (rows in the file)         | Dynamic (from ``composite_arg``) |
+------------------------+----------------------------------+----------------------------------+
| Known at               | Submission time                  | Runtime                          |
+------------------------+----------------------------------+----------------------------------+
| Use case               | Pre-defined sample sheets        | Split-process-merge patterns     |
+------------------------+----------------------------------+----------------------------------+
| Item shaping           | ``columns`` (filter/rename)      | Zipping lists and scalars        |
+------------------------+----------------------------------+----------------------------------+

Use ``batch_file_arg`` when the items are described by an external file a user
maintains. Use ``batch_list_arg`` when the items are written into the pipeline
itself, or when their number is only known after an upstream step has run.

.. _batch_align:

Per-Iteration Dependencies with ``batch_align``
-----------------------------------------------

By default, when a batch step depends on another batch step, **every** iteration
of the consumer waits for **all** iterations of the producer. That is often what
you want, since a ``multiqc`` step really does need every sample's QC output,
but it needlessly serializes pipelines where the branches are independent.

Consider a per-donor pipeline that basecalls several tumor samples and then runs
a per-sample copy-number step. Without ``batch_align`` the slowest basecaller
holds up every downstream sample:

.. code-block:: text

    wakhan_hapcorrect_TUMOR2  <- dorado_TUMOR, dorado_TUMOR2, dorado_TUMOR3, dorado_TUMOR4

The optional step-level ``batch_align`` attribute narrows each iteration to just
the producer iteration it actually consumes. It applies to any batch step, so it
works the same for ``batch_file_arg`` and ``batch_list_arg`` items, because it
matches on the *resolved* iterations rather than on where they came from. It is
valid only on a ``batch_snippet`` or ``batch_pipeline``, and every dependency it
names must also appear in ``depends_on``.

Dict form: match on a batch key
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is the recommended form:

.. code-block:: yaml

    step_wakhan_hapcorrect:
      name: wakhan_hapcorrect
      type: batch_snippet
      depends_on:
        - step_pomfret                  # not aligned: every iteration waits for it
        - step_ont_tumors.step_dorado
      batch_align:
        step_ont_tumors.step_dorado: sample_name
      arguments:
        --input_batch:
          type: batch_list_arg
          value:
            --tumor_bam: "%(output_cram_tumor)s"
            --sample_name: "%(sample_name_tumor)s"
            --output_dir: "%(wakham_dir)s"

Iteration *i* of this step now depends only on the producer iteration whose
``--sample_name`` has the same value, giving:

.. code-block:: text

    wakhan_hapcorrect_TUMOR2  <- dorado_TUMOR2, pomfret_NORMAL

The key is written without dashes and must name a batch argument present on
**both** sides. Short (``-i``) and long (``--sample_name``) flags are both
resolved. With a ``batch_file_arg`` producer that means a column of the sample
sheet; with ``columns`` renaming in play, use the *renamed* key, since matching
happens after the batch items are resolved. Multiple matching producer
iterations are unioned, so a consumer iteration may legitimately depend on a
sub-batch.

The dependency may be a dotted reference into a ``batch_pipeline``'s sub-step
(``step_ont_tumors.step_dorado`` above); alignment then pairs each of the
consumer's iterations with that sub-step's job from the matching sub-pipeline
run.

List form: positional alignment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: yaml

      batch_align:
        - step_ont_tumors.step_dorado

Iteration *i* pairs with producer iteration *i*. Simpler, but it only makes
sense when both batches are built from the same ordered source: the same rows
of one batch file, or lists zipped in the same order. This is also the form that
maps directly onto scheduler array-job semantics (SLURM
``--dependency=aftercorr``, LSF element-wise ``done(job[*])``).

**Omitting** ``batch_align`` keeps the legacy fan-in, unchanged. Nothing about
existing pipelines changes unless the attribute is added.

Validation
~~~~~~~~~~

Both forms fail loudly rather than silently mis-wiring the DAG. At runtime, a
positional alignment over batches of different lengths, or a keyed alignment
that finds no matching producer iteration, raises a ``PipelineItemError`` naming
the step, the dependency and the key:

.. code-block:: text

    Error in batch_snippet 'wakhan_hapcorrect': batch_align on
    'step_ont_tumors.step_dorado' found no iteration with --sample_name 'TUMOR5'

Structural mistakes are caught earlier, at parse time, as a
``PipelineDependencyError``, so ``pype validate`` and the LSP report them before
anything is submitted:

- ``batch_align`` on a step that is not a ``batch_snippet``/``batch_pipeline``
- a dependency listed in ``batch_align`` but missing from ``depends_on``
- aligning on a producer step that is not itself a batch step

Note that a *partial* key mismatch behaves like the analogous case for
``columns``: only a complete absence of matching iterations is reported. If one
consumer iteration matches and another does not, the failing iteration is the
one that raises.

Interaction with final_outputs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Because aligned iterations have their own dependencies rather than a shared
barrier, a partly finished batch has a mix of iterations that are done and
iterations whose producer still has to run. A batch step is therefore only
treated as complete, and its feeders only cancelled, once **every** iteration
has its final outputs. See :ref:`final_outputs`.

See Also
--------

- :ref:`batch_file_arguments` for batch items sourced from an external file,
  including column filtering and renaming with ``columns``
- :ref:`batch_list_arguments` for batch items defined inline or generated
  dynamically from upstream results
- :ref:`composite_arguments` for wiring one step's results into another's
  arguments
- :ref:`pipelines` for general pipeline documentation