pype.process

class pype.process.DockerConfig(exec_path, runtime, cache_dir=None)

Bases: object

Configuration for Docker and container runtime environments.

This class manages container runtime configuration and provides properties to identify the runtime type (Docker, Singularity, uDocker).

Parameters:

• exec_path (Path)

• runtime (str)

• cache_dir (Path | None)

exec_path

Path to the container runtime executable

Type:

pathlib.Path

runtime

Name of the container runtime (‘docker’, ‘singularity’, ‘udocker’)

Type:

str

cache_dir

Optional cache directory, required for Singularity containers

Type:

pathlib.Path | None

exec_path: Path

runtime: str

cache_dir: Path | None = None

property is_singularity: bool

Check if runtime is Singularity.

property is_udocker: bool

Check if runtime is uDocker.

__init__(exec_path, runtime, cache_dir=None)

Parameters:

• exec_path (Path)

• runtime (str)

• cache_dir (Path | None)

Return type:

None

pype.process.get_module_cmd()

Get the environment modules command interface.

pype.process.program_string(program_dict)

Format program string from dictionary.

Parameters:

program_dict (Dict[str, str])

Return type:

str

class pype.process.Volume(path, output=False, bind_prefix='/var/lib/pype')

Bases: object

Volume binding configuration for containerized environments.

Handles the mounting of files and directories between host and container environments, managing read-only and read-write access modes.

Parameters:

• path (str)

• output (bool)

• bind_prefix (str)

path

Path to the file or directory on the host system

Type:

str

output

If True, mount as read-write; if False, mount as read-only

Type:

bool

bind_prefix

Base path in the container where volumes will be mounted

Type:

str

path: str

output: bool = False

bind_prefix: str = '/var/lib/pype'

remove_mode()

Remove mode specifier for udocker compatibility.

Return type:

None

singularity_volume()

Format for Singularity binding syntax.

Return type:

None

replace_bind_volume(bind_path)

Replaces the bind volume in the container environment with the specified bind path.

This is useful to manage binding point to multiple paths (defined in multiple Volume classes) that are subfolders of another bind volume in the host system.

Parameters:

bind_path (str) – Binding point to replace instead of the current one randomly generated by the class.

replace_bind_dirname(bind_path)

Replaces the bind volume in the container environment with the dirname of the specified bind path.

This is useful to give the same binding point to multiple paths (defined in multiple Volume classes) that are in the same folder in the host system.

Parameters:

bind_path (str) – Binding point to replace instead of the current one randomly generated by the class.

to_str()

Get formatted volume string.

Return type:

str

__init__(path, output=False, bind_prefix='/var/lib/pype')

Parameters:

• path (str)

• output (bool)

• bind_prefix (str)

Return type:

None

class pype.process.Command(cmd, log, profile, name='')

Bases: object

High-level interface for executing commands with container support.

This class provides a unified interface for running commands locally or within containers (Docker, Singularity, uDocker). It handles: - Command execution and piping - Volume mounting and file access - Container runtime configuration - Environment modules integration - Input/output file tracking

Parameters:

• cmd (str)

• log (Any)

• profile (Any)

• name (str)

cmd

Command to execute as list of arguments

name

Identifier for the command (used in logs)

profile

Configuration profile for the environment

log

Logger instance for command execution tracking

docker

Container runtime configuration

inputs

List of input files required by the command

outputs

List of output files produced by the command

volumes

List of volumes to mount in container environments

__init__(cmd, log, profile, name='')

Initialize command execution environment.

Parameters:

• cmd (str) – Command string to execute

• log (Any) – Logger instance for command output

• profile (Any) – Environment profile configuration

• name (str) – Optional identifier for the command

add_output(out_file, harvest=True)

Add output file to track.

Parameters:

• out_file (str)

• harvest (bool)

Return type:

None

add_input(in_file, match='exact')

Add input file(s) to track.

Parameters:

• in_file (str)

• match (str)

Return type:

None

add_volume(path, output=False)

Add volume to track.

Parameters:

• path (str)

• output (bool)

Return type:

None

normalize_volumes()

Normalize volumes to avoid duplicates for bind mounts.

Returns:

List of unique volumes for bind mounts (deduplicated). Note: self.volumes is kept intact for path substitution.

Return type:

list

add_namespace(program)

Parameters:

program (ProfileProgram)

Return type:

None

run(local_script=False, capture_stderr=False)

Execute the command.

Parameters:

• local_script (bool) – If True, execute as local script

• capture_stderr (bool) – If True, capture stderr output (default False to maintain current behavior)

Return type:

None

docker_run(local_script)

Execute command within a container environment.

Handles the complexities of running commands in containers including: - Volume mounting and path translation - User permissions - Working directory setup - Container-specific command modifications

Parameters:

local_script (bool) – If True, indicates the command is a local script that needs to be mounted in the container

Return type:

None

local_run(local_script=False, capture_stderr=False)

Execute command locally.

Parameters:

• local_script (bool) – If True, the last element of self.cmd is a script file whose content is rewritten for overlay path substitution.

• capture_stderr (bool) – If True, capture stderr output (default False to maintain current behavior)

Return type:

None

module_load()

load env_modules dependencies

Return type:

None

module_run(local_script=False, capture_stderr=False)

Execute command with environment modules.

Parameters:

• local_script (bool) – Passed through to local_run for overlay script rewriting.

• capture_stderr (bool) – If True, capture stderr output

Return type:

None

conda_run(local_script=False, capture_stderr=False)

Execute command in conda environment.

Supports both name-based and path-based conda environments: - If program.path is set: uses -p <path>/<name> - Otherwise: uses -n <name>

Parameters:

• local_script (bool) – Passed through to local_run for overlay script rewriting.

• capture_stderr (bool) – If True, capture stderr output

Return type:

None

pipe_in(command, local_script=False)

Pipe input from another command.

Parameters:

• command (Command)

• local_script (bool)

Return type:

None

release_stdout()

Release stdout to terminal (for final commands in execution).

Use this for terminal commands where output should be visible to the user.

Return type:

None

capture_stdout()

Capture stdout for piping (for intermediate commands).

Use this to ensure stdout is captured in subprocess.PIPE for piping to another command.

Return type:

None

child_close()

Close child process.

Return type:

None

close(ignore_returncode=False)

Close command and return result with exit code.

Parameters:

ignore_returncode (bool) – If False (default), raise exception on non-zero exit. If True, ignore exit code (legacy behavior).

Returns:

Tuple of (communicate_result, returncode)

Raises:

CommandError – If returncode != 0 and ignore_returncode is False

Return type:

Tuple[Any, int]

replace_values_in_code(code_file)

Replace path mappings in a script file.

Uses a single-pass regex alternation so substituted text is never re-matched (prevents double-substitution when a bind_file contains another overlay path as a prefix). Docker volumes take precedence over overlay volumes when the same path appears in both.

Parameters:

code_file (str)

Return type:

None

pype.process.is_direct_child_of(x, y)

Check if x is a direct child of directory y.

Parameters:

• x (str)

• y (str)

Return type:

bool

pype.process.has_same_basedir(x, y)

Check if x and y have the same base directory.

Parameters:

• x (str)

• y (str)

Return type:

bool

class pype.process.Namespace(program, log, profile)

Bases: object

Environment and dependency management system.

Manages different execution environments (path-based, environment modules, containers) and their dependencies. Supports: - Path-based program execution - Environment modules loading - Container image specification - Dependency resolution

The namespace format is ‘type@item’ where: - type: One of ‘path’, ‘env_module’, or ‘docker’ - item: The specific program/container/module to use

For example: - ‘docker@ubuntu:latest’ - ‘env_module@gcc/9.3.0’ - ‘path@/usr/local/bin/python’

Parameters:

• program (ProfileProgram)

• log (Any)

• profile (Any)

__init__(program, log, profile)

Initialize namespace configuration from a ProfileProgram.

Parameters:

• program (ProfileProgram) – ProfileProgram object (always — dict support removed)

• log (Any) – Logger instance

• profile (Any) – Environment profile containing program configurations

Attributes set:

name: Profile dict key — unique identifier for this program type: Namespace type (‘docker’, ‘env_module’, ‘conda’, or ‘path’) namespace: Original namespace string (e.g., ‘docker@sequenza/sequenza’) str: Parsed namespace item (e.g., ‘sequenza/sequenza’) version: Program version string list: List of resolved namespace items docker_extra_args: Extra arguments for container runtimes

Raises:

CommandNamespaceError – For invalid namespace format or unsupported types

Parameters:

• program (ProfileProgram)

• log (Any)

• profile (Any)

first()

Return type:

str

last()

Return type:

str