pype.process#

class pype.process.DockerConfig(exec_path, runtime, cache_dir=None)[source]#

Bases: object

Configuration for Docker and container runtime environments.

This class manages container runtime configuration and provides properties to identify the runtime type (Docker, Singularity, uDocker).

Parameters:

exec_path (Path)
runtime (str)
cache_dir (Path | None)

exec_path#

Path to the container runtime executable

Type:: pathlib.Path

runtime#

Name of the container runtime (‘docker’, ‘singularity’, ‘udocker’)

Type:: str

cache_dir#

Optional cache directory, required for Singularity containers

Type:: pathlib.Path | None

exec_path: Path#

runtime: str#

cache_dir: Path | None = None#

property is_singularity: bool#: Check if runtime is Singularity.

property is_udocker: bool#: Check if runtime is uDocker.

__init__(exec_path, runtime, cache_dir=None)#

Parameters:

exec_path (Path)
runtime (str)
cache_dir (Path | None)

Return type:

None

pype.process.get_module_cmd()[source]#: Get the environment modules command interface.

pype.process.program_string(program_dict)[source]#

Format program string from dictionary.

Parameters:: program_dict (Dict[str, str])
Return type:: str

class pype.process.OverlayVolume(path, original_dir, scratch_dir, bind_file, harvest=True)[source]#

Bases: object

Overlay path mapping for an output registered via add_output().

Mirrors the role of Volume for docker (path → bind_file substitution and harvest tracking), but lives in self.overlay_volumes rather than self.volumes so it never interferes with container-specific logic.

Parameters:

path (str)
original_dir (str)
scratch_dir (str)
bind_file (str)
harvest (bool)

path#

original declared output path

Type:: str

original_dir#

parent dir of path (same root-level logic as Volume.to_bind)

Type:: str

scratch_dir#

scratch dir that maps to original_dir (harvest source)

Type:: str

bind_file#

scratch equivalent of path (substitution target in scripts/cmd)

Type:: str

path: str#

original_dir: str#

scratch_dir: str#

bind_file: str#

harvest: bool = True#

__init__(path, original_dir, scratch_dir, bind_file, harvest=True)#

Parameters:

path (str)
original_dir (str)
scratch_dir (str)
bind_file (str)
harvest (bool)

Return type:

None

class pype.process.Volume(path, output=False, bind_prefix='/var/lib/pype')[source]#

Bases: object

Volume binding configuration for containerized environments.

Handles the mounting of files and directories between host and container environments, managing read-only and read-write access modes.

Parameters:

path (str)
output (bool)
bind_prefix (str)

path#

Path to the file or directory on the host system

Type:: str

output#

If True, mount as read-write; if False, mount as read-only

Type:: bool

bind_prefix#

Base path in the container where volumes will be mounted

Type:: str

path: str#

output: bool = False#

bind_prefix: str = '/var/lib/pype'#

remove_mode()[source]#

Remove mode specifier for udocker compatibility.

Return type:: None

singularity_volume()[source]#

Format for Singularity binding syntax.

Return type:: None

replace_bind_volume(bind_path)[source]#

Replaces the bind volume in the container environment with the specified bind path.

This is useful to manage binding point to multiple paths (defined in multiple Volume classes) that are subfolders of another bind volume in the host system.

Parameters:: bind_path (str) – Binding point to replace instead of the current one randomly generated by the class.

replace_bind_dirname(bind_path)[source]#

Replaces the bind volume in the container environment with the dirname of the specified bind path.

This is useful to give the same binding point to multiple paths (defined in multiple Volume classes) that are in the same folder in the host system.

Parameters:: bind_path (str) – Binding point to replace instead of the current one randomly generated by the class.

to_str()[source]#

Get formatted volume string.

Return type:: str

__init__(path, output=False, bind_prefix='/var/lib/pype')#

Parameters:

path (str)
output (bool)
bind_prefix (str)

Return type:

None

class pype.process.Command(cmd, log, profile, name='')[source]#

Bases: object

High-level interface for executing commands with container support.

This class provides a unified interface for running commands locally or within containers (Docker, Singularity, uDocker). It handles: - Command execution and piping - Volume mounting and file access - Container runtime configuration - Environment modules integration - Input/output file tracking

Parameters:

cmd (str)
log (Any)
profile (Any)
name (str)

cmd#: Command to execute as list of arguments

name#: Identifier for the command (used in logs)

profile#: Configuration profile for the environment

log#: Logger instance for command execution tracking

docker#: Container runtime configuration

inputs#: List of input files required by the command

outputs#: List of output files produced by the command

volumes#: List of volumes to mount in container environments

__init__(cmd, log, profile, name='')[source]#

Initialize command execution environment.

Parameters:

cmd (str) – Command string to execute
log (Any) – Logger instance for command output
profile (Any) – Environment profile configuration
name (str) – Optional identifier for the command

add_output(out_file, harvest=True)[source]#

Add output file to track.

Parameters:

out_file (str)
harvest (bool)

Return type:

None

add_input(in_file, match='exact')[source]#

Add input file(s) to track.

Parameters:

in_file (str)
match (str)

Return type:

None

add_volume(path, output=False)[source]#

Add volume to track.

Parameters:

path (str)
output (bool)

Return type:

None

normalize_volumes()[source]#

Normalize volumes to avoid duplicates for bind mounts.

Returns:: List of unique volumes for bind mounts (deduplicated). Note: self.volumes is kept intact for path substitution.
Return type:: list

add_namespace(namespace)[source]#

Add namespace to command.

Parameters:: namespace (Dict[str, Any] | Any) – Program configuration as dict or ProfileProgram object
Return type:: None

run(local_script=False, capture_stderr=False)[source]#

Execute the command.

Parameters:

local_script (bool) – If True, execute as local script
capture_stderr (bool) – If True, capture stderr output (default False to maintain current behavior)

Return type:

None

docker_run(local_script)[source]#

Execute command within a container environment.

Handles the complexities of running commands in containers including: - Volume mounting and path translation - User permissions - Working directory setup - Container-specific command modifications

Parameters:: local_script (bool) – If True, indicates the command is a local script that needs to be mounted in the container
Return type:: None

local_run(local_script=False, capture_stderr=False)[source]#

Execute command locally.

Parameters:

local_script (bool) – If True, the last element of self.cmd is a script file whose content is rewritten for overlay path substitution.
capture_stderr (bool) – If True, capture stderr output (default False to maintain current behavior)

Return type:

None

module_load()[source]#

load env_modules dependencies

Return type:: None

module_run(local_script=False, capture_stderr=False)[source]#

Execute command with environment modules.

Parameters:

local_script (bool) – Passed through to local_run for overlay script rewriting.
capture_stderr (bool) – If True, capture stderr output

Return type:

None

conda_run(local_script=False, capture_stderr=False)[source]#

Execute command in conda environment.

Supports both name-based and path-based conda environments: - If program.path is set: uses -p <path>/<name> - Otherwise: uses -n <name>

Parameters:

local_script (bool) – Passed through to local_run for overlay script rewriting.
capture_stderr (bool) – If True, capture stderr output

Return type:

None

pipe_in(command, local_script=False)[source]#

Pipe input from another command.

Parameters:

command (Command)
local_script (bool)

Return type:

None

release_stdout()[source]#

Release stdout to terminal (for final commands in execution).

Use this for terminal commands where output should be visible to the user.

Return type:: None

capture_stdout()[source]#

Capture stdout for piping (for intermediate commands).

Use this to ensure stdout is captured in subprocess.PIPE for piping to another command.

Return type:: None

child_close()[source]#

Close child process.

Return type:: None

close(ignore_returncode=False)[source]#

Close command and return result with exit code.

Parameters:: ignore_returncode (bool) – If False (default), raise exception on non-zero exit. If True, ignore exit code (legacy behavior).
Returns:: Tuple of (communicate_result, returncode)
Raises:: CommandError – If returncode != 0 and ignore_returncode is False
Return type:: Tuple[Any, int]

replace_values_in_code(code_file)[source]#

Replace path mappings in a script file.

Processes self.volumes (docker container paths) first, then self.overlay_volumes (scratch paths). seen_paths ensures a path is only substituted once — for docker runs this means docker volumes take precedence and overlay entries for the same path are skipped.

Parameters:: code_file (str)
Return type:: None

pype.process.is_direct_child_of(x, y)[source]#

Check if x is a direct child of directory y.

Parameters:

x (str)
y (str)

Return type:

bool

pype.process.has_same_basedir(x, y)[source]#

Check if x and y have the same base directory.

Parameters:

x (str)
y (str)

Return type:

bool

class pype.process.Namespace(program_dict, log, profile)[source]#

Bases: object

Environment and dependency management system.

Manages different execution environments (path-based, environment modules, containers) and their dependencies. Supports: - Path-based program execution - Environment modules loading - Container image specification - Dependency resolution

The namespace format is ‘type@item’ where: - type: One of ‘path’, ‘env_module’, or ‘docker’ - item: The specific program/container/module to use

For example: - ‘docker@ubuntu:latest’ - ‘env_module@gcc/9.3.0’ - ‘path@/usr/local/bin/python’

Parameters:

program_dict (Dict[str, Any] | Any)
log (Any)
profile (Any)

__init__(program_dict, log, profile)[source]#

Initialize namespace configuration.

Parses a program configuration and sets up namespace properties for command execution. Supports both dict and ProfileProgram object inputs.

Parameters:

program_dict (Dict[str, Any] | Any) –
Program configuration as dict or ProfileProgram object with keys: - namespace: String in format ‘type@item’ where type is one of:

’docker’, ‘env_module’, or ‘path’
- version: Program version (optional for path type)
- dependencies: Optional list of dependent program names
log (Any) – Logger instance for namespace operations
profile (Any) – Environment profile containing program configurations

Attributes set:: type: Namespace type (‘docker’, ‘env_module’, or ‘path’) namespace: Original namespace string (e.g., ‘docker@ubuntu’) str: Parsed namespace item (e.g., ‘ubuntu’) version: Program version string list: List of resolved namespace items (empty until processing) docker_extra_args: Extra arguments for docker/singularity commands

Raises:

CommandNamespaceError – For invalid namespace format or unsupported types

Parameters:

program_dict (Dict[str, Any] | Any)
log (Any)
profile (Any)

first()[source]#

Return type:: str

last()[source]#

Return type:: str