In a nutshell:

Available presets

There are available preset of snippets and pipelines. You can access the list of repository and manage the installed modules with the repos subcommand

$ pype repos
usage: pype repos [-r REPO_LIST] {list,install,clean,info} ...

positional arguments:
  list                List the available repositories
  install             Install modules from selected repository
  clean               Cleanup all module folders
  info                Print location of the modules currently in use

optional arguments:
  -r REPO_LIST, --repo REPO_LIST
                    Repository list. Default:

Overview of the framework

The framework is heavily based on the python argparse module aiming for a self explanatory use of the main command line interface: pype.

The management of tools version and sources (assembly/organisms) is organized in profiles files.

 $ pype profiles
 usage: pype profiles {info,check}

 positional arguments:
 info        Retrieve details from available profiles
 check       Check if a profile is valid

$ pype profiles info --all
     default:          Default profile b37 from 1k genome project
     hg19:             Profile for the UCSC hg19 assembly
     hg38:             Profile for the hg38 assembly human genome

Each profile is defined by a YAML file, an example:

    description: HG19 default profile
    date:        10/03/2016
genome_build: 'hg19'
    genome_fa:    /path/to/b37/genome.fa
    genome_fa_gz: /path/to/b37//genome.fa.gz
    exons_list:   /path/to/exome_hg37_interval_list.txt
    db_snp:       /path/to/dbSNP/dbsnp_138.b37.vcf.gz
    cosmic:       /path/to/cosmic/CosmicCodingMuts.vcf.gz

        path: samtools
        version: 0.1.18
        path: samtools
        version: 1.2

Each tools is wrapped into a python function (snippets), which is designed to load program specified in the selected profile, and usually execute the tools using the subprocess module.

$ pype snippets
error: too few arguments
usage: pype snippets [--log LOG]

positional arguments:
biobambam_merge     Mark duplicates and merge BAMs with biobambam
mutect              Call germline and somatic variants with mutect
oncotator           Annotate various file format with Oncotator
bwa_mem             Align fastQ file with the BWA mem algorithm
freebayes           A haplotype-based variant detector

The tools can be chained and combined one-another within a YAML files (pipelines), which specify dependencies and input files. The profile is parsed to provide command line option by the argparse interface

$ pype pipelines
error: too few arguments
usage: pype pipelines [--queue {msub,echo,none}] [--log LOG]

positional arguments:
bwa                 Align with bwa mem, merge different lanes and
                    performs various QC stats
freebayes           Freebayes pipeline from Fastq to VCF
mutect              Mutect from fastq files
bwa_mem             Align with bwa mem and performs QC stats

optional arguments:
--queue {msub,echo,none}
                   Select the queuing system to run the pipeline
--log LOG          Path used to write the pipeline logs. Default
                   working directory.

A typical scenario that demonstrate the advantages offered by this package is the alignment of two DNA sequencing (NGS) samples, a normal and a tumor samples, and perform a comparison between two samples to find somatic mutations. In the command line this task require for each sample a command for the alignment, a command for sorting and a command for indexing. After the two samples are processed the results need to be compared by a mutation caller, usually specifying some database such as COSMIC or dbSNP.

The easiest and fastest way to implement the tasks is by wrapping the command line into bash script and submit them to a queuing system, or run it directly in a shell. This would impose a self organization of logs and scripts, and still possibli results in a lot of maintenance to keep the scripts up to date.

Other pipeline systems can solve the problem, but they may not be executable on the fly by command line interface (eg, need to configure file previous execution) or they may not allows to change genome build or database version easily.

with the bio_pype framework each step can be launched separately, example of usage of the alignment snippet:

pype snippets bwa_mem
pype snippets bwa_mem -h HEADER -1 F1 -2 F2 [-t TMP] [-o OUT]

optional arguments:
  -h HEADER          Header file containing the @RG groups,
                     or the comma separated header line
  -1 F1              First mate fastQ file
  -2 F2              Second mate fastQ file
  -t TMP, --tmp TMP  Temporary folder
  -o OUT, --out OUT  Output name for the bam file

or use the pipeline:

pype pipelines mutect
usage: pype pipelines mutect --tumor_bam TUMOR_BAM --out_dir OUT_DIR
                             --sample_name SAMPLE_NAME --normal_bam NORMAL_BAM
                             [--tmp_dir TMP_DIR] --tumor_bwa_list
                             TUMOR_BWA_LIST --qc_bam_tumor QC_BAM_TUMOR
                             --normal_bwa_list NORMAL_BWA_LIST --qc_bam_normal

  Required pipeline arguments

  --tumor_bam TUMOR_BAM
                        Input Bam file of the tumor sample, type: str
  --out_dir OUT_DIR     Output file name prefix for the analysis, type: str
  --sample_name SAMPLE_NAME
                        Sample name or identifier of the run, type: str
  --normal_bam NORMAL_BAM
                        Input Bam file of the normal/control sample, type: str
  --tumor_bwa_list TUMOR_BWA_LIST
                        Batch file to run the bwa_mem pipeline on different
                        lanes of the tumor, type: str
  --qc_bam_tumor QC_BAM_TUMOR
                        Tumor BAM files QC directory output path, type: str
  --normal_bwa_list NORMAL_BWA_LIST
                        Batch file to run the bwa_mem pipeline on different
                        lanes of the normal/control, type: str
  --qc_bam_normal QC_BAM_NORMAL
                        Normal/Control BAM files QC directory output path,
                        type: str

  Optional pipeline arguments

  --tmp_dir TMP_DIR     temporary folder, type: str. Default: /scatch

Python and Environment Modules:

example of loading the hg37 profile in the python prompt:

from pype.modules.profiles import get_profiles

profiles = get_profiles({})
profile = profiles['hg37']
genome = profile.files['genome_fa_gz']

The framework also supports Environment Modules:

from pype.modules.profiles import get_profiles
from pype.env_modules import get_module_cmd, program_string

profiles = get_profiles({})
module('add', program_string(profile.programs['samtools_1']))