Developer Guide

How to extend Bio_pype: write reusable snippets, compose them into pipelines, wire arguments and results between steps, build execution environments, and hook into the validation, storage and queue internals.

New to Bio_pype? Start with the User Guide to understand how workflows are installed and run before authoring your own.

Writing modules

• Snippets
  • Markdown Snippets (Recommended)
  • Namespaces
  • Python Snippets (Advanced)
  • Choosing Between Markdown and Python
  • Best Practices
  • Common Patterns
  • Additional Resources
  • Quick Reference
• Pipelines
  • Info Section
  • Steps/Items
  • DAG Execution Model
  • Job Deduplication and Batch Items
  • Running Pipelines
  • Complete Example
  • Quick Reference
• Composite Arguments and Snippet Results Pattern
  • Quick Concept
  • Argument Value Syntax Rules
  • Understanding Snippet Results
  • Understanding Composite Arguments
  • The Disambiguation Problem and Solution
  • Real Example: GATK Somatic Variant Calling Pipeline
  • When result_arguments is Empty
  • Summary
• Batch List Arguments and Dynamic Batch Expansion
  • Overview
  • The batch_list_arg Type
  • Complete Example: Split-Process-Merge Pipeline
  • Required Snippet Results
  • Key Concepts
  • Comparison with batch_file_arg
  • See Also

Authoring tools

• Pipeline Builder
  • Starting the Builder
  • What the Builder Provides
  • Refreshing Snippets
• Validation and Editor Integration
  • Command Line Validation
  • What Gets Validated
  • LSP Server
  • Editor Integration
  • Exit Codes
  • Troubleshooting
  • See Also

Execution internals

• Storage Backends
  • Overview
  • Interface
  • Data flow during snippet execution
  • OverlayVolume
  • Snapshot registry (cloud backends)
  • Implementing a new backend

API reference

• API Documentation
  • pype.utils
  • pype.process
  • pype.misc
  • pype.logger
  • pype.exceptions
  • Core Components
  • Utilities