Implement adr2 relative pipelines + api changes.

In brief, this release lets pipelines reference custom modules & child pipelines relative to the pipeline itself, rather than the current directory. This lets you create portable, re-usable & composable pipeline libraries.

Breaking Changes

This is a major version increment because it comes with BREAKING CHANGES:

1. API: pipelinerunner.run() replaces both pipelinerunner.main() and pipelinerunner.main_with_context()
2. API: def get_pipeline_definition(pipeline_name, working_directory) signature for custom pype loaders changes to def get_pipeline_definition(pipeline_name, parent)
3. CLI: the —dir flag now only sets the directory for ad hoc custom Python modules, it does NOT also set the directory for pipelines anymore
4. Final removal of deprecated get_formatted_iterable, get_formatted_string #195 & pypyr.steps.contextset #184. Where previously these would just give deprecation warnings, they are now completely removed.
5. pypyr.pypeloaders.fileloader renamed pypyr.loaders.file

Non-Breaking Changes

• You can now access the current pipeline’s metadata & loader information from within a pipeline with context.current_pipeline
• Improve handling of absolute paths in file loader only to search path once, rather than unnecessarily go through the same relative path lookup sequence with the same path.
• Typing support added for the pypyr API entrypoint.

Detailed Technical Breakdown:

• Introduce new classes to model pipeline payload, rather than just using the bare dict-like yaml directly.
  • PipelineInfo - pipeline metadata set by loader. This maintains a pipeline’s parent/path info so that child pipelines can load relative to the parent.
  • PipelineDefinition - this wraps the pipeline payload and its metadata (PipelineInfo) to allow pypyr to cache it all with one reference
• Add new Pipeline class for the run-time properties of a single run.
  • The Pipeline references the shared cached PipelineDefinition.
  • Move run + load_and_run logic from pypyr.pipelinerunner to the new Pipeline class. This massively streamlines the pipeline invocation process, since run/load_and_run can just operate on the shared Pipeline state rather than sling a bunch of args between different functions as before.
• Add a call-stack of running Pipeline instances on Context. i.e Parent -> child1 -> child2 where the root pipeline calls child pipelines via pype
  • Add current_pipeline attribute to Context, controlled with a context manager to scope itself to an individual pipeline run’s lifespan.
    • This means that steps can access current pipeline’s properties.
    • This allows pypyr.steps.pype to find the current (i.e parent) pipeline’s metadata such as path, to load child pipeline relative to the calling pipeline’s location.
    • When a child pipeline completes, the calling pipeline (i.e the previous Pipeline in the call-stack) becomes the current pipeline
• Amend pypyr.steps.pype to instantiate Pipeline object to load_and_run() child.
  • pype now deals the context.current_pipeline.pipeline_definition.info metadata to work out whether to cascade parent path down to child, so child can load relative to the parent.
  • Notably, the parent loader now cascades to the child, so pipeline authors don’t need explicitly to set the same custom loader repeatedly for each child.
  • Given the context manager controlling current pipeline scope in Pipeline.load_and_run_pipeline remove the clumsy side-shuffle for pipeline_name, working_dir to swap out these values as child pipe runs and swap these back when it completes/errors.
• Remove global PipelineCache. Replace with distinct pipeline cache per loader.
  • This resolves a long standing limitation where pypyr assumed unique pipeline names across all loaders.
  • The per-loader pipeline cache stores PipelineDefinition objects.
  • Introduce Loader class, which wraps loader & its pipeline cache. Loader is what the LoaderCache caches.
  • Thus LoaderCache -> Loader -> _pipelineCache -> PipelineDefinition
• File loader has a private file cache keyed on absolute path of file
  • This is to prevent >1 load+parse where the same underlying pipeline.yaml file has different names in the loader’s pipeline cache
    • e.g (name=‘dir/mypipe’, parent=None) and (name=‘mypipe’, parent=‘dir’) both resolve to dir/mypipe.yaml
  • Caching a reference to the PipelineDefinition object, so not duplicating memory
• Improve handling of absolute paths in file loader only to search path X1, rather than unecessarilly go through the same relative path lookup sequence with the same path.
• File loader get_pipeline_definition now returns a PipelineDefinition with PipelineFileInfo to store file-system specific metadata for the loader pipeline
• Remove working_dir global. The py_dir input on run() now refers ONLY to module paths, NOT pipeline locations.
  • The CLI —dir flag, or py_dir input on run() basically adds the specified directory to sys.path.
• Add current pipeline’s parent directory to sys.path on load. This allows child pipelines to resolve custom modules relative to itself.
• pypyr.dsl.Step does not need StepsRunner anymore, because it can get it from the context.current_pipeline instead.
• Recode (some) integration tests to take advantage of list pypyr.steps.append step and checking that for output on return context rather than intercepting logger.NOTIFY.
• Rename master branch to main in CI/CD GitHub actions
• Add typing annotations to the public run() function and the Pipeline class public accessors. The idea is NOT to type pypyr exhaustively, just to provide annotations for the sensible/likely entrypoint to enhance API user experience. Include py.typed in pypyr package.
• Remedy packaging snafu where tests.common was deploying alongside pypyr because exclude condition in find_packages didn't include wildcard for sub packages.

What's Changed

• Relative pipelines & API run() replaces main/main_with_context by @yaythomas in https://github.com/pypyr/pypyr/pull/243

Full Changelog: https://github.com/pypyr/pypyr/compare/v4.6.0...v5.0.0