PrefectHQ/prefect
Fork: 1582 Star: 16280 (更新于 2024-10-29 12:26:08)
license: Apache-2.0
Language: Python .
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
最后发布版本: 2.20.4 ( 2024-08-31 00:38:01)
Prefect
Prefect is a workflow orchestration framework for building data pipelines in Python. It's the simplest way to elevate a script into a resilient production workflow. With Prefect, you can build resilient, dynamic data pipelines that react to the world around them and recover from unexpected changes.
With just a few lines of code, data teams can confidently automate any data process with features such as scheduling, caching, retries, and event-based automations.
Workflow activity is tracked and can be monitored with a self-hosted Prefect server instance or managed Prefect Cloud dashboard.
Getting started
Prefect requires Python 3.9 or later. To install the latest or upgrade to the latest version of Prefect, run the following command:
pip install -U prefect
Then create and run a Python file that uses Prefect flow
and task
decorators to orchestrate and observe your workflow - in this case, a simple script that fetches the number of GitHub stars from a repository:
from prefect import flow, task
from typing import List
import httpx
@task(log_prints=True)
def get_stars(repo: str):
url = f"https://api.github.com/repos/{repo}"
count = httpx.get(url).json()["stargazers_count"]
print(f"{repo} has {count} stars!")
@flow(name="GitHub Stars")
def github_stars(repos: List[str]):
for repo in repos:
get_stars(repo)
# run the flow!
if __name__=="__main__":
github_stars(["PrefectHQ/Prefect"])
Fire up the Prefect UI to see what happened:
prefect server start
To run your workflow on a schedule, turn it into a deployment and schedule it to run every minute by changing the last line of your script to the following:
if __name__ == "__main__":
github_stars.serve(
name="first-deployment",
cron="* * * * *",
parameters={"repos": ["PrefectHQ/prefect"]}
)
You now have a server running locally that is looking for scheduled deployments! Additionally you can run your workflow manually from the UI or CLI. You can even run deployments in response to events.
Prefect Cloud
Prefect Cloud provides workflow orchestration for the modern data enterprise. By automating over 200 million data tasks monthly, Prefect empowers diverse organizations — from Fortune 50 leaders such as Progressive Insurance to innovative disruptors such as Cash App — to increase engineering productivity, reduce pipeline errors, and cut data workflow compute costs.
Read more about Prefect Cloud here or sign up to try it for yourself.
prefect-client
If your use case is geared towards communicating with Prefect Cloud or a remote Prefect server, check out our prefect-client. It is a lighter-weight option for accessing client-side functionality in the Prefect SDK and is ideal for use in ephemeral execution environments.
Next steps
- Check out the Docs.
- Join the Prefect Slack community.
- Learn how to contribute to Prefect.
最近版本更新:(数据更新于 2024-09-03 02:46:08)
2024-08-31 00:38:01 2.20.4
2024-08-30 02:00:42 3.0.0rc20
2024-08-23 04:33:41 2.20.3
2024-08-23 02:00:56 3.0.0rc19
2024-08-16 02:20:38 2.20.2
2024-08-16 02:00:43 3.0.0rc18
2024-08-15 00:25:30 3.0.0rc17
2024-08-13 05:31:16 3.0.0rc16
2024-08-10 00:02:59 2.20.1
2024-08-09 02:00:48 3.0.0rc15
主题(topics):
automation, data, data-engineering, data-ops, data-science, infrastructure, ml-ops, observability, orchestration, pipeline, prefect, python, workflow, workflow-engine
PrefectHQ/prefect同语言 Python最近更新仓库
2024-11-05 16:16:26 Guovin/TV
2024-11-05 15:03:24 Cinnamon/kotaemon
2024-11-05 11:00:51 home-assistant/core
2024-11-04 23:11:11 DS4SD/docling
2024-11-04 10:56:18 open-compass/opencompass
2024-11-04 08:51:21 yt-dlp/yt-dlp