flink-extended/dl-on-flink
Fork: 201 Star: 690 (更新于 2024-10-31 14:15:14)
license: Apache-2.0
Language: Java .
Deep Learning on Flink aims to integrate Flink and deep learning frameworks (e.g. TensorFlow, PyTorch, etc) to enable distributed deep learning training and inference on a Flink cluster.
最后发布版本: 0.5.0 ( 2022-06-27 14:48:38)
Deep Learning on Flink
Deep Learning on Flink aims to integrate Flink and deep learning frameworks (e.g. TensorFlow, PyTorch, etc.) to enable distributed deep learning training and inference on a Flink cluster.
It runs the deep learning tasks inside a Flink operator so that Flink can help establish a distributed environment, manage the resource, read/write the data with the rich connectors in Flink and handle the failures.
Currently, Deep Learning on Flink supports TensorFlow.
Supported Operating System
Deep Learning on Flink is tested and supported on the following 64-bit systems:
- Ubuntu 18.04
- macOS 10.15
Support Framework Version
- TensorFlow: 1.15.x & 2.4.x
- PyTorch: 1.11.x
- Flink: 1.14.x
Getting Started
Deep learning on Flink currently works with Tensorflow and PyTorch. You can see the following pages for the usage and examples.
Build From Source
Requirements
- python: 3.7
- cmake >= 3.6
- java 1.8
- maven >=3.3.0
Deep Learning on Flink requires Java and Python works together. Thus, we need to build for both Java and Python.
Initializing Submodules before Building Deep Learning on Flink from Source
Please use the following command to initialize submodules before building from source.
git submodule update --init --recursive
Build Java
mvn -DskipTests clean install
After finish, you can find the target distribution in the dl-on-flink-dist/target
folder.
Build Python
Install from Source
You can run the following commands to install the Python packages from source
# Install dl-on-flink-framework first
pip install dl-on-flink-framework/python
# Note that you should only install one of the following as they require
# different versions of Tensorflow
# For tensorflow 1.15.x
pip install dl-on-flink-tensorflow/python
# For tensorflow 2.4.x
pip install dl-on-flink-tensorflow-2.x/python
Build wheels
We provide a script to build wheels for Python packages, you can run the following command.
bash tools/build_wheel.sh
After finish, you can find the wheels at tools/dist
. Then you can install the
python package with the wheels.
pip install tools/dist/<wheel>
For More Information
License
最近版本更新:(数据更新于 2024-09-05 13:00:18)
2022-06-27 14:48:38 0.5.0
2022-02-22 22:38:03 0.4.0
2020-11-26 15:00:56 0.2.2
2020-11-25 15:37:32 0.2.1
2019-07-12 15:49:07 0.1.0
主题(topics):
deep-learning, flink, python, pytorch, tensorflow
flink-extended/dl-on-flink同语言 Java最近更新仓库
2024-11-05 19:32:42 xiaojieonly/Ehviewer_CN_SXJ
2024-11-05 04:13:47 Stirling-Tools/Stirling-PDF
2024-11-04 02:03:13 exzhawk/EhViewer
2024-11-03 00:43:09 PBH-BTN/PeerBanHelper
2024-10-31 18:52:00 CodePhiliaX/Chat2DB
2024-10-31 11:19:43 mybatis-flex/mybatis-flex