从 Oozie 工作流到 Airflow DAG 的迁移工具_python_代码_下载

<!-- Copyright 2019 Google LLC Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> # Oozie to Airflow [](https://travis-ci.org/GoogleCloudPlatform/oozie-to-airflow) [](https://codecov.io/gh/GoogleCloudPlatform/oozie-to-airflow) [](https://github.com/ambv/black) [](https://opensource.org/licenses/Apache-2.0) [](https://dependabot.com) [](https://pyup.io/repos/github/GoogleCloudPlatform/oozie-to-airflow/) A tool to easily convert between [Apache Oozie](http://oozie.apache.org/) workflows and [Apache Airflow](https://airflow.apache.org) workflows. The program targets Apache Airflow >= 1.10 and Apache Oozie 1.0 XML schema. If you want to contribute to the project, please take a look at [CONTRIBUTING.md](CONTRIBUTING.md) # Table of Contents <!-- START doctoc generated TOC please keep comment here to allow auto update --> <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> - [Background](#background) - [Running the Program](#running-the-program) - [Installing from PyPi](#installing-from-pypi) - [Installing from sources](#installing-from-sources) - [Running the conversion](#running-the-conversion) - [Structure of the application folder](#structure-of-the-application-folder) - [The o2a libraries](#the-o2a-libraries) - [Supported Oozie features](#supported-oozie-features) - [Control nodes](#control-nodes) - [EL Functions](#el-functions) - [Workflow and node notifications](#workflow-and-node-notifications) - [Airflow-specific optimisations](#airflow-specific-optimisations) - [Removing unnecessary control nodes](#removing-unnecessary-control-nodes) - [Removing inaccessible nodes](#removing-inaccessible-nodes) - [Common Known Limitations](#common-known-limitations) - [File/Archive functionality](#filearchive-functionality) - [Not all global configuration methods are supported](#not-all-global-configuration-methods-are-supported) - [Support for uber.jar feature](#support-for-uberjar-feature) - [Support for .so and .jar lib files](#support-for-so-and-jar-lib-files) - [Custom messages missing for Kill Node](#custom-messages-missing-for-kill-node) - [Capturing output is not supported](#capturing-output-is-not-supported) - [Subworkflow DAGs must be placed in examples](#subworkflow-dags-must-be-placed-in-examples) - [EL functions support](#el-functions-support) - [Notification proxy is not supported](#notification-proxy-is-not-supported) - [Cloud execution environment for Oozie to Airflow conversion](#cloud-execution-environment-for-oozie-to-airflow-conversion) - [Cloud environment setup](#cloud-environment-setup) - [Examples](#examples) - [EL Example](#el-example) - [SSH Example](#ssh-example) - [Email Example](#email-example) - [MapReduce Example](#mapreduce-example) - [FS Example](#fs-example) - [Java Example](#java-example) - [Pig Example](#pig-example) - [Shell Example](#shell-example) - [Spark Example](#spark-example) - [Sub-workflow Example](#sub-workflow-example) - [DistCp Example](#distcp-example) - [Decision Example](#decision-example) - [Hive/Hive2 Example](#hivehive2-example) - [Demo Example](#demo-example) - [Childwf Example](#childwf-example) <!-- END doctoc generated TOC please keep comment here to allow auto update --> # Background Apache Airflow is a workflow management system developed by AirBnB in 2014. It is a platform to programmatically author, schedule, and monitor workflows. Airflow workflows are designed as [Directed Acyclic Graphs](https://airflow.apache.org/tutorial.html#example-pipeline-definition) (DAGs) of tasks in Python. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Apache Oozie is a workflow scheduler system to manage Apache Hadoop jobs. Oozie workflows are also designed as [Directed Acyclic Graphs](https://oozie.apache.org/docs/3.1.3-incubating/DG_Overview.html) (DAGs) in XML. There are a few differences noted below: | | Spec. | Task | Dependencies | "Subworkflows" | Parameterization | Notification | |---------|--------|-------------|---------------------------------|----------------|------------------------------|---------------------| | Oozie | XML | Action Node | Control Node | Subworkflow | EL functions/Properties file | URL based callbacks | | Airflow | Python | Operators | Trigger Rules, set_downstream() | SubDag | jinja2 and macros | Callbacks/Emails | # Running the Program Note that you need Python >= 3.6 to run the converter. ## Installing from PyPi You can install `o2a` from PyPi via `pip install o2a`. After installation, the [o2a](bin/o2a) and [o2a-validate-workflows](bin/o2a-validate-workflows) should be available on your path. ## Installing from sources 1. (Optional) Install virtualenv: In case you use sources of `o2a`, the environment can be set up via the virtualenv setup (you can create one using [virtualenvwrapper](https://virtualenvwrapper.readthedocs.io/en/latest/) for example). 2. Install Oozie-to-Airflow - you have 2 options to do so: 1. automatically: install `o2a` from local folder using `pip install -e .` This will take care about, among others, adding the [bin](bin) subdirectory to the PATH. 2. more manually: 1. While in your virtualenv, you can install all the requirements via `pip install -r requirements.txt`. 2. You can add the [bin](bin) subdirectory to your PATH, then all the scripts below can be run without adding the `./bin` prefix. This can be done for example by adding a line similar to the one below to your `.bash_profile` or `bin/postactivate` from your virtual environment: ```bash export PATH=${PATH}:<INSERT_PATH_TO_YOUR_OOZIE_PROJECT>/bin ``` Otherwise you need to run all the scripts from the bin subdirectory, for example: ```bash ./bin/o2a --help ``` In all the example commands below, it is assumed that the [bin](bin) directory is in your PATH - either installed from PyPi or from the sources. ## Running the conversion You can run the program by calling: `o2a -i <INPUT_APPLICATION_FOLDER> -o <OUTPUT_FOLDER_PATH>` Example: `o2a -i examples/demo -o output/demo` This is the full usage guide, available by running `o2a -h` ``` usage: o2a [-h] -i INPUT_DIRECTORY_PATH -o OUTPUT_DIRECTORY_PATH [-n DAG_NAME] [-u USER] [-s START_DAYS_AGO] [-v SCHEDULE_INTERVAL] [-d] Convert Apache Oozie workflows to Apache Airflow workflows. optional arguments: -h, --help show this help message and exit -i INPUT_DIRECTORY_PATH, --input-directory-path INPUT_DIRECTORY_PATH Path to input directory -o OUTPUT_DIRECTORY_PATH, --output-directory-path OUTPUT_DIRECTORY_PATH Desired output directory -n DAG_NAME, --dag-name DAG