Airflow — Cross DAG Dependency Simplified
Cross DAG dependency or sensing the completion of external airflow dags or tasks can be very useful when you have dependency between DAGs or task in a different DAG to complete for a specific execution date/time. This can be achieved using airflow operator called as ExternalTaskSensor.
Sensors are pre-built in airflow. As the title suggests, they sense for the completion of a state of any task in airflow, simple as that. We will be using sensors to set dependencies between our DAGS/Pipelines, so that one does not run until the dependency had finished. There is no need to write any custom operator for this.
Let us understand a use case for the same, say we have a child DAG called test_child_dag_1 and a parent DAG called parent_dag_ext_sensor, we want to run the parent DAG only when the child DAG is successfully ran. Here, we can make use of ExternalTaskSensor operator which helps us define Cross Dag Dependency between our child and parent DAG.
test_child_dag_1.py
parent_dag_ext_sensor.py
In our parent DAG, we have used ExternalTaskSensor operator which will sense the execution of the child DAG as shown from line no. 20 to 25. Only when the child DAG execution is successful then only the parent DAG task will run.
Similarly, if the parent DAG depends on more than one child DAG execution to be sensed, the similar approach can be applied by ExternalTaskSensor, like in the below example we have two child DAGs whose completion will trigger the parent DAG.
test_child_dag_1.py having dag id as test_child_dag_1 and say test_child_dag_2.py having dag id as test_child_dag_1a
external_task_sensor_1: is to sense the first child DAG, external_task_sensor_2 is to sense the second child DAG.
Hope this helps!
