If you trigger the DAG and go to Admin and XComs, you will see that: With the PythonOperator, we can use the keyword return along with the value in the Python callable function to push the value we want to share into the database and create the corresponding XCom automatically. There are different ways to create an XCom but let’s begin with the most basic one: by returning a value. In this Airflow XCom example, we will push an XCom for each model A, B, and C with their corresponding accuracy. Create an XCom for each training_model task.The objective is to create one XCom for each model and fetch the XComs back in the task choose_model to select the best model. Finally, we pick the best model based on the generated accuracies in choose_model. That function randomly generates an accuracy for each model A, B, and C. Each task uses the PythonOperator to execute the function _training_model. Then we dynamically create three tasks, training_model_ with a list comprehension. downloading_data uses the BashOperator to execute a bash command that waits for three seconds. Start_date=datetime(2023, 1, = BashOperator(ĭownloading_data > training_model_task > choose_modelĬreate a file xcom_dag.py and put the code in it. Here is the data pipeline we will use: from airflow import DAGįrom import BashOperatorįrom import PythonOperator Great! Now you know what an XCom is, let’s create your first Airflow XCom! How to use XCom in Airflow? To access XComs, go to the user interface, then Admin and XComs. That has some implications that you will see later in this tutorial. Keep in mind that Airflow stores XComs in the database. A dag id: The identifier of the DAG that creates the XCom.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |