Can cause high connection overhead if called repeatedly in top-level DAG parsing code. 5. Production Best Practices & Anti-Patterns ❌ Anti-Patterns to Avoid
# Anti-pattern @task def bad_task(): import pandas as pd df = pd.DataFrame('a': range(10000)) return df # Will hit size limit or cause DB slowdown airflow xcom
# Pull the return value (default key) ret_val = ti.xcom_pull(task_ids='push_task') Can cause high connection overhead if called repeatedly
| Limitation | Explanation | |------------|-------------| | | XComs are metadata, not data lakes. For large datasets, pass S3/GCS paths. | | No ordering guarantee | If multiple tasks push the same key, the pull behavior may be ambiguous. | | Database pressure | Many large XComs can degrade metadata DB performance. | | Not real-time | XComs are written at task completion and read after that. | | Security | XCom values are stored in plain text unless encrypted at the DB level. | For large datasets, pass S3/GCS paths
dag = DAG( 'xcom_example', default_args=default_args, schedule_interval=timedelta(days=1), )
task_1 >> task_2