在特定情况下,修改DAG后,为了避免当前日期之前任务的运行,可以使用backfill填补特定时间段的任务
airflow backfill -s START -e END --mark_success DAG_ID
官网链接: https://airflow.readthedocs.io/en/latest/cli.html
Run subsections of a DAG for a specified date range. If reset_dag_run option is used, backfill will first prompt users whether airflow should clear all the previous dag_run and task_instances within the backfill date range. If rerun_failed_tasks is used, backfill will auto re-run the previous failed task instances within the backfill date range.
airflow backfill [-h] [-t TASK_REGEX] [-s START_DATE] [-e END_DATE] [-m] [-l]
[-x] [-i] [-I] [-sd SUBDIR] [--pool POOL]
[--delay_on_limit DELAY_ON_LIMIT] [-dr] [-v] [-c CONF]
[--reset_dagruns] [--rerun_failed_tasks]
dag_id| dag_id | The id of the dag |
| -t, --task_regex | |
| The regex to filter specific task_ids to backfill (optional) | |
| -s, --start_date | |
| Override start_date YYYY-MM-DD | |
| -e, --end_date | Override end_date YYYY-MM-DD |
| -m, --mark_success | |
Mark jobs as succeeded without running them Default: False | |
| -l, --local | Run the task using the LocalExecutor Default: False |
| -x, --donot_pickle | |
Do not attempt to pickle the DAG object to send over to the workers, just tell the workers to run their version of the code. Default: False | |
| -i, --ignore_dependencies | |
Skip upstream tasks, run only the tasks matching the regexp. Only works in conjunction with task_regex Default: False | |
| -I, --ignore_first_depends_on_past | |
Ignores depends_on_past dependencies for the first set of tasks only (subsequent executions in the backfill DO respect depends_on_past). Default: False | |
| -sd, --subdir | File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’ Default: “[AIRFLOW_HOME]/dags” |
| --pool | Resource pool to use |
| --delay_on_limit | |
Amount of time in seconds to wait when the limit on maximum active dag runs (max_active_runs) has been reached before trying to execute a dag run again. Default: 1.0 | |
| -dr, --dry_run | Perform a dry run Default: False |
| -v, --verbose | Make logging output more verbose Default: False |
| -c, --conf | JSON string that gets pickled into the DagRun’s conf attribute |
| --reset_dagruns | |
if set, the backfill will delete existing backfill-related DAG runs and start anew with fresh, running DAG runs Default: False | |
| --rerun_failed_tasks | |
if set, the backfill will auto-rerun all the failed tasks for the backfill date range instead of throwing exceptions Default: False | |