One - One Code All

Blog Content

airflow控制task并发度

Python 并行计算   2014-06-01 19:37:26

3个参数用于控制Task的并发度,
parallelism, 一个Executor同时运行task实例的个数
dag_concurrency, 一个dag中某个task同时运行的实例个数
max_active_runs_per_dag: 一个dag同时启动的实例个数

celeryd_concurrency:


务必保证核数够用,否则需要调低dag_concurrency, max_active_runs_per_dag,max_threads,parallelism, 否则worker出现莫名其妙的问题。

比如: The maximum number of running tasks for this task's DAG 'phase0' has been reached.


上一篇:C++ 连接MySql
下一篇:pandas中dataframe过滤列str.contain出现ValueError: cannot index with vector containing NA/NaN values错误的解决方法

The minute you think of giving up, think of the reason why you held on so long.