One - One Code All

Blog Content

airflow升级后celery连接mq的方式不同

Python 并行计算   2018-09-30 10:53:04

airflow升级后celery连接mq的方式不同

airflow升级到1.10后,原本部署到多个worker,只有一个worker在运行。

推测:未连接上rmq.

debug...

pip list 查看安装库版本:

celery           4.1.1
flower           0.9.2
mysqlclient      1.3.13
SQLAlchemy       1.1.18
redis            2.10.6


celery官网参考:http://docs.celeryproject.org/en/latest/userguide/configuration.html#rpc-backend-settings



RPC backend settings
result_persistent
Default: Disabled by default (transient messages).

If set to True, result messages will be persistent. This means the messages won’t be lost after a broker restart.

Example configuration
result_backend = 'rpc://'
result_persistent = False



celery官网参考:http://docs.celeryproject.org/en/latest/userguide/configuration.html#example-configuration-file



Example configuration file
This is an example configuration file to get you started. It should contain all you need to run a basic Celery set-up.

## Broker settings.
broker_url = 'amqp://guest:guest@localhost:5672//'

# List of modules to import when the Celery worker starts.
imports = ('myapp.tasks',)

## Using the database to store task state and results.
result_backend = 'db+sqlite:///results.db'

task_annotations = {'tasks.add': {'rate_limit': '10/s'}}



上一篇:python获取SSL证书忽略CA安全认证
下一篇:python中打印输出详细异常信息traceback使用,异常的获取与处理

The minute you think of giving up, think of the reason why you held on so long.