airflow升级后celery连接mq的方式不同
airflow升级到1.10后,原本部署到多个worker,只有一个worker在运行。
推测:未连接上rmq.
debug...
pip list 查看安装库版本:
celery 4.1.1
flower 0.9.2
mysqlclient 1.3.13
SQLAlchemy 1.1.18
redis 2.10.6
celery官网参考:http://docs.celeryproject.org/en/latest/userguide/configuration.html#rpc-backend-settings
RPC backend settings
result_persistent
Default: Disabled by default (transient messages).
If set to True, result messages will be persistent. This means the messages won’t be lost after a broker restart.
Example configuration result_backend = 'rpc://' result_persistent = False
celery官网参考:http://docs.celeryproject.org/en/latest/userguide/configuration.html#example-configuration-file
Example configuration file
This is an example configuration file to get you started. It should contain all you need to run a basic Celery set-up.
## Broker settings. broker_url = 'amqp://guest:guest@localhost:5672//' # List of modules to import when the Celery worker starts. imports = ('myapp.tasks',) ## Using the database to store task state and results. result_backend = 'db+sqlite:///results.db' task_annotations = {'tasks.add': {'rate_limit': '10/s'}}