Django, Celery and RabbitMQ

12th September, 2017

Running Asynchronous tasks using Django, Celery with RabbitMQ


  1. Install RabbitMQ
    # For Mac
    $ brew install rabbitmq
    $ export PATH=$PATH:/usr/local/sbin
    # Start server (Mac)
    $ sudo rabbitmq-server -detached #-detached flag indicates the server to run in the background
    # To stop server
    sudo rabbitmqctl stop
    #Add user settings (optional)
    $ sudo rabbitmq-server -detached
    $ sudo rabbitmqctl add_user myuser mypassword
    $ sudo rabbitmqctl add_vhost myvhost
    $ sudo rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"
    # For Ubuntu
    $ apt-get install -y erlang
    $ apt-get install rabbitmq-server
    # Then enable and start the RabbitMQ service:
    $ systemctl enable rabbitmq-server
    $ systemctl start rabbitmq-server
    #Check rabbitmq server status
    $ systemctl status rabbitmq-server​
  2. Assuming you have a Django application set up, install celery and dj-celery
    $ pip install celery django-celery-beat 
  3.  In your Django file, add the following
    CELERY_BROKER_URL = "amqp://myuser:mypassword@localhost:5672/myvhost"
  4. Create a Celery instance file called . Create this file at the same level of file and add the following:
    import os
    from celery import Celery
    # set the default Django settings module for the 'celery' program.
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')
    app = Celery('proj')
    # Using a string here means the worker doesn't have to serialize
    # the configuration object to child processes.
    # - namespace='CELERY' means all celery-related configuration keys
    #   should have a `CELERY_` prefix.
    app.config_from_object('django.conf:settings', namespace='CELERY')
    # Load task modules from all registered Django app configs.
  5. Now you need to ensure that this app is loaded. To do so, add the following in the file.
    # This will make sure the app is always imported when
    # Django starts so that shared_task will use this app.
    from .celery import app as celery_app
    __all__ = ['celery_app']
  6. Add 'django_celery_beat' to the list of Installed apps in the Django
  7. Add the following settings for Celery in the file.
    # Celery settings
    CELERY_BROKER_URL = "amqp://celeryuser:celerypassword@localhost:5672/celeryhost"
    # for security reasons, mention the list of accepted content-types (in this case json)
    CELERY_TIMEZONE = 'Europe/Berlin'
    CELERY_BEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'
  8. Run migration for the django_celery_beat app
    $ python migrate
  9. Create a periodic task in file in respective django app. e.g.:
    from celery.task.schedules import crontab
    from celery.decorators import periodic_task
    from celery import shared_task
    from celery.utils.log import get_task_logger
    logger = get_task_logger(__name__)
    def test_celery_worker():
        print("Celery Workers are working fine.")
    # A periodic task that will run every minute (the symbol "*" means every)
    @periodic_task(run_every=(crontab(hour="*", minute="*", day_of_week="*")))
    def task_example():"Task started")
        # add code"Task finished")​
  10. Finally check the server is running and initiate the django celery processes.
    $ sudo rabbitmq-server -detached  #for starting rabbitmq server (if its not running)
    $ python shell
    >>> from django_celery_beat.models import PeriodicTask
    >>> PeriodicTask.objects.update(last_run_at=None)
    #Initiating a worker
    $ celery -A proj worker -l info -n worker
    # Initiating the scheduler
    $ celery -A proj beat -l info
  11. At the same time run the Django runserver command in another terminal.