{"id":6756,"date":"2020-06-30T16:54:04","date_gmt":"2020-06-30T16:54:04","guid":{"rendered":"https:\/\/cheesecakelabs.com\/blog\/?p=6756\/"},"modified":"2022-07-01T17:08:43","modified_gmt":"2022-07-01T17:08:43","slug":"asynchronous-task-queue-django","status":"publish","type":"post","link":"https:\/\/cheesecakelabs.com\/blog\/asynchronous-task-queue-django\/","title":{"rendered":"Asynchronous task queue with Django, Celery and AWS SQS"},"content":{"rendered":"

When dealing with heavy workload functionalities that can have a big impact on web application performance, you may run an asynchronous task queue (scheduled or not). These asynchronous background tasks can not only drastically improve the scalability of the application by moving those highly consuming operations to the background, but also improve the functionality usability. Once spread on different components, each with its own responsibility, your code can even look cleaner, more isolated and can be easier to maintain.<\/p>\n

This article introduces a few topics regarding a prebuilt architecture using Django<\/a>, Celery, Docker, and AWS SQS. The codebase is available on Github<\/a> and you can easily follow the README steps to have the application up and running with no effort.<\/p>\n

The following section brings a brief overview of the components used to build the architecture.<\/span><\/p>\n

Components description<\/b><\/h2>\n

Message broker<\/h4>\n

\"\"<\/a><\/p>\n

Message broker usage<\/p>\n

First things first. What is a message broker? Our always friendly Wikipedia says that: <\/span>It mediates communication among applications, minimizing the mutual awareness that applications should have of each other in order to be able to exchange messages, effectively implementing decoupling. <\/span><\/i>(2020)<\/span><\/p>\n

In other words, it is an intermediate layer in which applications can communicate with – read and\/or write messages – and by having that, it allows the possibility of building two decoupled applications that do not rely on each other.<\/span><\/p>\n

 <\/p>\n

Amazon SQS<\/h4>\n

\"\"<\/a><\/p>\n

Amazon SQS queue messaging flow<\/p>\n

What is AWS SQS? “<\/span>Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. […] Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.” <\/span><\/i>(Amazon, 2020)<\/span><\/p>\n

There are a few alternatives for SQS, like Kafka, Redis or RabbitMQ and all of them can be easily set up on Amazon using ElasticCache or AmazonMQ. Besides that, you do have the option of deploying your message broker on EC2 instances, which could be cheaper depending on the number of messages. But you should also consider the time and effort required to set it up on EC2 and the extra effort in case you decide to use it with Auto Scaling. <\/span><\/p>\n

What I see as the main benefit of using SQS is that it is fully managed by AWS and you pay on-demand according to your usage and if your application or feature requirements are according to SQS usage, it is very straightforward to configure and use it.<\/span><\/p>\n

 <\/p>\n

Celery<\/h4>\n

\"\"<\/a><\/p>\n

Celery library logo<\/p>\n

What is Celery? “<\/span>Celery is an asynchronous task queue\/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well. […] Tasks can execute asynchronously (in the background) or synchronously (wait until ready).” <\/span><\/i>(Celery, 2020)<\/span><\/p>\n

Essentially, Celery is used to coordinate and execute distributed Python<\/a> tasks. It allows the possibility of moving a specific code execution outside of the HTTP request-response cycle. This way, your server can respond as quickly as possible for a specific request, as it spawns an asynchronous job as a worker to execute the specific piece of code, which improves your server response time. Also, with Celery you have the option of running jobs in the background on a regular schedule.<\/span>
\n\"\"<\/a><\/p>\n

Putting things together<\/b><\/h2>\n

To get everybody on the same page: AWS SQS is the message broker chosen here and Celery is the component responsible for orchestrating – consume: read and write – the message queue. Since Celery is a library, it needs to be set up on top of Django.<\/span><\/p>\n

Note: The source code used in this blog post <\/span><\/i>is available on GitHub<\/span><\/i><\/a>. <\/span>It assumes the following folders\/app structure:<\/span><\/i><\/p>\n

 \n.\n\u2514\u2500\u2500 src\n    \u251c\u2500\u2500 app\n    \u2502   \u251c\u2500\u2500 __init__.py\n    \u2502   \u251c\u2500\u2500 settings.py\n    \u2502   \u251c\u2500\u2500 urls.py\n    \u2502   \u2514\u2500\u2500 wsgi.py\n    \u251c\u2500\u2500 manage.py\n    [\u2026]\n<\/code><\/pre>\n

\nDocker isn’t necessary for the system to work, but it makes it a lot easier to reproduce the final system architecture locally and it also serves as a guide when configuring the system architecture on a cloud computing service environment.<\/span><\/p>\n
On the docker-compose file below, there are 5 configured services:<\/span><\/p>\n
 \nversion: \"3\"\nservices:\n  web:\n    build: .\n    volumes:\n      - .:\/usr\/src\/app\n    ports:\n      - 8000:8000\n    environment:\n      ENV: development\n    depends_on:\n      - db\n      - sqs\n\n  worker:\n    build: .\n    volumes:\n      - .:\/usr\/src\/app\n    command: bash -c \"cd src\/ && celery -A app worker --loglevel=debug\"\n    depends_on:\n      - web\n      - sqs\n\n  beat:\n    build: .\n    volumes:\n      - .:\/usr\/src\/app\n    command: bash -c \"cd src\/ && celery -A app beat --loglevel=debug\" \n    depends_on:\n      - web \n      - sqs \n\n  db:\n    image: postgres:12\n    ports:\n      - 5432:5432\n    volumes:\n      - .\/docker\/db\/pgdata:\/var\/lib\/postgresql\/data\n    environment:\n      POSTGRES_USER: postgres\n      POSTGRES_PASSWORD: postgres\n      POSTGRES_DB: app\n\n  sqs:\n    image: roribio16\/alpine-sqs\n    ports:\n      - 9324:9324\n      - 9325:9325\n    volumes:\n      - .\/config\/elasticmq.conf:\/opt\/config\/elasticmq.conf\n<\/code><\/pre>\n
Here is the lowdown on the docker-compose file:<\/span><\/p>\n