When Requests Don't Need to Wait: How to Use Queues in Web Applications

Users typically do not care how the server works. They just want the button they pressed to respond immediately, the page not to hang, and the final results to appear without having to restart the process. The problem is that many web applications still force a single request to handle everything at once: saving data, sending emails, creating PDFs, processing images, and logging reports.

This is where queues come in handy. A queue is a line of jobs that can be processed by workers outside the main request flow. With this pattern, the application simply receives the user's request, stores the jobs that need to be done, and then returns a response more quickly. Heavy jobs are processed later by a dedicated process.

Problems with Too Many Jobs in One Request

Imagine a user uploading proof of payment. After the file is received, the server may need to do several things:

• store the file in storage;
• check the size and format;
• resize the image;
• send email notifications;
• update the order status;
• log the activity for audit purposes.

If all of this is done before the server sends a response, the user has to wait for the entire sequence to finish. When one of the processes is slow, the request gets held up. Even if the email delivery fails, the application may appear as if the entire transaction failed, even though the order data has already been saved.

This issue is not just about speed. Long requests can consume connections, trigger timeouts from web servers or proxies, and make traffic spikes feel heavier than they should.

Understanding Queues with a Simple Analogy

A queue can be imagined like a ticket number at a counter. The web application is responsible for receiving requests and providing a ticket number. Workers are responsible for calling that number and working on tasks one by one.

In technical terms, a job is a unit of work that needs to be processed. Examples include SendWelcomeEmail, GenerateMonthlyReport, or ResizeUploadedImage. Jobs are stored in the queue, and then workers take them for execution.

The storage medium for the queue can be a MySQL table, Redis, or a dedicated queue service. For small applications, a database table may be sufficient. For higher job volumes, Redis is usually more suitable as it is designed for fast in-memory operations.

Common Workflow

The basic pattern can be summarized in four steps:

1. The user performs an action, such as submitting a registration form.
2. The application saves the main data and creates a new job.
3. The server immediately sends a response that the request has been received.
4. The worker takes the job and processes it in the background.

A simple example is that the account registration process does not need to wait for the welcome email to be sent. The application can save the account first, add the email sending job, and then display a message that the account has been successfully created.

user = createUser(formData)
queue.push({
  type: "send_welcome_email",
  userId: user.id
})
return response("Account successfully created")

This example is just an illustration of the concept. In a real application, jobs should carry the minimum necessary data, such as the user ID, rather than the entire user object. Workers can fetch the latest data from the database when the job starts processing.

Queues Do Not Mean All Jobs Must Be Asynchronous

Queues are suitable for jobs that do not need to be completed before the user can proceed. Email delivery, thumbnail creation, data export, synchronization with external services, and report generation are common examples.

Conversely, do not move everything to the queue. Form validation, access checks, and saving core data usually need to be completed before a response is given. Users need assurance of whether the main transaction succeeded or failed.

A simple guideline: if the job determines whether the main transaction can be considered successful, perform it within the request. If the job is merely a follow-up, it can be considered for the queue.

A Often Overlooked Aspect: Retry and Idempotency

Workers can fail. The connection to the email service may drop, third-party APIs can return errors, or the target server may be busy. Therefore, the queue system needs to support retry, which means trying the job again after a failure.

However, retries can be dangerous if jobs are not designed safely. For example, a balance topping job may fail after successfully sending a request to the bank but before receiving a response. When retried, the transaction could be sent twice.

The solution is to make operations idempotent. This means that running the same job multiple times still results in the same final effect. One common way is to use an idempotency key or a unique transaction ID that is checked before the operation is executed.

For emails, duplication may just be annoying. For payments, stock reductions, or order creation, duplication can be a serious issue.

Dead-Letter Queue and Failure Logging

Not all jobs will succeed after several retries. Jobs that continue to fail should be moved to a dead-letter queue, which is a special place for jobs that require manual inspection or separate handling.

Do not just log a message saying "job failed." Store information that helps with diagnosis, such as job type, related data ID, number of attempts, time of failure, and error messages. Avoid including passwords, tokens, or sensitive personal data in the logs.

A simple dashboard to view failed jobs is often more useful than waiting for users to report that their emails or reports were never received.

What Does This Mean for Us?

Queues are not just a technique to make applications look fast. Queues help separate two types of work: immediate responses needed by users and follow-up work that can be processed later.

This separation makes the architecture easier to develop, but it also adds responsibilities. Teams need to monitor workers, manage retries, prevent duplication, and ensure data consistency. An unmonitored asynchronous system merely shifts problems from the user's screen to a less visible place.

What You Can Do Now

1. Note the endpoints that are often slow and identify non-essential jobs within them.
2. Choose one simple job, such as email delivery, as the first candidate to move to the queue.
3. Add job statuses: waiting, processing, succeeded, or failed.
4. Test retry scenarios and ensure jobs do not create duplicate data.
5. Create logs sufficient to find the cause of failures without leaking sensitive information.
6. Set up monitoring for workers and the number of jobs piling up.

Start with one small flow. Queues will provide benefits when used for clear problems, not when added just because they sound more modern. With the right boundaries, applications can respond faster without sacrificing the reliability of the processes behind the scenes.

– Rio Yotto @rioyotto