Duplicate Requests, Messy Data: How to Make API Processes More Retry-Resistant

Duplicate requests are one of the issues that often only become apparent after an application is used in real conditions. Users may press the pay button twice, the connection may drop after the server receives the request, or mobile applications may automatically retry requests because they think the previous request failed. If the endpoint is not designed to handle these conditions, a single user action can result in two orders, two bills, or two data changes.

This is where the concept of idempotency becomes important. Simply put, an idempotent operation is one that produces the same final state even if the same request is sent multiple times. This concept is very useful for APIs, payment systems, checkout processes, webhooks, and asynchronous jobs.

Why can requests be sent more than once?

Request repetition does not always mean the user made a mistake. There are several common causes:

• The user repeatedly presses the submit button because the page appears unresponsive.
• The internet connection drops after the request reaches the server, but before the response returns to the browser.
• The client automatically retries when it receives a timeout or network error.
• The load balancer, message queue, or worker retries jobs that are considered failed.
• Webhooks from external services are resent because the sender has not received a successful response.

The problem is that the server often cannot distinguish between new requests and repeated old requests. Two requests with the same content are not necessarily considered the same operation. Therefore, the application needs a special marker called an idempotency key.

What is an idempotency key?

An idempotency key is a unique value created by the client to mark a single operation. This value is sent via the HTTP header, for example, Idempotency-Key: 7f2c.... The server then stores the key along with the results of processing the request.

If a request with the same key comes again, the server does not process the operation from the beginning. The server simply returns the previously stored result. This way, the client can retry without creating new data.

Imagine you are ordering food at a counter. The order number is not just a queue marker, but also the identity of the transaction. If you ask twice using the same order number, the staff does not create a new order. They simply show the status of the existing order.

Simple flow example

For example, an application has an endpoint to create orders:

POST /api/orders

The client creates a unique key before sending the request:

Idempotency-Key: order-8b2d9e21

The server then follows this flow:

1. Checks if the key has already been stored.
2. If not, the server starts the transaction and stores the key with the status processing.
3. The server creates the order, stores the result, and then changes the status to completed.
4. If the same key is sent again, the server returns the result of the existing order.

In practice, the key should be created by the client for a single action, not recreated every time a retry occurs. If the application creates a new key for each attempt, the server will consider each request as a different operation.

Don't just check the key

Storing the idempotency key alone is not enough. The server also needs to ensure that the key is not used for different request contents. For example, if the key order-8b2d9e21 is first used to purchase product A, then used again for product B. This condition should be rejected as it can lead to confusing behavior.

One common approach is to store a hash of important parts of the request, such as user ID, amount, currency, and order details. When the same key is used again, the server compares the hash. If they differ, the server returns an error like 409 Conflict.

Example of data structure that can be stored:

idempotency_key: order-8b2d9e21
user_id: 42
request_hash: a91f...
status: completed
response_code: 201
response_body: {...}
expires_at: 2026-09-16 10:00:00

How to implement it in PHP and MySQL?

For PHP and MySQL-based applications, you can create a special table to log idempotent operations:

CREATE TABLE idempotency_keys (
  id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
  user_id BIGINT UNSIGNED NOT NULL,
  idempotency_key VARCHAR(100) NOT NULL,
  request_hash CHAR(64) NOT NULL,
  status VARCHAR(20) NOT NULL,
  response_code INT NULL,
  response_body JSON NULL,
  created_at DATETIME NOT NULL,
  updated_at DATETIME NOT NULL,
  UNIQUE KEY unique_user_key (user_id, idempotency_key)
);

The unique index is important because it helps the database prevent two processes from inserting the same key simultaneously. Checking at the code level alone has a race condition gap, where two requests read the condition "not present" at nearly the same time and both create transactions.

Use database transactions to combine key logging and business data creation. If the process fails, the status can be changed to failed or the idempotent record can be cleaned up as needed. However, do not delete it carelessly if deletion could cause retries to process the transaction again.

How to handle processing status?

A more complicated situation occurs when the first request is still being processed, and then a second request with the same key comes in. The server needs to have clear rules. Some options that can be used:

• Return 409 Conflict with a message that the operation is still in progress.
• Make the client wait a moment and then try again.
• Return the job status if the process is done asynchronously.
• Use a lock with a timeout so that stuck processes do not lock the key forever.

Do not leave the processing status without a timeout. If the worker dies after recording that status, the next request could continue to be considered in progress. Store the last update time and prepare a recovery mechanism.

Idempotency is not the same as preventing all duplication

The idempotency key protects a single identical operation from repetition. However, this feature does not automatically prevent users from creating two different orders by pressing the buy button at different times. If the business need is to prevent identical orders within a certain time frame, you still need additional rules, such as validating cart status or duplication limits.

Idempotency is also different from input validation. Validation ensures that incoming data is correct, while idempotency ensures that repeating an operation does not produce additional side effects.

What you can do now

1. Identify endpoints that cause side effects, such as creating orders, payments, shipping, or balance changes.
2. Add support for the idempotency key header on those endpoints.
3. Create a logging table with a unique index in the database.
4. Store the request hash so that the key cannot be used for different data contents.
5. Test timeout scenarios, retries, double clicks, and two simultaneous requests.
6. Determine the key retention period based on the risk of the operation. Payment processes usually require a longer retention period than contact forms.

In web applications, retries are not something that can be completely avoided. Networks are not always stable, browsers can resend requests, and distributed systems can run jobs more than once. By designing endpoints to be idempotent from the start, applications become more resilient to real-world conditions—not just when everything runs perfectly.

– Rio Yotto @rioyotto