Building robust APIs - lessons learned from the payment world
Working in the payment industry means handling money and payment transactions, where every failure has immediate consequences. A failed API call is not just a technical error — it means a customer’s checkout did not go through, a refund was not processed or a merchant could not see their transactions. Systems must be highly available and fault-tolerant. From the merchant-facing APIs to internal service communication and external integrations, APIs are everywhere in modern systems. And they must be robust.
In this article, I will share practical lessons learned over the years, highlighting challenges I have faced and the solutions that worked. The focus is on HTTP APIs, with additional coverage of async APIs and webhooks to provide a comprehensive view of robust API design.
Start with an API contract
Over the years, I have learned that a robust API should not start with code alone. It should start with a contract.
Think of an API contract the same way you would think of a contract in real life. Both parties agree on what they will do and what they can expect. In software that contract is explicit, reviewable and shareable.
Depending on the kind of API, that contract can be expressed with different tools and formats. For HTTP APIs, OpenAPI is usually my first choice. It is easy for developers to understand and well-supported by tooling. For asynchronous message-based systems, AsyncAPI is a good fit, but you can also use OpenAPI here to define the Schema of the messages. For event schemas or message payloads, formats like Avro or JSON Schema can work well.
The exact format is less important than the principle itself. Define the contract clearly before implementation becomes too far advanced. That makes the interface explicit, reviewable across teams, and serves as documentation that stays in sync with the code.
I am a big fan of the API-first approach: define the contract first, then generate code from the specification. This is not just for external APIs. Internal service-to-service communication also benefits from this.
Here is why this matters for robustness: when you generate code and validation from the same source of truth, you get more than just documentation. You get request validation that is automatically kept in sync with the contract. You get type safety in your server and client code. You reduce the chance that someone forgets to implement validation or implements it differently on different sides of the API.
One important principle: never silently manipulate the input you receive. If incoming data is wrong, incomplete, or malformed, the API should reject it clearly and report what was wrong. Changing data on the fly may look convenient, but it creates surprises, hides bugs in the client, and can become a security issue. Let the client know when something is wrong so they can fix it.
Clear resource design reflects the business domain
A robust API should be easy to understand without reading a full manual. That does not mean it needs to be minimal at all costs. It means the API should make the important business concepts visible and understandable.
When I start designing an API, I think about the domain first. In payment systems, that means merchants, commerce cases, checkouts, payments, refunds, captures and many other concepts. The API should reflect that structure. A developer using the API should immediately see how things fit together.
Consider two approaches:
The first is flat: GET /v1/payments/{paymentId}
The second is nested: GET /v1/merchant/{merchantId}/commerce-cases/{commerceCaseId}/checkouts/{checkoutId}/payments/{paymentId}
The flat approach is shorter, which looks attractive on first sight. And for some systems, it may be perfectly fine.
But the nested approach tells a clearer story. You can see the relationship between a payment and the checkout it belongs to. You can see which merchant owns the commerce case (a logical wrapper around connected checkouts). The URI itself communicates the business hierarchy.
Yes, the URIs get longer. But in internal systems and operational APIs, that clarity is worth the extra length. It also makes it easier to add authentication and authorization rules based on the path structure.
The lesson is not “always nest everything”. The lesson is “let the domain model guide the API structure”. Do not flatten everything into collections just to save a few characters in the URL.
When to use action endpoints
In REST API design, there is often a strong push toward pure CRUD operations. Everything should be a resource, and every operation should map to POST, PUT or PATCH, GET or the new QUERY, DELETE. In theory, that is elegant. In practice, it does not always fit the business.
Sometimes what you really want to express is not “create a new object in a collection” but rather “perform an action on this existing object”. If the business domain says “refund a payment” or “capture a payment”, those are actions.
You can model actions in two ways and both can be valid depending on the context:
As an action endpoint:
POST /v1/payments/{id}/refundPOST /v1/payments/{id}/capture
This communicates: “here is an action you can perform on a payment”. It is clear, direct, and immediately understandable.
As a subresource:
POST /v1/payments/{id}/refundsPOST /v1/payments/{id}/captures
This models the outcome: “you are creating a new refund or capture object associated with this payment”. This can also be the right choice, especially if you often need to retrieve or query those objects later.
The important lesson is this: do not force your business into one style just for purity. Choose the modeling approach that best matches the business semantics. If a capture is conceptually a new object in the system, model it as a resource. If a capture is a business action on an existing payment, an action endpoint can be clearer.
Your API consumers will thank you for making the intent obvious.
Plan for versioning from the beginning
I have seen many projects start without thinking about API versioning. They launch version 1, everything works fine, and versioning feels like an unnecessary complication. Then a year (or two) later, consumers have built systems around the API. A new requirement comes in that would be much cleaner if the API looked different. Suddenly versioning becomes urgent and can be painful.
I prefer to think about versioning from the beginning, even if you do not use it right away. The simple act of including a version in the path signals to everyone that this is a contract, not an implementation detail.
My personal preference is path versioning:
GET /v1/payments
This approach has several advantages:
- It is easy to spot in logs and monitoring.
- It is easy to document and reason about.
- It is easy to run multiple versions side by side during migration.
- It makes it clear that the API is a contract that must be managed deliberately.
There are other versioning strategies: header-based versioning, query parameters or content negotiation. The most important decision is not which technique you choose. The important decision is that you choose one intentionally, document it and use it consistently.
If versioning is an afterthought, it becomes a source of confusion. Teams forget which version they are using. Deployments become risky because you cannot be sure what clients expect. If versioning is part of the design from day one, it becomes much easier to evolve the API safely without breaking consumers who are not ready to migrate yet.
Idempotency: build for retries
In payment systems, retries are not an edge case. They are a normal part of operations.
A customer refreshes the page too fast and the browser sends the request twice. A network timeout occurs midway through processing, so the client retries. A queue processor crashes after consuming a message but before confirming it, so the message gets redelivered. A webhook delivery fails, so the system retries it.
If the API cannot handle that safely, duplicates will eventually happen. A customer might be charged twice. A refund might be applied twice. A payment might be captured twice.
That is why idempotency is one of the most important building blocks of a robust API.
For HTTP APIs, this often means accepting an idempotency key — a unique identifier that the client provides. The server stores the result of the first request, and if the same idempotency key comes in again, it returns the same outcome without redoing the work.
POST /v1/payments
X-Idempotency-Key: unique-key-12345
{...request body...}
The same principle applies to async messages. If a message processor consumes a message and then crashes before confirming receipt, the message will be redelivered. The processor should be able to detect that it has already processed this message (using a message ID or similar) and skip the work.
And it also applies to webhooks. Webhooks are often delivered with retry logic, so webhook consumers must be built to handle duplicates gracefully.
The mindset shift is important: do not design your API to prevent retries. Design your API to handle retries safely. Make retries a normal part of the system, not an exception.
Use the right status codes and handle errors well
One of the biggest sources of confusion in APIs is the misuse of HTTP status codes.
I have seen APIs that return 200 OK for everything, putting error details in the response body.
Others use 500 Internal Server Error for business validation failures.
This makes it very hard to understand what actually happened.
Let me give you a concrete example. A merchant tries to capture a payment that was already captured. This is a valid HTTP request. The schema is correct. But the business logic cannot allow it.
Some APIs would return 200 OK with an error message in the body.
But that is wrong.
The request succeeded from a technical standpoint, but the business operation failed.
A better response is 422 Unprocessable Entity:
{
"status": 422,
"code": "PAYMENT_ALREADY_CAPTURED",
"message": "This payment has already been captured",
"details": {
"paymentId": "pay_123",
"capturedAt": "2026-10-02T10:30:00Z"
}
}
Why does this matter?
In operations, you want to track different kinds of failures differently.
If you look at the access log and see a 422, you immediately know that the request was technically valid but a business rule prevented it.
You can set up different alerting rules for 422 responses than for 500 errors.
You can build better dashboards and reports.
And also as a client of the API you want to know what happened.
When you get a 422, you immediately know that the request was valid but could not be processed.
When you would get a 200, you would need to check the body and see that this was a successful retry of an already processed capture.
You can build a simpler integration.
In general, I think API error handling should make these distinctions clear:
- 4xx errors: The request had a problem. Retrying with the same request will not help.
- 5xx errors: The server had a problem. The request might work if retried later.
Within 4xx, here are some common examples:
400 Bad Request: The request was malformed (missing fields, wrong format).401 Unauthorized: Authentication failed.403 Forbidden: Authentication succeeded, but the client does not have permission.404 Not Found: The requested resource does not exist.409 Conflict: The request conflicts with the current state (e.g., optimistic locking).422 Unprocessable Entity: The request is well-formed, but a business rule prevents processing.
The error response itself should also be consistent and helpful. It should include:
- A machine-readable error code (not just the HTTP status)
- A human-readable message
- Enough context to understand what went wrong (without exposing sensitive data)
Never just say "Error" or "Invalid request".
Give the developer enough information to understand what they need to fix.
Correlation IDs: trace requests across services
Picture this: a merchant reports that a checkout failed. It might have happened 10 minutes ago or 10 hours ago. Now you need to figure out why.
If your system is made of many microservices, that single checkout request probably triggered:
- an API call to a payment service,
- a database write,
- an internal message to a fraud service,
- a call to an external payment processor,
- and possibly webhooks to notify the merchant.
Without a way to connect all these events, you have to search through logs from 5 different systems, hoping to find events that happened at roughly the same time. That is painful and error-prone.
That is why I always use correlation IDs.
A correlation ID is a unique identifier that you attach to a request when it enters your system. You then propagate that ID through every internal call, message, and log entry.
Here is how it works:
- Client sends a request (with or without a correlation ID)
- If there is no ID, the API gateway or entry point generates one
- The ID is included in every log line
- The ID is passed in headers for internal HTTP calls
- The ID is included in async messages
- The ID is included in webhook payloads
- Downstream services log with the same ID
Now when a merchant reports an issue, you search for that one correlation ID in your logs and can see the entire journey of the request.
I have seen this save hours of debugging time. Without correlation IDs, a failed checkout can take a whole team to trace. With correlation IDs, one person can find the issue in minutes.
In the payment world, that matters a lot. A failed checkout is not just a technical event. It is a customer-facing incident that is costing the merchant money. The faster you can trace it and understand what happened, the faster you can recover and prevent it from happening again.

Async APIs and webhooks follow the same principles
Although this post focuses on HTTP APIs, a modern payment system rarely relies on HTTP alone. Many systems also depend on asynchronous message queues and webhooks. The same principles of robustness apply to all of them.
Async communication is powerful because it:
- Decouples services so they can work at different speeds
- Absorbs spikes in traffic without overwhelming downstream systems
- Enables long-running processes that do not need immediate responses
- Helps the system stay available even when dependencies are slow
But asynchronous systems also introduce new challenges. The coupling is looser, so it is easier to lose track of what is happening. Messages can be delayed, duplicated or lost. Webhooks might be delivered multiple times or never at all.
That is why the same design discipline applies:
- The contract should be clear and version-managed (using AsyncAPI or similar)
- Messages should be idempotent so retries are safe
- Dead-letter queues should catch messages that cannot be processed
- Retry policies should be deliberate and documented
- Correlation IDs should be included in every message
- Errors should be logged and made observable
If you treat async APIs with the same care as HTTP APIs, your system will be much more resilient.
In practice, this means using the same robustness patterns in async flows as in HTTP APIs: idempotency keys (or message IDs), correlation IDs and clear error handling. It also means deliberate retries, dead-letter queues and signed webhooks (for example with HMAC) to verify authenticity. The next section goes deeper into retries and dead-letter queue strategy.
Retries and dead-letter queues are foundational for async systems
In async systems, things go wrong all the time. A downstream service is temporarily unavailable. A database is locked. A timeout occurs.
The question is not whether failures will happen. The question is how your system will respond.
That is why every async system needs a clear retry strategy and a place to put messages that cannot be processed.
Retries: When a message cannot be processed, the system should retry it. But the retry should be controlled:
- How many times should we retry?
- How long should we wait between retries?
- Should we use exponential backoff to avoid overwhelming a recovering service?
- Should we give up after a certain timeout?
The answer depends on your business flow. If the failure is temporary (a service is restarting), retrying after a few seconds might be all you need. If the failure is because of missing data (waiting for information from another service), you might need to retry for hours or even days. If the failure is because of a programming error or data corruption, retrying will never help. Here you can send the message directly to a dead-letter queue.
Dead-letter queues: Once you have decided that a message should not be retried anymore, where does it go?
A dead-letter queue (DLQ) is a holding area for messages that the system could not process. It is not a failure, it is a safety mechanism. Instead of silently dropping the message or filling up logs with errors, the message goes to the DLQ where it can be inspected, debugged and potentially replayed once the underlying issue is fixed.
I have seen DLQs save entire systems. A bug in message processing caused messages to fail. The DLQ captured them all. Once the bug was fixed, we could replay the messages and recover without losing a single one.
Webhooks: For webhooks, the same idea applies. If a webhook delivery fails, the system should retry it. But it should do so with a clear backoff strategy and eventually give up and log the failure.
The lesson is this: in async systems, plan for failures from day one. Do not add retries and DLQs as an afterthought. Make them part of the architecture.
Define backward compatibility early
Backward compatibility is one of those topics that sounds obvious until two teams interpret it differently. I have been in meetings where one team said “we did not break the API” and another team said “our client code stopped compiling”. They were both right from their perspective.
That is why I think teams should define what backward compatibility means for their API.
Here is my personal perspective:
For a minor version bump (1.2.0 to 1.3.0): A request that worked with 1.2.0 should still work with 1.3.0. The JSON structure should still be valid. Required fields should still be required in the same way.
At the same time, internal implementation details might change. A code generator might rename classes or restructure packages. If the HTTP contract is stable but the generated code changes, that is not necessarily a breaking change to the API itself, even though it might require rebuilds in consuming services.
For a major version bump (1.x to 2.0.0): Breaking changes are acceptable and expected. You can change field names, restructure the data model, remove deprecated endpoints, and change behavior. But it is still worth doing this intentionally and documenting it well.
With path versioning (/v1 to /v2): The same logic applies. You can maintain multiple versions for a period of time, giving consumers a chance to migrate.
What breaks compatibility:
- Removing a required field (requests that used to work now fail)
- Changing the type of a field (int to string, for example)
- Changing the meaning of a field or status code
- Adding a new required field without a default
- Removing an endpoint that is actively used
What usually does not break compatibility:
- Adding an optional field to a response
- Adding an optional parameter to a request
- Adding a new endpoint
- Adding more specific error information
- Becoming stricter about input validation (rejecting invalid input that was previously accepted)
The key insight is this: if you define compatibility rules early and stick to them, evolution becomes much easier. Consumers know what to expect. You have permission to improve the API without worrying that everything will break.
Monitoring and alerting are not optional

Here is something I have learned the hard way: an API is not robust if nobody can see whether it is actually working.
I have seen teams ship APIs with impressive architectures and thoughtful design, only to go blind when they hit production. They had no idea how many errors were happening or how long requests were taking.
Monitoring and alerting are not something you add after the fact. They are part of the API design from the beginning.
What should you monitor?
- Request volume: How many requests per second? Is it growing? Are there unexpected spikes?
- Latency: How long do requests take? Is performance degrading?
- Error rates: How many requests fail? Broken down by error type (4xx vs 5xx)?
- Business-level metrics:
- How many payments succeeded?
- How many were rejected by business rules?
- How many refunds were requested?
- How long do checkouts take?
- Async-specific metrics:
- Message queue depth (how many messages are waiting?)
- DLQ size (how many messages could not be processed?)
- Retry counts and backoff behavior
- Webhook delivery success rates
What should you alert on?
Not everything that is worth monitoring is worth alerting on. An alert should mean “something is wrong and requires immediate attention”.
Good candidates for alerts:
- Error rate crosses a threshold (e.g., more than 5% of requests failing)
- A critical business metric stops working (e.g., zero payments processed in 5 minutes when that should not happen)
- DLQ is growing (messages are piling up that could not be processed)
- An external dependency becomes unavailable
The tooling:
There are many monitoring platforms: Prometheus, Datadog, New Relic, CloudWatch and others. The specific tool matters less than the discipline. You should have dashboards that you look at regularly. You should have alerts that you actually respond to. You should have runbooks that tell you what to do when an alert fires.
In the payment world, this is critical. An outage does not just mean “the API is down”. It means “merchants cannot accept payments”. It means revenue is not flowing. It means support tickets are piling up. The faster you can detect and respond to issues, the less damage there is.
Lessons learned over the time
If I could go back 10 years and tell myself one thing about API design, it would be this: the small details matter a lot.
Here are the lessons that have stuck with me:
Let the domain model guide the structure. Do not flatten everything into collections just to save a few characters in the URL. If the business hierarchy is complex, let the API reflect that.
Start with a contract. Define the API with OpenAPI or AsyncAPI before you start coding. Generate code and validation from that contract. It keeps everything in sync and catches errors early.
Never silently manipulate input. If data is invalid, reject it clearly. Do not try to fix it on the fly. Let the client know what went wrong so they can fix it properly.
Design for retries, not against them. Assume that every request might be retried. Make idempotency key support part of the design. Do not assume a request will only happen once.
Use the right status codes. A 200 OK with an error message is worse than a 422 with a clear error code. Status codes are part of your API contract. Use them deliberately.
Reject invalid input with clear errors. Do not just say “error”. Tell the client exactly what was wrong and how to fix it.
Share context across services. Use correlation IDs from the beginning. They are cheap and invaluable when debugging.
Plan for versioning from day one. Choose a strategy (path versioning, header versioning, etc.) and use it consistently. Do not let versioning become a crisis.
Define what compatibility means. Minor versions maintain backward compatibility. Major versions can break. Be explicit about this so consumers know what to expect.
Build for async systems. Use clear contracts (AsyncAPI or even also OpenAPI and just define the schema of the message), implement idempotent processing, build retry strategies, and always have a dead-letter queue.
Monitor from the beginning. Set up dashboards and alerts before you need them. You cannot fix what you cannot see.
These are not fancy architectural patterns or trendy technologies. They are the basic building blocks of systems that work reliably in production.
Conclusion
Building robust APIs is not about following one perfect architectural style. It is about making deliberate trade-offs and designing for the realities of production systems.
In the payment world, the stakes are high. An API outage is not just a technical problem. It means merchants cannot accept payments. It means transactions might be lost or duplicated. It means support teams are overwhelmed and customers are frustrated.
A robust API is one that is:
- understandable because the contract is clear and the structure makes sense
- predictable because error handling is consistent and well-defined
- observable because you can see what is happening in production
- safe to use under pressure because retries are handled, conflicts are detected, and nothing happens silently
- maintainable because versioning was planned from the beginning and breaking changes are managed deliberately
That is what makes an API valuable for merchants, for internal teams, and for everyone who has to operate it.
The good news is that none of this requires expensive infrastructure or complex frameworks. It requires discipline. It requires thinking about the edge cases. It requires planning before you code. It requires monitoring in production.
And in my experience, it is worth every bit of effort. An API that is robust is an API that the whole organization can trust. That is the real goal.
