Leticia Sainz
Case study 02 · Distributed Systems

When the payment succeeds, but the order does not.

A functional and technical investigation of a distributed order-processing failure across REST APIs, microservices and asynchronous messaging.

API TestingRESTPostmanMicroservicesLogsIntegration TestingAWS SNS/SQSRoot Cause AnalysisTechnical Analysis
01

System

Follow the business transaction, not just the endpoint.

01Frontend
02Orders API
03Payment Service
04AWS SNS
05AWS SQS
06Order Processor
07Order DB
08Notification Service
02

Incident

A successful payment with an incomplete business outcome

Payment
ApprovedPAY-99821
Order
Pending paymentORD-78451
Notification
Not sent

Customer has been charged.

“Your payment was successful. We are processing your order.”
03

Hypotheses

Before repeating the test, map where state can diverge.

  1. H01Payment Service did not publish the event
  2. H02SNS failed to route the event
  3. H03SQS message was not consumed
  4. H04Consumer rejected the event because of invalid contract data
  5. H05Order update failed at persistence layer
  6. H06Order was updated but a downstream read returned stale data
04

API validation

Both synchronous APIs pass — the transaction still fails.

POST /api/v1/orders
{
  "customerId": "CUS-1842",
  "items": [
    {
      "productId": "PROD-901",
      "quantity": 2,
      "unitPrice": 24.9
    }
  ],
  "currency": "EUR"
}
201 CreatedOrder = PENDING_PAYMENT
POST /api/v1/payments
{
  "orderId": "ORD-78451",
  "amount": 49.8,
  "currency": "EUR",
  "paymentMethod": "CARD"
}
200 OKPayment = APPROVED
A successful API response does not guarantee a successful business transaction.
05

Event contract

Inspect the event that connects the services.

payment.completed · evt-78310
{
  "eventType": "payment.completed",
  "eventVersion": "2.0",
  "eventId": "evt-78310",
  "timestamp": "2026-09-10T09:42:18Z",
  "data": {
    "orderId": "ORD-78451",
    "paymentId": "PAY-99821",
    "amount": 49.8,
    "currency": "EUR",
    "status": "APPROVED"
  }
}
Consumer contract
Supported
1.0
Received
2.0
Rejected

Unsupported eventVersion: 2.0

06

Log analysis

Follow the evidence service by service.

order-processing · production-like trace
PAYMENTPayment approved PAY-99821
PAYMENTPublishing payment.completed event
SNSPublish successful messageId=MSG-7719
SQSMessage delivered to order-processing-queue
ORDER_PROCESSORReceived message MSG-7719
ORDER_PROCESSORParsing event payment.completed
ORDER_PROCESSORERROR Unsupported eventVersion: 2.0; expected 1.0
ORDER_PROCESSORMessage moved to DLQ
07

Root cause

The defect is in the contract between services.

01Payment ✓
02Event publication ✓
03SNS ✓
04SQS ✓
05Consumer ✕
06Order update — not executed
07Notification — not executed
The producer publishes eventVersion 2.0, while the Order Processor supports only 1.0. The rejected message moves to the DLQ, leaving the customer charged and the order pending.
08

Defect

Translate technical evidence into business impact.

IDORD-2173
TitleOrder remains PENDING_PAYMENT after successful payment due to unsupported event contract version
SeverityCritical
AffectedPayment Service · Order Processor
ExpectedAPPROVED payment → order CONFIRMED → confirmation notification
ActualPayment APPROVED; event rejected; order remains PENDING_PAYMENT
Business impactCustomer charged without confirmed order; possible support contacts, reconciliation work, duplicates and loss of trust.
09

Regression & contract tests

Turn the incident into stronger quality coverage.

Contract testing

Validate producer/consumer compatibility before deployment.

Integration testing

Verify event publication, delivery and consumption.

E2E testing

Checkout → payment → order confirmation → notification.

Monitoring

Alert when messages enter the Dead Letter Queue.

Postman collection
PassPOST /orders · 201 Created
PassPOST /payments · 200 OK
FailGET /orders/ORD-78451 · expected CONFIRMED
Additional scenariosUnsupported versionMissing orderIdDuplicate eventDelayed eventConsumer unavailable
10

Idempotency

The same event must never create the same effect twice.

Automated behaviour
evt-78310 × 2

If event evt-78310 is delivered twice, the order should transition to CONFIRMED only once.

processedEvents.length = 1
11

Investigation outcome

From symptom to root cause

Failure boundary isolated

The investigation narrowed the issue from the complete order flow to the interaction between the payment event and its consumer.

Root cause identified

The producer published payment.completed using event contract v2.0, while the Order Processor only supported v1.0.

Technical evidence correlated

API responses, event payloads, service logs and DLQ behaviour were connected to validate the hypothesis.

Business impact explained

A technically successful payment could leave the customer with a charged payment but an order still in PENDING_PAYMENT.

Regression gap exposed

Producer and consumer contract compatibility was identified as a critical integration risk.

Preventive coverage defined

Contract testing, integration testing, E2E validation, idempotency and DLQ monitoring were identified as complementary controls.

12

Takeaway

Quality is an end-to-end system property.

Quality is not only validating individual services. It is understanding whether the entire system delivers the expected business outcome.