integration-error-observability

Integration Error Observability

You are designing what an external developer can find out about their own broken integration, without opening a support ticket. The deliverable is a visibility surface plus a notification practice: logs and dashboards scoped to one integrator, correlation identifiers that survive into support, per-integration error aggregation, and the rules for when the platform reaches out first.

The asymmetry is the whole problem. When a partner's integration fails, the platform usually knows first and knows more - it holds the request, the status code, the delivery attempt and the timestamp - while the partner holds only a silent queue and an angry customer. Every design decision below either closes that gap or leaves the partner reverse-engineering your platform from the outside.

Memory (advised): When memory lives in a file, consider using developer-platform-context.md; if a different memory system is in use, rely on that instead. The file is an advisory reference, not a mandatory requirement. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.

Clarifying questions

Ask these before designing anything. Each answer moves a later ranking. Batch them - this is a tactical design task, not a strategy interview.

1. What can an integrator see today without asking you? Send the URL if any surface exists. "Nothing" is a valid and common answer.
2. Who breaks: engineers at a single customer integrating for their own use, commercial partners whose app runs across a fleet of installs, no-code and agent builders, or a mix? (see next section)
3. Classify the last 20 integration support tickets: request not found, webhook never arrived, silent data mismatch, auth or credential failure, rate-limit confusion. The shape of that pile decides which surface to build first.
4. Is every inbound request already tagged with an integration identity - API key, app or client ID - at ingestion? Aggregation is only ever as granular as the tag it is keyed on.
5. Do webhooks exist, and does the delivery machinery already record per-attempt outcomes? (decides the default rung in step 2)
6. When an integration breaks, does the partner lose money within hours, or is it noticed at the next reconciliation? (decides the notification posture in step 5)
7. What can you legally show? Do request and response bodies carry PII, secrets, or data belonging to another tenant, and who signs off on exposing them to a third party?
8. Deadline, payoff shape and effort ceiling: by what date must this be live, is it a one-off fix after a bad support quarter or a compounding platform surface you will run for years, and what can you spend - engineering weeks, log storage, an on-call rotation to answer the alerts you start sending?

Question 8 exists because every menu below diverges sharply on effort and on how long the payoff lasts. A hard deadline promotes whatever you already record and merely have to expose, while a compounding-surface mandate promotes the rungs that need new storage, new state, or a new outbound channel.

Question 7 can delete rungs outright rather than reorder them - a legal answer of "those bodies can never leave our tenancy boundary" removes payload-level logging from the menu instead of demoting it.

If your harness has persistent memory, store the surface's core decisions - which logs are exposed, the retention window, the identity the aggregation is keyed on, the alert thresholds and cooldown, the redaction rules - so a later run adding an event type or tuning an alert starts from the design instead of re-deriving it.

Integration ownership

How many end users sit behind the failing integration decides whether a dashboard is enough:

• Single-account integrators - a customer's own engineers wiring your API into their systems. One account, one dashboard, one set of credentials. A good self-serve log surface genuinely finishes the job for them: they will go look when something breaks, because their own users are already complaining.
• Partners and ISVs with a fleet of installs - their app runs across tens or thousands of customer accounts, and each install can fail independently. A per-account dashboard is structurally useless to them: nobody watches a thousand dashboards, so they need cross-install aggregation and push notification instead, or they learn about breakage from their own customers. This is the segment that makes steps 5 and 6 load-bearing rather than optional.
• No-code and agent builders - assemble a flow, cannot read a stack trace, and will not correlate anything. They need one plain-language failure list scoped to their flow, and a "what do I do about it" line next to each entry. Detail that helps the other two segments reads as noise here.

Serve every segment your answer to question 2 names. A surface designed only for the first segment is the documented default failure: the platform ships logs, calls integration visibility done, and its partner base still finds out about outages from end users.

Workflow

1. Audit what an integrator can see today.
2. Expose per-integration failure detail.
3. Make the correlation ID reach support.
4. Aggregate error rates per integration identity.
5. Decide the notification posture.
6. Build the provider-side fleet view.

Each step has a section below, in order.

1. Audit what an integrator can see today

Start from the support pile (question 3), not from an architecture diagram. For each of the last 20 integration tickets, answer one question: could the integrator have answered this themselves with a surface you could reasonably ship? The share that says yes is your baseline, and it is the number every later step is trying to move.

• Walk each failure class separately: synchronous API errors, webhooks that never arrived, auth or credential expiry, rate-limit rejections, and silent data mismatches. Platforms routinely ship a surface for one and assume it covers the others.
• Check the failure that produces no error at all: an integration that stopped calling you entirely. Nothing in an error log shows an absence, so a volume-drop signal is a separate design item, not a side effect of error logging.
• Record retention honestly. A log an integrator cannot reach back to the moment of the incident is a log that generates a support ticket anyway.
• Note who can see it. A surface that only the account owner can open fails the partner whose engineer is not a user of that account - an access-model gap, not a logging gap.

Write the audit as a table of failure class against current visibility, and carry it into every step below. The gaps are the build order.

2. Expose per-integration failure detail

Three distinct surfaces exist here, and conflating them is the common design error. Stripe's developer dashboard is the reference implementation to copy the shape of, and it keeps them separate (sourced: Stripe docs):

• Request log - every API call with its outcome. Answers "did my call work".
• Event log - the business-level events those calls produced. Answers "what did it change".
• Webhook delivery log - every delivery attempt per endpoint with the failure reason. Answers "did you tell me about it".

An integrator debugging in the dark needs whichever one matches their failure class.

Rungs, from what you can expose cheapest to what costs most to build:

• value (support tickets deflected): request log with per-request detail > delivery-attempt log > replay > failure counts only
• effort: replay > request log with per-request detail > delivery-attempt log > failure counts only
• compliance cost: request log with per-request detail > delivery-attempt log > replay == failure counts only - exposing request and response bodies to a third party triggers a data-classification and redaction review, and it is close to irreversible: once integrators build tooling against a field set you cannot quietly withdraw it.
• efficiency: delivery-attempt log > request log with per-request detail > failure counts only > replay

Default rung: expose the delivery-attempt log you already write, when the platform sends webhooks. The delivery machinery records every attempt and its failure reason regardless. Exposing it is mostly an access-control and interface job, and it retires the single most support-expensive question in webhook platforms ("your webhook never arrived").

Where there are no webhooks, the request log is the default instead.

Move up to the full request log - per call: timestamp, endpoint, status code, error code, request ID, and as much of the payload as question 7 permits - as soon as the audit shows synchronous API failures in the ticket pile. This is the rung that covers every integrator segment rather than only the webhook consumers.

Failure counts alone are the floor rung: a number per integration with no detail behind it. It is cheap, and worth shipping the same week if nothing exists today. But understand what it buys: it tells a partner something is wrong and gives them no way to act, which converts a silent failure into a support ticket rather than into a fix.

The starved option: replay. Letting an integrator re-send a failed event or re-run a request recovers value no other rung measures - the deflection axis above does not count it. But it carries the highest effort on the menu: idempotency guarantees, a bounded window, and a clear relationship to your own retry schedule. It loses every efficiency round as a result.

Promote it when the audit shows integrators regularly losing events during their own downtime, which is exactly when a dashboard that only explains the loss is cold comfort.

Design it as independent of automatic retry. On Stripe, a failed event can be manually resent for up to 15 days after creation, and doing so does not cancel the platform's own retry schedule (sourced: Stripe docs). So a partner's manual replay never fights the platform's machinery.

This ranking is a default, not a law. Re-rank it against what you already know:

• A platform with no webhooks has no delivery log to expose, and starts at the request log.
• A team already storing full request/response bodies for its own debugging gets the request-log rung for the cost of an access-scoped view.
• A "no bodies leave our boundary" answer to question 7 deletes payload detail from the menu, leaving metadata-only logging as the ceiling rather than a compromise.

See references/visibility-surfaces.md for the three-surface taxonomy expanded, the fields a log row has to carry, a good/bad log-row pair, replay semantics, and the tenant-isolation and redaction rules.

3. Make the correlation ID reach support

Two identifiers, kept separate in your vocabulary:

• Request ID - identifies one HTTP call and its response.
• Correlation ID - identifies a whole multi-call transaction that may span many request IDs.

Most platforms need both and ship only the first.

The convention is X-Request-Id carrying an opaque random value - a UUIDv4 by default, since it is collision-safe under load and leaks nothing, unlike a sequential or timestamp-derived identifier. Named platform variants exist (GitHub's request-id, Vercel's X-Vercel-Id, Cloudflare's Cf-Ray) and are citations, not requirements. The W3C traceparent header is the standards-track alternative that carries a span hierarchy and sampling flag instead of a bare string.

• value (a support ticket resolved in one lookup): searchable in the developer-facing log > echoed in the response header > present in the error body > propagated to sub-processors
• effort: propagated to sub-processors > searchable in the developer-facing log > echoed in the response header == present in the error body - that tie is genuine rather than an evasion: both write an identifier you already hold into the response you are already serialising, in the same code path, with no new storage behind either.
• efficiency: echoed in the response header > present in the error body > searchable in the developer-facing log > propagated to sub-processors

Default rung: echo the ID in the response headers of every response, success and failure alike. An integrator cannot paste an identifier into a support ticket if it was never returned to them, and this rung is usually a middleware change measured in hours. Success responses matter too - plenty of tickets are about a call that returned 200 and did the wrong thing.

Promote to searchable-in-the-log the moment step 2 ships. These two rungs are the pair that actually closes the loop: the header hands the integrator a string, the log lets them resolve it themselves before they ever write to you. Shipping the header alone produces a distinctive failure - integrators dutifully quoting an ID nobody outside your platform can look up.

The error body rung exists but belongs to a sibling: samber/developer-platform-skills@api-error-design owns the shape of the error payload, including whether the ID rides in it. Coordinate rather than redesign it here.

The starved option: propagation to sub-processors. Forwarding the same identifier onward when your platform calls a payment processor, carrier or other downstream provider, and keeping it on retries, is high effort and invisible until it matters. Promote it when your support answer today is "that failure came from our provider, we cannot see it" - at that point one lookup spanning the whole chain is the difference between a resolved ticket and a handoff.

This ranking is a default, not a law. Re-rank it against your stack:

• A platform already emitting W3C trace context internally gets propagation nearly free, which flips the effort line.
• A platform whose failures are overwhelmingly its own code gains little from propagation regardless of cost.

See references/correlation-id-conventions.md for header conventions, request-versus-correlation-ID modelling, propagation rules, the support-lookup workflow, and a negative example.

4. Aggregate error rates per integration identity

A platform-wide error rate can look healthy while one specific integration is failing 100% of its calls. Per-identity aggregation is what makes that visible, and the architecture is consistent across API-monitoring practitioner sources (sourced: Zuplo, Last9, dotcom-monitor, Odown):

1. Tag at ingestion, not after the fact - API key, app or client ID, attached to the request as it arrives. Retrofitting the tag later means your history is unqueryable for exactly the period you want to investigate.
2. Split by status-code family rather than one blended rate - 4xx (the integrator's payloads or auth), 5xx (yours), and 429 isolated on its own because rate-limit rejection is neither party's bug and has a d

...