Engineering Journal · ARCHITECTURE · July 29, 2025 · 17 min read

The Risk Lives Outside Your Repository

Every payment, bank, mailbox, and accounting API you call is a contract you did not write. The outage is only the loud version. The quiet version is a change you hear about from operations.

By Golam Sorwar, Tech Lead and Full Stack Engineer in Dublin.

I used to put integrations on the same list as features. Add Stripe. Add a bank transfer. Add QuickBooks. Add the mail API. The ticket estimated the happy path plus some error handling, and we called that the cost. The cost is not the adapter. The cost is that part of your product now fails for reasons that will never appear in your git history. A provider changes a webhook shape. A payout is delayed and nobody told your state machine. An OAuth refresh starts failing on a Sunday. Rate limits tighten in a region you do not live in. Your tests stay green. The organisation still stops. This is not a guide to any one vendor. Those guides have their place, and I have written the practical ones for payments and accounting. This is the architectural claim underneath them: an integration is a business dependency wearing a client library. If you do not design for that, you will keep being surprised that "the API works" and the week does not.

Context

The products I work on are integration-heavy because the organisation is. Money leaves through more than one rail. Invoices have to exist in an accounting system someone else owns. Email has to leave through a provider that will judge your reputation for you. Identity sometimes sits with a campus or a vendor. Mobile clients consume the same facts with a release cycle you do not command.

None of that is optional sophistication. It is the minimum set of relationships a real operation already has. Engineering did not invent the bank. Engineering invented the idea that the bank could be a function.

The team is small, so every integration has a human who "knows it." That knowledge is part of the risk. When they are away, the vendor dashboard becomes a haunted house. When they are present, we under-invest in making the failure obvious, because they can still read the runes.

The real problem

The apparent problem is reliability of HTTP. Timeouts, retries, mapping. Those are real and they are the part we already know how to talk about.

The deeper problem is that you have imported another organisation's release process, legal constraints, and operational habits into yours. They will change a field. They will deprecate a scope. They will have an incident during your peak, not during theirs. They will define success in a way that does not match your finance calendar. You cannot code review that. You can only decide how much of your product is allowed to be attached to it.

There is a second deeper problem: silent success. Many integrations fail by lying. They accept a request and process it later. They return 200 and then send the real answer to a webhook you were not ready for. They persist a customer in a slightly different shape than you sent. Your user sees "done." Your other system sees nothing. That gap is where operations invent spreadsheets.

I have also seen the cultural version. We treat an integration as finished when the demo works, because the demo uses a sandbox that is friendlier than production. Sandbox is not a smaller production. It is a different product. If your design only works there, you have a prototype with a logo.

Constraints

You cannot not integrate. Telling a college to stop using a bank, or a finance team to stop using their ledger, is not architecture. It is theatre.

You do not control their uptime, their versioning manners, or their support queue. SLAs, when they exist, are not the same as your user's patience.

Data has to remain explainable after a failure. "We will replay" is easy to say and hard to do if you never stored the inbound payload, the decision you made, and the external id.

Clients of your own API — including a Flutter app — will keep old behaviour. If your reaction to a vendor change is to change your public shape immediately, you have multiplied the blast radius.

A small team cannot wrap every vendor in a perfect anti-corruption layer on day one. If the standard is perfection, people will skip the layer and "just call the SDK" in a controller. The standard has to be the minimum that keeps failures diagnosable.

Options considered

Call the vendor from the request that needs it. Advantage: simple to write, easy to demo. Disadvantage: your user's latency is now their latency, and your error page is now their outage. Risk: retries in the request double-charge or double-create. Maintenance looks small until the afternoon an endpoint hangs. Business impact is immediate and public.

Put every vendor behind a full internal platform. Advantage: one language, one set of dashboards. Disadvantage: you have built a product for yourself that needs staff. Risk: the platform lags the vendor and people bypass it. Complexity is high. I have sympathy for this in a large org. I have suspicion for it as the first move in a small one.

Isolate each integration behind a narrow module, async where the user can wait, with stored intents and stored evidence. Advantage: you can retry without improvisation; you can show a human what you sent and what came back. Disadvantage: more moving parts than a controller. Risk: jobs pile up and you have only moved the outage into a queue. That is still better if the queue is visible.

Buy a unified "integration layer" product. Advantage: someone else maintains connectors. Disadvantage: you have added a fourth organisation to a three-organisation problem. Risk: mapping becomes a black box you cannot replay. Sometimes worth it. Often a way of postponing the moment you understand your own money.

Accept the vendor as the UI. Deep-link staff into Stripe or QuickBooks and keep your app thin. Advantage: less code. Disadvantage: you have trained the company to live in someone else's product, and you have no single story. I will do this for rare admin actions. I will not do it for the daily path.

Decision

We treat integrations as untrusted neighbours. The application is allowed to ask them to do something. It is not allowed to forget what it asked, or to assume the neighbour's status language is our product language.

That means an internal record first, a vendor call second, a recorded result third. Where the user can wait — email, invoice sync, non-blocking confirmation — the vendor call leaves the request. Where they cannot wait, we still write the intent before we talk to the network.

I chose that because I was tired of incidents whose only artefact was a stack trace and a memory. If you cannot answer "what did we send, what did they say, what did we conclude," you do not have an integration. You have a hope.

Why not a grand platform? Because a small team needs isolation more than it needs a brand. A module, a table of evidence, a job, a dashboard of stuck work. That is enough to stop the neighbour's worst day from becoming an unsearchable myth.

Implementation / Process

The pattern is repetitive on purpose. For each vendor we name the operations we actually use, not the ones in their marketing. Create payment. Refresh token. Post invoice. Send batch. Then we store an intent with our own id, send their id back into that row, and keep a short log of payloads we are allowed to keep.

Auth is infrastructure. Token refresh that lives in three jobs will fail in the fourth. Centralise it, encrypt it, alert when refresh fails, and do not log the secrets you just rotated.

Timeouts and retries are specific. A read can retry. A create cannot, unless you have an idempotency key they honour or you can safely ask "do you already have this?" before creating again. I have seen well-meaning retry middleware invent duplicate customers in an accounting system. The middleware was proud. Finance was not.

We map their states to ours in one place. The rest of the application is not allowed to switch on a vendor string. That sounds fussy until the vendor adds a state that means "successful but not really."

Observability is about stuck work and unknown states, not only error rates. A 99% success rate can still hide the ten invoices that did not sync, and those ten are the month.

When we add a new rail — another bank, another mail sender — we copy the pattern, not the old controller that "already did something like this." The second integration is how the first one's shortcuts get promoted into a standard.

// Intent first. The vendor is a witness, not the memory.
$intent = OutboundCall::create([
    'vendor' => 'accounting',
    'operation' => 'upsert_invoice',
    'idempotency_key' => $invoice->syncKey(),
    'status' => 'pending',
]);

$result = $accounting->upsertInvoice($invoice, $intent->idempotency_key);
$intent->record($result);

Problems and failures

We stored too little, then too much. Too little and you cannot replay. Too much and you have a pile of personal data in a debug table that nobody thought was a datastore. Retention belongs in the design, not in a later panic.

We trusted sandbox field names. Production added a required field that the sandbox never enforced. The first real customer of that path became the test. That is an integration testing failure and a humility failure.

Partial success toward two vendors in one job created a repair genre. If you post to accounting and then fail to update your own row, you will do it again, or you will think you did not do it. Split the work. Make each step restartable.

Operations found vendor dashboards faster than they found our admin, because our admin was shy about showing the evidence. If the human still has to leave your product to understand your product, the isolation layer is unfinished.

I under-communicated change windows. A vendor deprecation arrived in an email that looked like marketing. Someone has to own reading those, and it cannot be "whoever noticed." That is a process control, not a code control.

Trade-offs

We write more code than the SDK sample. That is the fee for being able to explain yourself. I will pay it for money, identity, and anything that emails thousands of people. I will not pay it for a one-off admin import.

We accept delay. Async is safer and less charming. Product has to be told that "sent" may mean "accepted by us" and not "accepted by the bank." If you cannot have that conversation, you will lie in the UI.

We accept that some rare actions stay in the vendor UI. That is a concession to team size. It is also a concentration of risk in those actions, so they should be rare on purpose.

Result

Incidents did not disappear. They became shorter to narrate. We could usually find the intent, the external id, and the last response, which changes the call with support from theatre into a ticket they can work.

New integrations still hurt. They hurt in a known way. That is the most I will claim. Anyone who promises that a pattern removes vendor risk is selling you a repository again.

The organisational result was a slightly more adult conversation about adding "just one more" provider. The cost could be described as another neighbour, not another afternoon.

What I would do differently today

I would have given deprecation mail a named owner on day one. Code ownership without inbox ownership is how you wake up on a deadline the vendor set two months ago.

I would have built the stuck-work view before the third integration, not after we already had folklore. The first integration teaches you the pattern. The second will copy your omissions.

I would push harder on not sharing one job across two vendors. Every time I allowed it for speed, I bought a repair tool I did not have time to make nice.

Broader lesson

Your repository is the part of the system you can diff. The risk is often in the parts you can only log. Design for evidence, replay, and a language that belongs to you.

If an integration cannot be explained without opening their dashboard, you have extended their product, not yours. Sometimes that is fine. Call it that, so you know what you have chosen.

Closing thought

Add the provider if the organisation already depends on them. Then spend the rest of the design on the day they change, delay, or lie in a polite JSON body. That day is not an edge case. It is the reason the integration exists in your architecture at all.