At Octane, every wallet transaction — authorize, verify, capture, reverse, settle — runs as a Temporal workflow. Not most of them, not the complicated ones. All of them, including the ones that look simple. That uniformity is the point, and getting there meant unlearning a habit most backend teams share.
This is the follow-up I promised at the end of the fraud post: the orchestration underneath the OCR check. The fraud decision changed what a transaction is; this is about what carries it.
A transaction is a process, not a request
The habit is to model a wallet charge like an API call: a request comes in, a handler runs, a response goes out. That shape holds for about thirty seconds of real thinking, because an actual fueling transaction looks like this: authorize against the ledger, wait for the station to confirm the pump, run the OCR verification, capture the real amount or release the hold, post to the ledger, notify the fleet.
That's not a request. It's a process that spans several systems, takes anywhere from two seconds to twenty minutes, and has to survive everything that happens in between — deploys, pod restarts, a station's connectivity dropping mid-pump, the ledger being briefly unavailable. The money doesn't care that you shipped a new version at the wrong moment.
What the naive shape costs
We started where everyone starts: a handler, a status column, and retries. The transaction row moves through states — authorized, verifying, captured — and background jobs push it forward. It works in demos. In production it fails in a specific, expensive way: the state machine exists, but it is scattered.
Part of it lives in the status column. Part lives in the handler that set it. Part lives in whichever retry job picks the row up, and in that job's assumptions about what has already happened. Answering the simplest operational question — where is this transaction and what happens next? — means reconstructing intent from three places. Every new edge case adds a fourth.
And the retry logic is worse than it looks. Retrying a step that actually succeeded is how you get double captures, so each retry needs its own idempotency reasoning, handwritten, per step, forever.
What changed with Temporal
Temporal's pitch is durable execution: you write the whole transaction as one ordinary function — authorize, then wait for the station, then verify, then capture — and the platform persists every step. If the worker dies mid-flow, another one resumes the function exactly where it stopped, with local variables intact. Timers, retries and external callbacks are part of the program, not infrastructure you assemble around it.
The fueling flow became readable top to bottom. Authorization is an activity with a timeout and a retry policy. The OCR check waits on a signal from the verification service — the workflow just sleeps, for minutes if the station is slow, without holding a thread or a connection. If verification fails or times out, the same function runs the compensation: release the hold, post the reversal, tell the fleet. The rollback lives next to the thing it rolls back, which is where sagas belong and almost never are.
Two properties did the real work for a wallet specifically:
- The state machine is the code. There is one place that says what a transaction does and in what order, and it is the function that does it. New engineers read the flow; they don't reconstruct it.
- Idempotency has a backbone. Every activity is keyed by the transaction id, so a retried step is a no-op instead of a double charge. We still write idempotent activities — Temporal doesn't remove that duty — but the retries themselves are no longer something we hand-write and hand-audit per step.
The rules that keep it honest
A workflow engine doesn't save you from yourself. The disciplines that mattered for us:
- Workflow code is deterministic. No clocks, no randomness, no I/O in the workflow itself — anything that touches the world is an activity. Break this and replays lie to you.
- Activities are idempotent, keyed by transaction id. The ledger enforces it too — Formance rejects a duplicate posting — so a retry is safe at every layer, not just the orchestrator's.
- Timeouts on everything, including the waits. A workflow that can wait forever will, one day, at 6 a.m. at a pump.
- Versioning before deploying. Workflows outlive deploys by design, so a change to the flow has to patch the running ones, not strand them.
Where it doesn't go
Not everything is a workflow. Reads never are — a balance check is a query, and routing it through an orchestrator adds latency for no correctness. And a genuinely simple operation that touches one system and finishes inside a request doesn't earn the machinery. The test I use: if the operation has a tomorrow — another system, another step, another chance to be interrupted — it's a process. Model it as one.
The ledger itself, the posting rules and the reconciliation queries that prove the numbers agree — that deserves its own post. It's on the list.
Muhammad Alnaqeeb is a senior software engineer in Cairo. He works on wallets, ledgers and fraud controls, most recently at Octane. Reach him at [email protected].
← More writing