When a Provider Goes Down, the Lead Still Arrives

The problem

The site's inquiry form needed three things: take a form submission, put it somewhere durable, and tell a human. That is the most boring possible requirement and also the one where quiet failure costs the most, because nothing on the visitor's screen distinguishes "we received this" from "we dropped this".

The usual shape — post to a database, return success — has a failure mode that is invisible from both ends. The database has an outage, or a field name drifts, or a dropdown option that the code expects does not exist; the request errors; the visitor sees a red box, or worse, a green one. Either way the inquiry is gone and nobody knows it existed.

The architecture

Rather than trying to make each step reliable, the Worker names every way the request can end and decides, once, which of those endings is allowed to lose the submission. That decision is written as an ordered ladder at the top of the file, and the code below is the ladder:

1. Schema validation rejects a malformed body with a 400.
2. Two cheap bot checks return a silent success, so a bot believes it succeeded, while logging enough that a false positive is still recoverable.
3. Human verification runs, and fails closed: a failure, a timeout, or a missing token is a rejection.
4. An optional abuse cap applies, and it is deliberately soft: an outage in the cap itself must never block a real customer.
5. Classification runs best-effort and can never throw or block.
6. The database write is attempted. Failure is caught, recorded, and not fatal.
7. The notification email is always attempted, carrying the full payload, with a loud subject line when step 6 failed — so the email is not just a notification, it is the durable fallback.
8. Success if either persistence path worked. Only when both fail does the request return an error, and then the whole payload is written to the log so it can still be recovered.

Two smaller decisions carry disproportionate weight. The write target comes from the Worker's bound configuration, never from the request, so no submission can steer a write somewhere else. And one of the bot checks was renamed after an adversarial review found its first field name was one a browser's autofill would populate — which would have silently dropped a real person's submission; it was renamed the same day, in the hardening pass that followed the review.

What we measured

Every rung has a unit test that drives the real handler over mocked bindings and a mocked network: the schema rejections, the bot checks, verification failing closed on a network error, the abuse cap degrading safely, the classifier throwing, the database throwing, the email throwing, and both throwing at once. The tests assert the exact database payload shape and the exact mail request, because those two are where a silent field-name drift would live.

The interesting result is the shape of the remaining risk. After the ladder, a submission that clears the checks is lost only where two independent providers fail inside the same request — and even then the payload is in the log. One single-point loss remains by design: a bot check that misfires on a real person returns the same silent success a bot would see, with only the log to recover from. That is the trade the ladder makes, and it is written down rather than discovered.

What transferred to the lab's method

"Which failures are allowed to lose a customer" turned out to be a better design question than "how do we make this reliable", because it produces a list you can test. The ladder comment at the top of the file is the specification, the tests are the proof, and a new rung has to be argued for in writing before it exists.