The Backend Behind An AI Image Product

A user clicks generate, and the image they asked for does not exist yet. It will exist in twenty seconds, or a minute, somewhere on a provider's GPU that I do not own and cannot poll cheaply. That gap between the click and the result is the entire engineering problem behind an AI image product I built, and it is the part nobody sees when the product works. The image just appears. Underneath, a small system is fighting to make sure it always does, even when the thing supposed to tell me the image is ready never says a word.

This post is about that system. Not the model, not the prompt, the boring plumbing that turns an unreliable async callback into a product that feels instant and never loses a job. The specific provider here is fal, but the lesson has nothing to do with fal. It is about building on top of a slow operation you hand off and get back later. What follows is the architecture I would tell another solo builder to copy.

Why Image Generation Forces You Into Async

A web request wants to finish in milliseconds. Image generation wants to take a minute. Those two facts cannot share a single synchronous HTTP request, and pretending they can is the first mistake people make.

If I held the user's connection open while the GPU worked, I would tie up a server worker for the full generation time, hit request timeouts, and fall over the moment more than a handful of people generated at once. So the request does not wait. It submits the job to the provider's queue, stores a record that says the job is in flight, and returns immediately. The user sees a pending state. The pixels arrive later through a separate channel, a webhook the provider calls back when the work is done.

This is the standard async handoff, and on paper it is clean. The request is fast, the queue absorbs the load, the webhook delivers the result. It works most of the time. The problem is the phrase most of the time. A product is defined by what happens the rest of the time, and that is where I spent the real effort.

Why The Webhook Is Not Enough

The webhook is the primary path, and if you only ever test on your own machine with one job at a time, it will look completely reliable. It is not. Webhooks fail in every way a network message can fail, and a generation product has to assume all of them will happen.

A webhook can be missed entirely, if my server was briefly down, mid deploy, or the provider's delivery attempt hit a blip. It can be delayed, arriving long after the user expected their image. It can be duplicated, so I receive two callbacks for the same job and naively process the result twice. And worst of all, it can arrive after I already gave up on the job and marked it failed, a result I told the user did not exist suddenly showing up.

Each of those failure modes, on its own, corrupts state. A missed webhook leaves a job stuck pending forever, a spinner that never resolves. A duplicate creates double work or a double charge against quota. A late one contradicts a decision I already made. If the webhook is your only mechanism, every one of these is a support ticket, and you cannot prevent them because the failure happens out on the network where you have no control. The only move is to stop trusting the callback as the source of truth and build a system that survives its absence.

Reconciliation Is The Real Architecture

The mental shift that fixed this was to stop thinking of the webhook as the thing that completes a job, and start thinking of it as one of several ways the system might learn a job is done. It is the fast path, not the only path. Behind it I run a small reconciliation system whose whole purpose is to drive every in-flight job to a correct final state regardless of whether its callback ever showed up.

There are four moving parts, and they are deliberately simple.

The webhook is the primary resolver. When it arrives and it is valid, I record the result and mark the job done. Fast, normal, the common case. This handler is idempotent, so a duplicate callback for a job that is already resolved is a no-op rather than double processing. That single property neutralizes the duplicate failure mode entirely, and idempotency is the cheapest insurance in the design.

The periodic sweep is the safety net. On a regular interval, a background task lists every job still marked in flight and asks the provider directly what happened to it. If the provider says that job finished and I never got the callback, the sweep resolves it right there using the provider's status as the source of truth. This is what kills the missed and delayed webhook. I do not need the callback to arrive, because I am also actively going to look. The webhook makes the common case fast, the sweep makes every case eventually correct.

The cleanup handles the truly abandoned. Some jobs never resolve, the provider errored deep in its own pipeline, or the work was lost. After a longer timeout than the sweep uses, a cleanup pass takes jobs that have sat in flight well past any reasonable generation time and marks them failed, so they do not linger as ghosts and any held resources get released. The two timers matter. The sweep runs often and is optimistic. The cleanup runs on a longer horizon and is pessimistic, it gives up on the genuinely dead.

The late-webhook rescue closes the last gap, and it is the nasty one. A job times out, the cleanup marks it failed, and then the callback finally arrives carrying a perfectly good result. Without a plan, my handler would either reject that result because the job is closed, or crash because the job is not in the state it expected. So the handler is written to accept a result for a job it had already given up on, and reconcile, the real image wins over the earlier timeout. A late success is still a success, and the user gets their image even though my system had stopped waiting.

Design For The Callback To Fail

The principle underneath all of this generalizes far past image generation. When you hand work to an external system and expect to be notified when it finishes, you have to design as if the notification will sometimes not come. Reconciliation is one answer to that, and a durable execution engine like Temporal is another take on the same long-running problem. Either way it is not a rare edge case you patch later, it is a baseline assumption you build around from the start.

That assumption changes the shape of the code. A naive implementation is event driven and nothing more, it waits for the webhook and acts. The version that survives production is event driven plus reconciling, it acts on the webhook when it comes and independently verifies state on a schedule, so the webhook is an optimization, not a dependency. The webhook makes things fast. The reconciliation makes things correct. You want both, and you must not confuse them.

The test for whether you have this right is one question. If every webhook your provider sends tonight silently vanished, would your jobs still resolve correctly by morning? If no, the webhook is load bearing in a way it should never be, and you are one network hiccup away from stuck state. If yes, slower but correct, you have built something that converges on its own, and convergence is the property that lets a one person backend run unattended.

What This Buys A Solo Operator

The payoff is not a clever architecture for its own sake. It is sleep. I am not watching a queue to manually rescue stuck jobs, because the system rescues them. I am not fielding tickets about spinners that never finish, because the sweep finishes them. I am not afraid of a deploy landing in the middle of a generation, because a webhook missed during that window gets picked up on the next sweep. The reliability comes from the design itself rather than from me babysitting it, and that is exactly what you want when there is one of you.

If you are building on any slow async provider, AI generation, video rendering, payment settlement, batch processing, do not let the happy-path callback be the only thing that resolves a job. Make your handler idempotent so duplicates are harmless. Add a sweep that reconciles in-flight work against the provider's own truth. Add a cleanup on a longer clock for the genuinely dead. And handle the late arrival, because the result that comes back after you quit is still a result your user wants. Build those four, and your backend stops depending on a message it cannot control and starts converging to the correct state on its own.

I build things like this for clients, full-stack apps, AI agents, and automation pipelines, usually shipped faster than expected because I work with AI tooling every day. If you want something built, book a call.