Why AI Music Generation Runs On A Workflow Engine

A music generation job is not a request. It is a small saga that unfolds over several minutes, touches multiple services, and can fail at any point along the way. When I started building Melodex, the AI music generator, the obvious first instinct was to throw the job on a queue, flip a status flag to "processing", and poll until it said "done". That works right up until the moment a step three quarters of the way through the pipeline times out, and now you have a half-finished job, a row in the database that says "processing" forever, and no clean way to know which steps already ran. This post is about the moment I stopped fighting that pattern and reached for a durable workflow engine instead, and just as importantly, when you should not.

When A Job Needs Durable Execution

The thing that makes AI generation hard to orchestrate is not any single step. It is the combination of three properties at once.

First, there are many steps. A generation pipeline is not one call to one model. It is a sequence of stages that prepare inputs, call out to generation services, post-process the result, and finalize the output. Each stage depends on the one before it.

Second, every step can fail independently, and the failures are normal, not exceptional. External generation services rate limit you. A request times out because a model was cold. A transient network blip drops a connection. None of these mean the job is doomed. They mean you should wait a moment and try that step again. In a synchronous request handler, a failure on step four is a thrown exception that unwinds the whole stack and loses steps one through three.

Third, each step takes real time, measured in minutes, not milliseconds. That single fact rules out every shortcut. You cannot hold the work inside an HTTP request, because the client will give up long before the job finishes. You cannot keep the state purely in process memory, because the process will not survive the full duration of the job without something going wrong eventually. The work has to live somewhere outside the request and outside any single process.

When a job has all three properties together, many steps, each failure-prone, each slow, the naive approaches start to buckle under their own bookkeeping.

What The Hand-Rolled Version Actually Costs

The honest way to see why a workflow engine earns its place is to write down what I actually built before I reached for one. It started small. A queue, a worker, and a status column. Then the real failure modes showed up one by one, and each one added a piece.

I needed to retry a failed step, so I added a retry_count and a maximum, plus logic to decide which exceptions were retryable. I needed to retry only the failed step, not the whole job, so the status column became a current_step column, and the worker had to be able to resume from an arbitrary point. I needed to avoid redoing expensive work, so each step had to be idempotent and had to record its own partial output somewhere durable before moving on. I needed to handle the worker dying mid-step, so I added a heartbeat and a reaper that found jobs whose worker had gone silent and requeued them, while being careful not to run a step twice. I needed backoff so retries did not hammer a struggling service, so I added timing fields and a scheduler that respected them.

Each of these was a few hours of work. Together they were a small distributed system, and it was a distributed system whose entire job was bookkeeping, not making music. The bugs in it were the worst kind, because they were timing-dependent and only showed up under failure, which is exactly when I least wanted to be debugging my own scaffolding. I ended up with a pile of scaffolding that reinvented, badly, the thing a workflow engine gives you on day one.

What Durable Execution Buys You

That thing has a name. Durable execution. It is the core promise of a workflow engine like Temporal, and it collapses all of that bookkeeping into two guarantees.

The first guarantee is automatic retries with a policy you declare instead of code you write. You wrap each unit of real work as an activity, and you attach a retry policy to it. Initial interval, backoff multiplier, maximum attempts. When the activity throws, the engine catches it, waits according to the policy, and runs it again, without unwinding the rest of the job. The retry logic that would have been scattered across your worker now lives in a few lines of configuration next to the step it protects. You stop writing try, except, increment, sleep, and you start writing "this step may be attempted up to N times with this backoff", which is what you actually meant.

The second guarantee is the one that is hard to build and easy to underestimate. The workflow resumes after a crash from where it left off. The engine persists the history of everything the workflow has done. Which steps completed, what they returned, where execution currently sits. If the worker process dies in the middle of a job, whether from a deploy, an out-of-memory kill, or the machine simply going away, another worker picks the workflow back up and replays its history to reconstruct exact state. Completed steps are not run again. The job continues from the point of failure as if nothing happened. There is no status column stuck on "processing", no orphaned job, no manual cleanup. The half-finished state that haunted the naive version simply cannot occur, because the engine treats progress as durable by construction. On the image side of my stack I solved the same stuck-job problem a different way, with a webhook-plus-reconciliation sweep over fal queues, so the workflow engine is one answer among a few, not the only one.

Put those two together and the failure modes that defined the problem stop being your code's responsibility. What you write is the happy path, the sequence of steps in the order they should run, and the engine makes that happy path survive a hostile world. For a pipeline whose normal operating condition includes rate limits, cold models, and timeouts, that is not a convenience. It is the difference between a system that needs babysitting and one that does not.

Where This Fits The Rest Of The Stack

None of this asks you to rebuild your application around it. Melodex is a FastAPI backend with a Vite frontend and an Astro blog, and the workflow engine slots in cleanly. The API layer stays thin. A request to generate comes in, the handler starts a workflow and returns immediately with a handle, and the frontend reads progress from there. The long, fragile, minutes-long work happens entirely inside the workflow, off the request path, where it belongs. The boundary between them is clean, which is its own kind of win, because the synchronous web layer and the asynchronous job layer stop leaking into each other.

The workers that execute the activities are just processes you run alongside the API. They can scale independently, because the engine, not the worker, holds the source of truth about job state. Losing a worker is not losing work. That decoupling is what lets you deploy the backend in the middle of the day without praying that no generation job is currently in flight.

When You Should Not Reach For This

Here is the counterpoint, because it is the most useful part of the post and the part most people skip. A workflow engine is real operational weight. It is another system to run, another thing to understand, another set of concepts your future self has to hold in their head when they read the code. That cost is not theoretical. It is paid every day the system exists, not just on the day you set it up.

For a two-step job, this is overkill, the kind of premature infrastructure that looks responsible while actually slowing you down. If your job is "call one service, write the result", a simple queue is fine, and honestly a background task might be fine. If you want something between a bare queue and a full workflow engine, the hosted options like Inngest and Trigger.dev cover a lot of that middle ground for a solo dev. The durable workflow does not earn its keep there, because there is barely any state to lose and barely any failure to recover from. You would be carrying the weight of a system designed for sagas to run an errand.

The engine earns its place under one specific shape. Many steps, each able to fail, each taking minutes. That is precisely the shape of an AI generation pipeline, which is why it is the right tool here and not a default I would reach for everywhere. The skill is not knowing how to run a workflow engine. It is recognizing the shape of the job in front of you and matching the tool to it, which sometimes means the boring queue and sometimes means the durable workflow, and being honest about which one you are actually looking at.

The Takeaway

The instinct to keep things simple is correct, and a workflow engine is not simple. But simple has to be measured against the real shape of the work, not the version you wish you had. The lesson I keep coming back to is to spend the operational budget where the failure modes actually live, and to leave it in your pocket everywhere else.

I build things like this for clients, full-stack apps, AI agents, and automation pipelines, usually shipped faster than expected because I work with AI tooling every day. If you want something built, book a call.