Stop Bouncing Cron Jobs Off A Remote API

For about two weeks, a daily report I rely on simply did not arrive, and nobody told me. No failed job, no red mark, no error in a log I would have thought to check. The work that was supposed to run every morning just did not run, and the system was perfectly happy to keep not running it forever. That is the part that still bothers me. A job that crashes loudly is annoying. A job that quietly stops existing is dangerous, because you do not find out until the absence has cost you something. This post is about how I got into that mess and the architecture change that made the whole failure mode impossible.

How The Trigger Got So Far From The Work

The setup looked reasonable on paper. I had a handful of scheduled jobs, blog rebuilds and a content report, and I wanted them to run once a day. Those rebuilds are what publish date-scheduled posts when their time arrives, so a rebuild that silently stops running quietly buries every future-dated post. The runner that does my CI work already lives on my own server, a self-hosted node I control. But instead of letting that node run the jobs directly, I had wired up a chain. A plain crontab on the server would fire on schedule, and its only job was to call GitHub's workflow_dispatch API with a fine-grained token, which would then kick off a workflow, which would eventually do the actual work back on the same machine.

Read that again, because the absurdity is the whole point. The work needed to happen on my server. The runner that does the work lives on my server. And yet the trigger left the building, went out to a remote API, authenticated with a credential, and came back. I had introduced a network round trip and an auth boundary between a clock and the thing the clock was supposed to start, when both the clock and the thing were sitting in the same box the entire time.

It did not feel wrong when I built it, because each piece individually made sense. Crontab triggers things. workflow_dispatch starts workflows. Tokens authenticate API calls. The problem is not any single component, it is the shape of the whole. Every hop you add is another thing that can break, and a remote API hop in particular drags a credential along with it. That credential is the part that turned out to matter.

The Failure Was Silent, Which Is The Worst Kind

Here is what actually went wrong. The token file the crontab used quietly went empty. Not deleted in a way that would scream, not corrupted into garbage that would throw, just empty. From that moment, every single trigger failed. The crontab fired on time, made its API call with no usable credential, got rejected, and moved on without a fuss. Cron does not care that your command failed. It runs the line and forgets about it. So the failures stacked up day after day, completely invisible.

And for a while, I did not even notice the symptoms, because a separate fallback was still firing and covering the gap. The report kept showing up from the other path, so nothing looked broken. This is the cruel twist of redundancy done casually. A fallback that hid the primary failure is what kept me from noticing for two weeks. It bought me time I did not know I needed, and it spent that time hiding the exact thing I would have wanted to know.

Then I removed the fallback, for unrelated reasons, and the report just vanished. No error. No alert. Only silence where a daily artifact used to be. It took me longer than I want to admit to connect the disappearance to a token file that had been empty for two weeks. The lesson landed hard. A credential that can silently become empty is a single point of failure, and when that failure mode produces no signal, the job is failing intermittently and telling you nothing, which is the most expensive way for anything to break.

Bring The Schedule To The Work

The fix was not to add monitoring on top of the broken chain. The fix was to delete the chain. If the work has to run on my node, and the node is already a Kubernetes cluster running k3s, then the right place for the schedule is inside that cluster, as native Kubernetes CronJobs. Kubernetes has a first class CronJob resource whose entire job is to run a workload on a schedule, on the same infrastructure where everything else lives. There is no API to call out to, no remote service to authenticate against, no trigger that leaves the machine. The clock and the work are now in the same place, which is where they should have been from the start.

Moving to in cluster CronJobs deleted the remote trigger layer wholesale, and it took the credential with it. The fine-grained token that could go empty is gone, because there is nothing left for it to authenticate. A credential that no longer exists cannot break. That is the cleanest kind of fix, the one that removes a category of failure rather than guarding against it. I did not make the token harder to break. I made the token unnecessary.

There is a real architectural point underneath the convenience. When your runner already lives on your own infrastructure, reaching out to a remote API just to trigger local work is adding distance for no benefit. Every layer between the schedule and the execution is surface area for silent failure. A scheduler that sits next to the work has fewer moving parts, fewer credentials, and fewer ways to fail without telling you. Proximity is a feature.

Making Each Job Stateless And Self-Contained

Deleting the trigger layer was the headline, but the way I built the replacement jobs matters just as much, because the same instinct that says move the schedule close to the work also says keep each job small and disposable. Every CronJob in the cluster runs stateless. It does not depend on leftover files from a previous run or on the state of the node. It spins up, does its one thing, and tears down, so a job that ran yesterday and a job that runs today are identical and isolated.

The repository access follows the same minimal philosophy. Each job uses a read-only deploy key, which is exactly as much permission as the job needs and not a scrap more. A write capable token would be a liability sitting in a job that only ever reads, so it does not get one. With that key, the job does a sparse clone, pulling down only the specific paths it actually needs rather than the whole repository, into an emptyDir volume that exists for the lifetime of the run and vanishes when the pod exits. Less code on disk, less to go wrong, nothing persisted that has to be cleaned up later.

The last detail is timezone pinning. A daily job that is supposed to fire at a particular local hour cannot be left to drift with server clocks or with daylight changes, so each CronJob is pinned to an explicit timezone. It fires at the right local hour year round, in summer and in winter, without me touching it twice a year. That is a small thing that turns into a recurring annoyance if you ignore it, and a non issue once you set it correctly the first time. Read-only access, ephemeral storage, sparse checkout, pinned schedule. Every one of those is the same idea applied at a different layer, which is to give the job the least it needs and nothing it can trip over.

What The Two Weeks Of Silence Taught Me

The thing I keep coming back to is that the original design was not reckless. It was the kind of setup a reasonable person builds when each piece looks fine on its own. The danger was emergent, hiding in the seams between components and in a failure mode that produced no signal. If I had only added an alert to the old chain, I would have papered over the symptom and left the bad shape in place, waiting for the next quiet way to break. Pulling the schedule into the cluster did not patch the failure. It removed the conditions that allowed it.

So the rule I took from this is simple enough to apply anywhere. Do not add a fragile remote-trigger layer when your runner already lives on your own infrastructure. That single point of failure and the silence around it are exactly what cost me the two weeks. Prefer a native scheduler that sits close to the work, and the whole category of problem stops being something you monitor and becomes something that cannot happen. For the failures that can still happen, I lean on a Discord ops bot that surfaces CronJob status straight from the cluster so nothing goes quiet again.

I build things like this for clients, full-stack apps, AI agents, and automation pipelines, usually shipped faster than expected because I work with AI tooling every day. If you want something built, book a call.