Published In The Database Is Not Live On The Internet

I had posts marked published that were not actually on the internet. The database said they existed. The status column said published. The row was right there. And yet if you typed the URL into a browser, you got a 404. I only found this because I went looking, and once I saw the first one I had to assume there were others. When you run several blogs, you cannot eyeball every post on every site, so I built a small content registry to do the looking for me.

The registry does three things in order. It scans every blog and records each post in a database, riding on the single pipeline file that already feeds my blogs into Postgres, so I have one place that knows what is supposed to exist. Then, for each post that claims to be published, it actually fetches the real URL and checks for an HTTP 200. Then it checks whether that same URL appears in the site's sitemap. Anything that 404s or is missing from the sitemap gets flagged. That is the whole idea. It is not clever. It is just the difference between trusting a record and checking reality.

The Two Facts Are Not The Same Fact

The thing I had quietly assumed for a long time is that "published in the database" and "live on the internet" are the same fact. They feel like the same fact. You set a status to published, the publish step runs, and you move on. But they are two completely separate states, and only one of them matters to a reader or to Google.

A post can be published in the database and still be missing from the live site for a dozen unremarkable reasons. The build that generates the static pages skipped it. A deploy half finished. A path changed and the old URL stopped resolving. The post exists in the source of truth and never made it to the edge, or it made it once and then quietly fell off. None of those failures announce themselves. There is no error in your inbox. The database row keeps saying published, calm and confident, while the URL returns nothing.

This is the trap with any system where the record and the reality live in different places. The record is cheap to write and easy to trust. The reality is the only thing your audience experiences. If you only ever look at the record, you can run for months believing everything is fine while a slice of your content is silently dark. The registry exists to keep those two facts honest with each other.

Why The Sitemap Check Earns Its Place

You might think an HTTP 200 is enough. If the page loads, it is live, what else is there. But I deliberately check the sitemap too, because reachable and discoverable are also two different facts.

A post can return a clean 200 and still be effectively invisible because it never made it into the sitemap. Search engines lean on the sitemap to know what to crawl and how the site is shaped. A page that answers when you ask for it directly, but is absent from the map, is a page nobody is going to ask for, because nobody knows it is there. From a reader's point of view who arrives by search, an unlisted post and a 404 are close to the same outcome. They never reach it.

So the registry treats both as required signals. The URL has to resolve, and the URL has to be in the sitemap. A post that passes one but fails the other is not healthy, it is half present, and half present is exactly the kind of thing that hides for a long time. Checking both is barely more code than checking one, and it closes a gap that a status-only or even a reachability-only check would leave wide open.

The Edge Case That Made It Cry Wolf

The first version had a bug that taught me more than the feature did. It checked liveness by date. If a post's publish date was today or in the past and the URL was not live, it flagged it as dead. Reasonable on the surface. It ran one morning and immediately flagged a post as dead that was scheduled to go live at noon.

The post was fine. It was not dead, it was just not due yet. A post scheduled for later in the same day is not a failure, it is a future event that has not happened. But my check compared dates only, so to it, "today" meant "should already be live," and a noon post looked exactly like a broken one when the scan ran in the morning. That is a false positive, and false positives are not a harmless annoyance. The whole value of a monitor is that when it flags something, you believe it. False positives train you to ignore the alerts, and then the monitor is useless.

The fix was to make the check time-aware. Instead of comparing dates, it compares the full publish timestamp to the current time. If the post's publish moment is still in the future, even by a few hours, it is not overdue, so the registry skips it entirely. It is not pass and it is not fail, it is simply not yet evaluated. The post only becomes subject to the liveness check once its scheduled time has actually arrived. A noon post is left alone all morning and only starts being held to the live-or-flagged standard once noon has passed.

That one change is the difference between a monitor I trust and one I would have muted within a week. The lesson tucked inside it is that any time-based check has to respect the full timestamp, not just the date, and it has to know the difference between late and not yet. Late is a problem. Not yet is normal. A system that confuses the two will burn its own credibility. It is the same full-timestamp logic that lets scheduled posts stay safely future-dated until their moment arrives instead of being treated as broken the second the date matches.

Build The Cheap Audit Before You Need It

What I like about this registry is how little it is. There is no exotic infrastructure here. It scans, it records, it fetches real URLs, it reads a sitemap, it compares. The expensive part was not the code, it was noticing that the gap existed at all, that a published row was never a promise about the live site.

If you are shipping content across more than one surface, or honestly even one surface you do not babysit, I would build the equivalent. Make something that proves liveness against the actual URL the way a visitor would experience it, not against your own database's opinion of itself. Make it check discoverability too, so a post that loads but is unlisted does not slip through. And handle the not-yet-due case from the start, because the first scheduled post you forget about will fire a false alarm and start eroding your trust in the whole thing on day one.

The deeper point generalizes well past blogs. Any time your system has a record that says something is done, and a reality that the record is supposed to describe, those are two facts and they drift apart quietly. The cheap insurance is a small, boring audit that goes and checks the reality directly, on a schedule, and is careful enough about edge cases that you actually believe it when it speaks. A published row is only ever a claim, and the live URL is the one thing your reader and Google actually see, so that is the one worth proving.

I build things like this for clients, full-stack apps, AI agents, and automation pipelines, usually shipped faster than expected because I work with AI tooling every day. If you want something built, book a call.