One Pipeline For File Blogs And Postgres Blogs

I have a set of blogs that need to be scanned by one pipeline, and they do not agree on where they keep their posts. Some store every post as a Markdown file on disk, with the metadata in the frontmatter. Others keep their posts as rows in a Postgres database, where the title and date and body are columns. A few only expose their posts as an RSS feed. The pipeline does not care about any of that. It cares about posts. It needs the same handful of fields out of each one, the title, the slug, the published date, the body, no matter whether that post started life as a file, a database row, or a feed item. The whole design comes down to one decision, which is where you let the differences between those sources live. Get it right and the engine stays small. Get it wrong and every new blog bleeds its quirks into your core logic.

The Problem Is Where The Differences Live

The tempting first version of this pipeline puts a big branch at the top. If the blog is a file blog, read the directory and parse the Markdown. If it is a database blog, open a connection and run a query. If it is a feed, fetch the URL and parse the XML. Each branch knows how to do its own thing, and the engine grows a fork for every kind of source it learns about.

That works for exactly as long as you have two sources. The moment you add a third, the branch gets a third arm, and every piece of code downstream that touches a post now has to know that a post might have come from any of those places. The database connection logic ends up tangled with the date formatting ends up tangled with whatever you do with the posts afterward. You cannot test the file path without dragging the database path along with it, because they share the same sprawling function.

The real problem is not that the sources are different. They genuinely are different, and no clever trick makes a Postgres row identical to a Markdown file. The problem is letting that difference spread. A file blog and a database blog have almost nothing in common in how you reach their data, but they have everything in common in what you want out of it. So the design job is to find the narrow place where they are the same and force all the difference to stay on one side of it.

A Small Interface Is The Whole Trick

The narrow place is an interface, and it is almost embarrassingly small. A scanner does one thing. You give it a blog, it returns that blog's posts, and the posts come back in a single shared shape every scanner agrees on. That is the entire contract. Scan a blog, return normalized posts. Nothing in that sentence mentions files, or SQL, or feeds, because the contract does not know those words exist.

Underneath that contract I have a few implementations. There is a Markdown scanner that walks a directory, reads each file, and pulls the fields out of the frontmatter. There is an RSS scanner that fetches a feed and turns each entry into a post. There is a Postgres scanner that runs a query and turns each row into a post. They could not be more different on the inside. One does filesystem work, one does HTTP and XML, one does database access. But they all hand back the same thing, a list of posts in the one shape the rest of the engine understands.

That is the move that makes everything else simple. Every scanner is interchangeable from the outside, because they are all just a way to turn one blog into a list of normalized posts. The differences did not vanish. They got pushed behind the interface, into the body of each scanner, where they belong and stay contained. The core of the engine never sees them.

The Engine Stays Source-Agnostic

Once the scanners hide the differences, the engine on top of them gets to be boring, which is the highest compliment you can pay a piece of infrastructure. It reads a config, figures out which scanner each blog needs, asks that scanner for the posts, and then does whatever the pipeline does with posts. It never asks where a post came from, because by the time a post is in its hands, that question has no answer worth caring about. A post is a post.

This is the payoff for spending the effort on the interface. Everything downstream of the scanner gets written exactly once and works for every source automatically. Whatever the pipeline does next, indexing the posts, checking that each one is still live, reporting on them, runs the same way whether the post was a file five minutes ago or a database row. There is only one path through the rest of the system, because the scanner already flattened the sources into one stream of identical objects.

The test for whether you got this right is simple. If adding a brand new kind of source, say a blog that keeps its posts in some other store entirely, means writing one new scanner and touching nothing in the core, the interface is doing its job. If it means going back into the engine to add another branch, the abstraction leaked somewhere, and the differences you thought you had contained are still escaping into the core logic.

Config Decides, Code Does Not

The other half of this design is that I do not hard-code which blog uses which scanner. That lives in a YAML config, one entry per blog. Each entry describes where that blog's posts live, what format they are in, and how its particular frontmatter or columns map onto the shared fields the pipeline expects. The mapping part matters more than it looks. One blog might call the publish field publishedAt and another might call it date, and the config is where you reconcile those names so the scanner knows that this blog's date column is the thing the pipeline calls the published date.

The engine reads that config and uses it to pick the right scanner for each blog and to feed that scanner the details it needs. This is config over code in the most literal sense. Adding a blog is editing a YAML file, not writing a function. Changing where a blog's posts live, or how its fields map, is a config edit too. The behavior of the pipeline is data, and the code is just the thing that interprets that data.

The reason this beats hard-coding a list of blogs in the source is that it keeps the knowledge about each blog in one declarative place instead of scattered through the logic. When you want to know how a given blog is wired up, you read its config entry and you know everything. You are not reverse engineering it out of branches buried in a function. The config is the description of the world, and the code stays a small, dumb, reliable interpreter of it.

The Transport Hides Inside The Scanner

Here is the detail I like most, because it shows the same principle working one level deeper. The Postgres scanner has to reach an actual database, and that database is not always in the same place. When the pipeline runs inside the cluster, the scanner reaches the database by execing into the database pod directly. When the pipeline runs on my laptop, there is no pod to exec into, so it reaches the same database over SSH instead. Two completely different ways of getting a query to the database, depending on where the code is running.

The important thing is that this choice lives entirely inside the Postgres scanner. The interface above it never hears about it. The engine asks the Postgres scanner for posts, and the scanner figures out, privately, whether it is in the cluster or on a laptop and picks its transport accordingly. From the outside, it is still just a scanner that returns normalized posts. The cluster-versus-laptop question is exactly the kind of difference that would have poisoned the core if it had been allowed to leak upward, and instead it is sealed inside the one component that actually needs to know.

That is the lesson recursing on itself. A small interface let one engine handle file blogs and database blogs, by pushing the file-versus-database difference behind it. Inside the database scanner, the same trick handles cluster-versus-laptop. Every time a difference threatens to spread, you find the narrow contract it has to honor and bury the difference beneath it.

The thing I would carry into any system like this is that a small, honest interface is worth more than a clever one. Scan a blog, return normalized posts is almost nothing as a contract, and that is precisely why it holds so much. It says only what every source has in common and stays silent about everything else, which leaves each implementation free to be as strange as its source requires. Keep the contract small, keep the core ignorant of where data comes from, and let the config carry the knowledge. The differences never go away. You just decide, deliberately, how far down they live.

I build things like this for clients, full-stack apps, AI agents, and automation pipelines, usually shipped faster than expected because I work with AI tooling every day. If you want something built, book a call.