Why querying a database after the fact isn't observability

Most teams start the same way. Something goes wrong with an agent, so you open a database client, write a query, and try to piece together what happened. For a while this works fine. Then it stops working, and it usually stops working at the worst possible moment, when you actually need it.

Here's what I've seen actually break, and why.

You find out too late

A query is reactive by definition. You only write one after you already suspect something went wrong. For a chat agent that finishes in a few seconds, that gap between failure and discovery is small enough not to matter. For an agent running six hours unattended, the gap is six hours. More often than I'd like, the customer notices before the team does, because the database was never watching anything, it was just waiting to be asked.

You lose the exact evidence you need most

This is the one that stings. When an agent crashes mid-run, whatever was sitting in memory at that moment is gone. If traces were being written in batches, or an SDK was flushing on some timer, the last batch never lands. You end up with a clean record of everything up to the last flush and total silence after it, which is exactly the window you actually needed to see.

Queries that used to be fast stop being fast

Once an agent is generating hundreds of spans per run, and you've got a few dozen runs happening at once, the query that used to take two seconds starts taking twenty. Then two minutes. Then it just times out. This isn't a tuning problem you can index your way out of, it's architectural: reconstructing a session means joining thousands of rows on demand, at query time, every single time someone opens a dashboard. That was a fine design when runs lasted seconds and there were only a handful of agents running. It stops being fine once you've got long, complex, parallel workflows for customers who expect an answer in the moment, not after someone runs an investigation.

You can't see what's happening right now, only what already finished

A database only shows you completed runs. It has nothing to say about the agent that's executing right now, this second, and might be stuck, slow, or quietly going off the rails. If most of your agents are mid-run at any given time, and only a fraction have completed, then every dashboard, every cost number, every quality score you're looking at reflects that small finished fraction. That's not just a delay. The numbers are structurally the wrong numbers.

A frozen agent looks identical to a slow one

There's no way to tell the difference between “this run started four hours ago and is progressing slowly” and “this run started four hours ago and died two hours ago” just by looking at a database. Both look like a row with an old timestamp. Telling them apart requires something actively watching for the absence of activity, not just recording the activity that happens.

You can only run the queries you thought to write in advance

When something actually breaks in production, the question you need answered is rarely one you had the foresight to write a query for. You want to know what the agent was doing at a specific moment, which tool call caused the spike, whether this exact failure happened before, what changed between the last good run and this one. Answering any of that from raw rows means writing a new query against a schema you probably didn't design yourself, while a customer is waiting on the other end.

It doesn't hold up across multiple customers

If you're running agents for more than one enterprise customer, a shared database turns into a compliance problem fast. Who's allowed to query whose data. Whether one customer's traces could ever be visible to another, even by accident. Whether you can guarantee their data stays in the region their procurement team requires. None of these have a clean answer once everything's sitting in one shared store, and eventually someone's security review is going to ask.

Why this actually matters if you're the one selling the agent

Everything above is the technical version. There's a business version sitting right behind it, and it's the one that actually keeps me up, because it's not abstract, it's what happens to your company if the technical gap stays open.

If you've signed an SLA on uptime, latency, or response time for an agent you're running on a customer's behalf, every one of the gaps above is a way that SLA quietly breaks without you knowing it broke, until the customer tells you.

It also does something to the relationship that's hard to undo. Once a customer figures out you found out about a problem from them instead of from your own system, they've learned something about you that doesn't go back in the box: you're running blind on infrastructure they're paying you to run confidently. That's a different kind of damage than a bug. Bugs get fixed. A customer's belief that you weren't watching doesn't get fixed by fixing the bug.

Enough of that, repeated enough times, is just churn with extra steps.

There's also what it does internally, to your own team. Without something actually watching, the job of noticing a stuck or degrading agent falls on humans staring at logs or waiting for a Slack message, which means you're asking people to manually monitor something that runs continuously, at all hours, across every customer. That's not a sustainable ask, and it's not really a monitoring strategy, it's hoping someone happens to be looking at the right moment.

And the customers who actually understand what they bought, the technically sophisticated ones, expect this by default. If you're selling an agent into a company that runs its own observability stack for its own systems, “we'll look into it when you report it” is not a credible answer from a vendor, it reads as a company that hasn't caught up to its own product.

The last piece is the one that's easy to miss because it's not a failure, it's a missed opportunity. If you can see a problem forming before the customer feels it, you get to reach out first. That's not just better service, it's the difference between a vendor customers tolerate and one they trust with more. Every time the customer finds the problem before you do, you've left that on the table.

Where this leaves it

Querying a database when something broke was the right approach for 2024, for a couple of agents running short sessions where an occasional slow investigation was tolerable. That's not what's running in production now. Agents run for hours. They crash mid-task. They serve customers with real compliance requirements and generate thousands of spans per run, and they need to be watchable while they're still running, not just readable after they've stopped.

A database is a place to put data after something happened. What's actually needed is something that's watching the whole time, catches the moment something goes wrong, and can tell you fast enough to matter. That's the difference I keep coming back to, and it's the reason we built GlassFlow the way we did instead of starting from a database schema and hoping the query layer would keep up.