Suzanne: I got a question at an event this week that I think is exactly the right one to be asking. Someone wanted to know what we, as a vendor, are doing to ensure our AI is reliable, and what we are doing to keep humans in the loop. Underneath it was the question that sits beneath almost every serious AI conversation right now: when an AI produces something (a report, an analysis, a recommendation) and someone acts on it, who actually owns that output?

My answer is that the human does. Always. I’ll happily make the case for our product, because I think it’s genuinely powerful, but this isn’t really a sales point. It’s something everyone deploying AI needs to think through, no matter what AI product they are using or building. And the mechanics matter more than the slogan. “Human in the loop” has become one of those phrases people say to sound responsible without committing to anything specific. I’d rather describe what it actually looks like, because there are three places where ownership is either protected or quietly given away: in the software itself, in the agents my team builds on top of it, and in the hands of the person who actually uses it.

The first place is in the AI’s responses themselves. We build guardrails so the system won’t make things up when it doesn’t have the data. If you ask it something it can’t answer from the information available, it won’t paper over the gap with a confident-sounding guess. It tells you it doesn’t have the data, and, this is the part I care about, it tells you what it would need in order to answer. That second behavior changes the whole dynamic. Instead of handing someone a plausible answer they have to debunk, it hands them a clear ask: here is the document, the figure, the source I’m missing. The person stays in control because they always know what the system actually knows and what it’s only inferring.

The second place is harder to automate, and it comes down to the people on my team who build the agents our customers use. How an agent is designed determines whether a human can really own its output, or only rubber-stamp it. We recently built an agent that drafts reports from project data. When my analyst showed me the output, the thing that stood out wasn’t the prose the agent generated. It was where it stopped. Throughout the document, the agent had inserted alerts asking for human input and human review in exactly the places where it shouldn’t be speaking on its own. One of them read, in effect, “please provide the current actions being undertaken.”

That is a design choice, and a deliberate one. The agent could have guessed at the current actions. It had enough surrounding context to produce something that read well and might even have been close. Instead, it flagged the gap and asked the human to fill it. The person reviewing the report can upload a current-action report, forward a string of emails, or simply type a few bullets, and the agent folds that material into the document quickly and cleanly. The human isn’t doing the tedious assembly work (the agent still does that), but the human is supplying and owning the judgment-heavy parts that no model should be inventing.

I think that distinction is the whole game. The goal isn’t to keep a human nominally “in the loop” so there’s someone to blame later. It’s to build the system so the human is in the loop at the points that actually require a person, the facts only they have and the call only they’re accountable for, while taking the drudgery off their plate everywhere else. An agent that asks good questions is doing more to keep a human in control than one that produces a flawless-looking draft with no seams showing.

There’s a louder version of this debate that I mostly find unhelpful. You’ve probably seen the tongue-in-cheek posts arguing that AI is already better at decisions than we are, or the earnest ones insisting it never will be. Set that aside for a second, because it’s beside the point. Even if a model were better than a person at some decision, and for plenty of narrow tasks it may well be, accountability doesn’t transfer to it. When a report goes up the chain and someone acts on it, a person’s name is on that decision. A model can’t be held responsible, can’t be asked to explain itself in a way that satisfies an inspector general or a court, and can’t carry the consequences. Ownership stays with the human not because the human is always smarter, but because ownership is a human thing. It can’t be delegated to software, no matter how good the software gets.

I want to be careful not to overstate this. I’m not saying humans should never hand a decision off to a machine. We already do, and we’re going to do far more of it. I fully expect agents will be booking our flights before long, and I’m fine with that. But that example is exactly where the real work lives. If you don’t want a six-hour layover in Chicago, you have to say so up front. Do you want the absolute cheapest fare, or are there constraints: no red-eyes, a preferred airline, a hard cap on connection time? Trust in an agent isn’t about whether it acts on your behalf. It’s about whether you’ve defined the guardrails well enough that you’re comfortable with the range of actions it can take. That’s the same idea, scaled. The higher the stakes, the more the guardrails matter and the more judgment you keep close, but the basic move never changes: decide in advance what a good outcome looks like, and let the agent operate inside that.

This matters more in government than almost anywhere else, which is the world most of our customers live in. The output of an AI system in that context isn’t a marketing draft someone will lightly edit. It’s a report that informs a funding decision, a status assessment a leader will brief upward, a summary that shapes how resources get allocated. The people doing that work can’t outsource the responsibility for it, and they shouldn’t want to. What they can do is use tools that make the responsibility easier to carry: tools that surface what they need to weigh in on, are honest about what they don’t know, and never quietly substitute a guess for a fact.

The third place is the one no software vendor can build for you: the individual who actually uses the tool. We can put guardrails in the software, and my team can design the agent to ask for human input in the right spots, but in the end someone has to read the output and own it. This is just how the work goes for me, even on things that aren’t high-stakes. When I use AI to help with a deck, a document, or an analysis, I don’t think I’ve ever shipped the first version it gives me. The last document I worked on, I was finally satisfied at version 42, and that’s not unusual; most things take several rounds before they’re right. It’s still dramatically faster than starting from a blank page, so this isn’t a complaint about the tool. It’s the point. The speed is real, but it buys you time to read, push back, and revise. It isn’t a license to skip those steps. The output isn’t done when the AI hands it to you. It’s done when you’ve read it closely enough to put your name on it.

I see the other side of this too, on the receiving end. I’ve been handed documents and output, from people inside the company and outside it, that were just plainly wrong. Either the person didn’t know enough to catch it, or they didn’t read what they were sending. And nothing frustrates me faster than asking someone a question and getting back a long copy-paste from ChatGPT. I could have done that myself. What I wanted was their specific opinion. If you use AI to help you formulate a thought or summarize something, that’s great; I do it constantly. If you use it well enough that I can’t tell, even better. But a raw, pasted-in model response is almost always obvious, and it tells me you skipped the part that was actually your job: deciding what you think and standing behind it.

Adeline: From the European side, the question of who owns the output has a formal answer: the EU AI Act places human oversight requirements on high-risk systems precisely because the law recognises that accountability cannot transfer to a machine. For the kinds of work done in European institutions (financial stability assessments, policy analysis, recommendations that inform decisions with real consequences) that framing is right. But the classification logic embedded in the Act raises a harder question that the regulation doesn’t fully resolve: what counts as high-risk, and who decides?

The Act’s risk tiers assume you can draw a clean line between consequential and routine outputs. In practice, that line moves. An analysis that looks routine at the moment of production can become high-stakes the moment it lands on the right desk. Designing governance around a static classification is difficult when the technology itself, and the ways institutions reach for it, are changing faster than any framework can track. This is the honest challenge for institutions trying to develop AI governance strategies right now: the uncertainty isn’t just about what the tools can do, it’s about what they will be asked to do.

Suzanne’s point about the oversight bottleneck is where I think the European conversation needs to go next. The Act mandates human oversight for high-risk systems, which is correct in principle. But if human oversight means a person reviews every output, that requirement will eventually become either a fiction or a brake. We haven’t resolved what meaningful oversight looks like when volume makes line-by-line review impossible. The better system design Suzanne describes (agents that surface only what genuinely requires human judgment) is one answer. But it is a design answer, not yet a governance answer, and translating it into institutional practice across organisations with different cultures, different risk tolerances, and different levels of AI maturity is the work that remains. The ownership question is settled. The operational question is not.

Suzanne: The person who asked me the accountability question was really asking whether they could trust what comes out the other end. The honest answer is that they can trust it to the extent that the system is honest about its own limits and the people using it stay responsible for the judgment. We build the guardrails and design the agents to make that posture the default rather than the exception. But the ownership was never going to live in the software. It lives with the person who signs their name to the result, and the best thing the technology can do is make that person’s job clearer, not pretend to take it off their hands.

Suzanne writes from the US. Adeline writes from the EU. Neither of us is handing the pen to the machine.