Newsletter

Who Should Manage AI Agents? A Simple Oversight Audit for Leaders

Jensen Huang's way of managing only worked at the top. A simple way to measure the supervision AI agents add to a manager's week, and decide who should handle it.

· 13 min read

Haroon ChouderyFounder of AI Ready. Previously co-founded Autoblocks and started A.I. For Anyone.

Welcome back.

Jensen Huang ran Nvidia with 60 direct reports. Even he said that setup doesn't work further down an org.

Yet many companies are about to give their managers a lot more to supervise. This time, it's AI agents.

This week, I'll walk you through a simple audit that shows how much supervision your AI agents need, and who on your team should be doing it.

Before we dive in, I'm hosting a free live Q&A with Chatbase this Thursday, November 19, at 2 PM ET, on how AI agents are changing customer support. Grab your spot here.

IN THIS EDITION

  • Byte-size news: agent incidents, bank controls, and where AI-saved time goes.

  • Main read: who should manage AI agents, and what supervising them really costs your managers

  • Your turn: an oversight workload audit you can run on one team this week

■ Byte-size News

1. Almost every I&O team has had an agent incident

Gartner surveyed 298 US and UK infrastructure and operations leaders. 95% had at least one agent-related incident in the past 12 months, yet 70% believe their agents stay within safe boundaries.

GARTNER · NOV 16

2. Big banks have agents in production, with few controls

BCG benchmarked 26 systemically important banks. 73% let agents into production for select uses, and only 8% have a full monitoring and control framework that covers them. Banks with more mature risk management were 2.5x more likely to be advanced AI adopters, so good oversight and speed can go together.

BCG · NOV 13

3. Time saved by AI tends to get absorbed

In Payscale's survey, 43% of employees say AI saves them three or more hours a week. Only 19% say that time goes into more strategic work. It's self-reported, but it's a good reason to check where the saved hours went.

PAYSCALE · NOV 10

How AI Agents Change Support: Insights from Chatbase

Customers are starting to send AI assistants to contact businesses for them.

So what should a support team do when an assistant asks to change an order or get into an account?

On Thursday, November 19, at 2 PM ET, I'm joining Clint Kruger from Chatbase for a free live Q&A and a short demo.

We'll get into what Chatbase is seeing, how to check who an assistant represents, what it should be allowed to do, and how support teams can get ready.

■ Haroon-in-the-loop

Who Should Manage AI Agents? A Simple Oversight Audit for Leaders

Jensen's 60 direct reports came with a condition

At Stripe Sessions in 2024, Patrick Collison asked Jensen Huang about his 60 direct reports. Jensen confirmed this, saying, “Sixty direct reports, yup.”

Then Collison asked if each of those 60 also had 60 direct reports of their own. Jensen said no: “I don't think that that's scalable downward.”

Jensen's setup worked because his direct reports were experienced executives who needed very little managing.

Even then, it came at a cost. After Jensen resigned from Nvidia last week, former Fortune 500 CEOs told Fortune that with 60 direct reports, none of his lieutenants may have been tested enough to take over.

A wide span can look efficient for a long time. At Nvidia, the cost only showed up when Jensen left.

Agents need a lot more checking than seasoned executives do, and many companies are about to hand that checking to managers without counting the time.

The supervision work doesn't show up in the headcount

Gartner calls this load the span of complexity. It looks at how many agents a team runs and how much autonomy they have. It also weighs how tightly their output is tied to business risk.

Gartner gives the example: A manager with eight people who each rely on several autonomous agents is dealing with far more than eight people's worth of work.

The same article says 75% of CHROs report managers are more overwhelmed than ever. The survey didn't measure how much of that comes from agents. But it tells you what you're adding agents to: managers who are already stretched.

Gartner recommends formally building AI oversight and escalation into managers' roles, with the time and authority to do it. It also warns that companies widening spans of control without accounting for agents will likely need to rehire middle managers to restore oversight.

Managers aren't the only ones losing time to agents. The people on their teams are spending time keeping them in check, too.

In Asana's October research, 46% of office workers using at least two agents a week estimate that more than 10% of their time goes to coordinating them. That includes choosing which agent to use, switching tools, managing outputs, and resolving conflicting results.

I'd expect a manager to feel this twice: once for their own agents, and again when someone on their team brings them the conflicting results.

In interviews with 17 experienced developers, Microsoft researchers found four forms of oversight. Two happen before the agent starts: setting constraints and working through the plan with it. The other two are monitoring it as it runs and reviewing the result.

Developers also said that each time the agent revised its work, they felt they had to review the whole thing again, because it might have changed things in other places.

I think if you only budget time for the final sign-off, you'll undercount the oversight. A lot of it happens before and during the work, and it repeats every time the agent tries again.

Many managers can't say who's accountable

Wharton's October report surveyed 1,007 managers and senior leaders, plus 409 other employees, at US companies with more than 1,000 employees.

Most of these companies already check their agents. Fewer than 1% of departments have no formal process for reviewing agent performance.

The gap is in who's accountable. When an agent makes a consequential error, managers and senior leaders split:

  • A governance or risk team: 28%

  • The deploying employee's manager: 26%

  • The employee: 25%

  • The vendor: 9%

  • Nobody, or they don't know: 11%

The split by level is starker. 20% of managers and directors said nobody is clearly accountable, or they can't say. At the VP level and above, it was 3%.

So the managers most likely to deal with an agent's mistake are the least sure of whose mistake it is.

Putting agents on the org chart doesn't settle it either.

A September working paper from Boston University and BCG surveyed 1,261 managers. 23% said their company already lists AI agents on its org chart. The researchers then gave managers identical documents with errors built in, and changed only who supposedly wrote them.

Across all managers, the label made little difference. But in companies that already list agents on their org charts, calling the author an "AI employee" instead of an "AI tool" cut the share of errors caught by 17%. Those managers were also 22 percentage points more likely to ask someone else to review it, and they saw themselves as less accountable.

The researchers call this a hot potato effect. It's one experiment, but it suggests that treating an agent like a coworker can make the person reviewing it feel less responsible.

Microsoft's 2026 Work Trend Index puts the questions well. Who reviews agent performance? Who has the authority to update the workflows agents run? And how does a local win get captured and scaled across the organization?

What it looks like when someone designs for it: BNY

In Microsoft's BNY case study, the bank has nearly 140 "digital employees." Each one has its own employee number and a supervisor who assigns tasks and reviews output.

Before any agent goes live, it passes through 16 steps of review boards, compliance gates, and model risk checks.

The first payment agent had one narrow job. It reads the vendor address on a cross-border payment and calls a mapping service to work out whether "York" means the UK, South Africa, or Western Australia. Then it validates the country code and submits the corrected payment for a human to review.

That tight scope was deliberate. At a bank where one misrouted payment can freeze a billion-dollar transfer, a narrow job is what makes the agent easy to check.

The people supervising these agents watch a live dashboard of processed volume, cycle times, missed transactions, and approval outcomes.

Even with all that planning, supervision took work. Reviewers were sending some of the agent's responses back. The responses weren't wrong, but they didn't match how the work used to be done, so the team coached and retrained it the way they would a new hire.

BNY says payment validations that took five or six minutes now clear in under 30 seconds, and open payment investigations dropped by nearly 80%. Those are the company's numbers, in a case study published by one of its vendors.

What I take from BNY is that the speed came with the oversight built in. Every digital employee has a named supervisor, and the first one was kept to a single narrow job so it was easy to check.

■ Your turn

Run an oversight workload audit

Pick one team and the AI workflows it runs every week. For each one, write down:

  • The job and what it's allowed to do. Does it draft something, or can it change something consequential?

  • The business owner. Who decides whether the work meets the team's standard?

  • The technical maintainer. Who fixes connections and permissions when they break?

  • When a person must step in. Which decisions need approval, and which failures trigger escalation? OpenAI's agent guide points to two triggers: an agent failing after repeated attempts, and high-risk actions like canceling orders, authorizing large refunds, or making payments.

  • Actual supervision time. That covers reviewing, correcting, handling exceptions, and updating instructions. If nobody knows the number, have the team log it for two weeks.

  • Backup and pause authority. What happens when the owner is out, or the workflow starts failing?

The business owner and the technical maintainer don't have to be the same person. But if a manager is accountable for a workflow and can't get it fixed or stop it, they're accountable for something they can't control.

Gartner recommends auditing spans of complexity but doesn't publish a scoring model, so this checklist is our own version.

Why counting agents won't tell you much

Let's try an example with made-up numbers.

Say one workflow handles 500 cases a week, and 4% need a person to step in. At 10 minutes each, that's 20 interventions and 200 minutes a week. And that's before routine reviews or maintenance.

Now change one number. At 2%, it drops to 100 minutes. At 10%, it's 500 minutes, which is more than a full working day every week.

Compare that with a manager running five narrow workflows that each handle 100 cases a week, with 1% needing a person. That's five interventions and 50 minutes a week in total.

So five agents can take less of a manager's time than one. What matters is how often a person has to step in, and how long it takes each time.

For a first trial, track three things

  • Supervision time: reviews, corrections, and exceptions.

  • Results: accepted work completed, and turnaround time.

  • Quality: mistakes, rework, and unresolved exceptions.

Compare all three with the old process. AI can reduce total work and still add oversight, and tracking these over a few weeks shows you which way it's going.

Then make the call before you expand

Once you can see the oversight work, you have four moves:

  • Remove reviews that aren't needed. Some checks exist out of habit. Start with low-risk work that's easy to undo, and keep reviewing high-risk actions until the agent has earned more trust.

  • Clarify authority. Name who approves, and who can pause the workflow.

  • Improve the workflow so fewer cases need a person. As the math shows, this is often where the biggest time savings are.

  • Allocate the time. Put oversight in the manager's goals and on their calendar.

■ Signing off

That's it for this week.

One question: what did you take off your managers' plates when you added AI oversight? Hit reply. I read every reply, and the best answers will shape what we look into next.

Until next time,
Haroon

Get AI Ready

Our weekly newsletter, from Haroon Choudery and the AI Ready team, for people putting AI to work in a business.

More issues