Forget AI ROI: AWS's applied AI chief on what to measure instead

"Don't try and measure the ROI of AI."

It is not the first thing you expect to hear from a senior executive whose job involves selling AI products to businesses. But Colleen Aubrey, senior vice president of applied AI solutions at Amazon Web Services (AWS), thinks most businesses are asking the wrong question when trying to transform with AI.

"Stay focused on the business metrics you’ve always measured," she says, "and go deeper – unpack them even more completely. Because the real opportunity is to reshape how the business operates. AI needs to deliver your ROI in the form of changing your business metrics."

AWS, of course, was born of a reshaping of Amazon’s own core business. It grew out of the infrastructure Amazon built in the early 2000s and was an early mover in cloud computing. AWS now accounts for more of Amazon’s operating profits than its retail business.

Aubrey’s team is charged with yet another act of reshaping. She is building a new pillar to AWS’s B2B proposition: purpose-built agentic applications that leverage Amazon's unique business experience and expertise in mass-market retail. It’s potentially another well-placed bet.

The AI opportunity

Although business leaders across the board appear to acknowledge the importance of AI, many studies show that AI penetration is shallower than you might think.  For example, NatWest Group recently surveyed 1,400 mid-market businesses in the UK for its AI Adoption Report. It found that only 44 per cent currently use AI (with a further 41 per cent expecting to adopt it within five years).

Those businesses that have started their AI journey have done so with productivity as the lens, Aubrey points out. This means using it for drafting reports, summarising documents and outsourcing the routine parts of knowledge work. Aubrey thinks this is "a very nice way to learn" – but not where the value is.

What interests her is AI's potential to move the bottlenecks that have constrained businesses for decades, which means adopting more agentic approaches.

From 20 SKUs to 20,000

A good example of this ambition and potential is Amazon Connect Decisions, launched in April. Amazon manages around 400 million SKUs across one of the world's most sophisticated logistics networks, and the company has built foundation models to handle supply and demand planning at that scale. Aubrey's group packaged this expertise into an agentic application for external customers.

Connect Decisions gathers data, cleans and normalises it, tests it against 20-odd forecasting models to find the best fit, then presents the result to a human planner for a conversation. It might suggest, for example, Aubrey explains, "a conversation about weather patterns we need to take account of, or, last year, was there seasonality I need to adjust for?"

One early launch partner was Wells Vehicle Electronics, a relatively small enterprise, she says. Its planners went from being able to pay attention to 20 SKUs to 20,000 and cut inventory levels – and therefore working capital – by 7 per cent within the first few weeks.

"Historically it was untenable to be able to inspect and optimise 20,000 SKUs," she says. "Today it's very tenable”, thanks to agentic AI.

Trust you can watch happening

Her team also developed Connect Customer, which handles customer journeys. United Airlines is an early adopter of a capability AWS calls Live Sync, and the example Aubrey gives is one most travellers will recognise.

A flight is cancelled. The customer calls. Voice AI picks up already knowing what has happened: "I know that your flight has been cancelled. I've already booked you on the next available flight but happy to look at options."

While the voice conversation continues, the airline's app updates in real time on the customer's screen. Four alternative flights appear – difficult for an AI to read aloud, but easy to scan visually. The customer can say, "I really need to get there by tomorrow afternoon", watch the options narrow, then either tap a choice or simply say "I'll take the third one" and see the booking happen on screen.

Trust is the crux of it. When you let an agent represent your business, your reputation is on the line. So, are AWS’s customers letting agents go further – making purchases, signing agreements?

"No purchasing and agreements that I'm aware of," says Aubrey.

AWS has developed a philosophy it calls the AI teammate, with four defining attributes. It must drive a business outcome. It must learn, rather than requiring you to re-explain your context every morning. It must have genuine domain expertise. And it must earn trust every day by showing its working.

If a planner looks at a demand forecast and thinks it looks wrong, Aubrey explains, they can interrogate it: Which model did you use? Why has this spiked in week 45? Every piece of data the system collects is inspectable.

"Ultimately, I want to keep the control of the business and the creativity of the business in the hands of people," she says. "I'd like people to be able to work on the business and not in the business. I'd like AI to work in the guts of it.”

Customer zero

Amazon has always built a good portion of its own software, and that was part of what drew Aubrey to her current role after 21 years at the company. AWS now uses Connect Customer for its own customer service team.

The internal rollout of AI has been instructive – and messy by design. Partly that's enabled by Amazon's culture of innovation. "The nice thing about Amazon is that we are not great at standing still," she says. "People in the company tend to lean forward into uncertainty and ambiguity."

After the necessary security and data guardrails were in place, the company essentially let everyone loose. An internal site at AWS called "AI for me" allowed staff to publish prompts they had found useful, which colleagues could discover, rate and download.

Then people began publishing task agents. Inevitably, there were 45 versions of the same idea. She says, "You sort of let this chaos go for a bit."

Growth without headcount

One theme I hear repeatedly from entrepreneurs is that AI lets them scale without adding people – and that many of them rather like it that way, because a smaller company feels more personal.

Aubrey sees the same thing, including inside Amazon. The delightful discovery for many leaders, she suggests, will be finding that a familiar ceiling has simply gone.

"My constraint is no longer the size of my customer service team. If I can have them managing teams of voice AI and chat AI to solve routine things – and even prevent routine issues getting in the way of customer experience – now I'm managing a team of AI."

This has a direct bearing on a business’s ability to scale. A spike in contacts no longer demands a spike in people. It demands a different question: what is disrupting the customer experience, and what defect could I go back and eliminate?

Shorter docs, working demos

British business leaders are fascinated by Amazon's meeting culture – the two-pizza rule, the famous silent reading of memos at the start of meetings. Does any of it survive in the AI era?

"I think it's evolving," says Aubrey. "The principles remain the same." It is still rare for her to attend a meeting without a document to read first.

The value, she argues, lies less in the reading than in the writing. "If I have to write it down, it really needs to be crisp. It helps you to refine your thinking, and the process of writing it down is a creative process." For the room, it means everyone shares an understanding before the conversation starts.

What has changed is the format. For product development, Amazon has shifted to a one- or two-page document plus a working demo. This might be an app or a new page on a website, vibe-coded with AI, because building a functioning prototype is now fast. "Then I don't need you to write six pages about that. Just write two pages and then show it to me."

For example, her UX design team now delivers user experiences in code. They still work in Figma, but they move quickly to a working prototype. "Five of us in a room looking at a prototype is a whole different level of understanding than looking at static mocks."

Designers who could not previously code are now pushing code to engineers. So are product managers.

"Rather than spending time hypothesising on what's going to be great in the end,” says Aubrey, “really treat it more as a two-way door decision. Build it, get it out, get feedback, iterate."