AI Product Manager

Ninety percent of raters liked the agent. Who never got to vote?

Read an adoption announcement like a PM deciding what evidence to collect next.

By Ashley Kays Rabbitt · November 18, 2026

Airtable’s engineering team reported that positive ratings for Omni rose from 75% to more than 90% after its rebuild. It also reported that weekly active users increased 60%. Those are encouraging signals from the company. A PM deciding whether to expand their own pilot still needs two missing denominators: how many people attempted the job, and how many could rate the experience? Airtable Engineering • Rebuilding Omni

Imagine 100 people start a report-building task. Forty leave before the agent presents a result. Of the remaining 60, twenty submit a rating and eighteen give a thumbs-up. You can fairly say that 90% of raters were positive. You still don’t know how many of the 100 went on to use a correct report.

Who gets counted in the fictional pilot?
100started the task
20submitted a rating
18gave a positive rating

Inspect the people who left

Sample a handful of abandoned attempts with appropriate permission. Separate early departures into observable categories: no data source connected, generation took too long, first result visibly wrong, customer left for an unknown reason. Preserve that last category. An analyst can observe a closed tab without knowing why someone closed it.

Add a success measure that exists even when nobody clicks the rating button. For this scenario, ask whether the person exported a report whose totals matched the source and used it in the intended review. Pair that check with a short follow-up to people who abandoned the flow.

The decision changes with the pattern. People who never connect a source may need a sample report. People who discover a wrong total need a computation fix. Average satisfaction could improve in either case, but each calls for a different week of product work.