AI use cases and applications
Business Metrics for Generative AI
Accuracy alone doesn't show value
The story: A chef can cook a technically perfect dish. But the restaurant owner cares whether diners come back, spend more, and leave happy. A perfect dish nobody orders isn't worth much.
In AI/AWS terms: Model accuracy alone doesn't show value. Business metrics tie a generative AI application to outcomes and ROI, and the right metric depends on the use case.
For the exam: Judge generative AI by business outcomes, and pick the metric that fits the use case.
The five metrics
The story: The restaurant owner checks:
- The star ratings diners leave.
- How much each diner spends on average.
- Whether the chef is just as good at breakfast, lunch, and dinner.
- How many people who look at the menu outside actually come in and order.
- How much time and money the new kitchen tools save.
In AI/AWS terms:
| Restaurant check | Metric | What it tells you | Typical use case |
|---|---|---|---|
| Star ratings | User satisfaction | How users rate the generated content or recommendations | Customer support, e-commerce sites |
| Spend per diner | Average revenue per user (ARPU) | Revenue per user | Personalized recommendations and upselling |
| Breakfast, lunch, dinner | Cross-domain performance | How well the model handles different domains or tasks | Assistants used across departments |
| Lookers who order | Conversion rate | Share of users who complete a desired action, such as a purchase | Marketing content, product recommendations |
| Time and money saved | Efficiency | Time, cost, or resources saved | Automating document or content work |
For the exam: For customer support, the best measure of success is customer satisfaction, not revenue.