Getting Started: Your First Decision Model Experiment

Image placeholder

Editorial, product-forward hero illustration on an off-white background — a horizontal arrangement of three large index cards or step cards, labelled sequentially '1. PICK', '2. DEFINE', '3. TEST'. The third card has a small stopwatch icon in its corner and a vertical column of 20 tiny checkboxes, some ticked (emerald green), some unticked (pale slate), one circled in navy as an edge case. Thin pale-slate dotted guideline runs underneath the three cards, with a tiny label 'one day' at its midpoint. Palette strictly off-white background, pale slate grays, one emerald green accent for the checked boxes and ticks, one navy ink for the labels. Clean, flat, geometric. Linear or Stripe editorial feel. No AI robots, no neural networks, no faces, no brain imagery.

• Aspect ratio: 16:9
• Source size: 1600 × 900 px (sharp on retina at 760 px display width)
• Format: WEBP (preferred), PNG fallback
• Target file size: under 300 KB

Save to /public/images/blog/<slug>/, then set src and alt on this <MediaFigure /> to swap it in. Keep the ratio prop so the browser reserves the right space.

By Dr. Sanju Abraham, Fractional CTO — 180+ projects shipped, 34 funded companies. Published October 6, 2026.

Key takeaways

• The smallest honest test of a decision model is three steps and twenty examples. Pick one repetitive task, write the question clearly, and run it against twenty real cases from your own data.
• Phase 1 is picking a task small enough to fail cheaply. A good starter is a repeat yes-or-no or sorting job your team already does by hand every week.
• Phase 2 is writing the question and the list of possible answers out loud. If you cannot write it in one sentence, the model will not answer it well either.
• Phase 3 is twenty real examples, scored against a ground-truth column you fill in yourself. We call this the 20-example rule because it is the smallest sample that reliably separates “promising” from “noise”.
• The decision to expand is not made by the model’s score; it is made by looking at where the model was wrong. Patterns in the errors tell you whether the next investment is worth it.

The short answer: A first decision model experiment should take one focused day and cost less than a lunch. Pick one repetitive task, write the question and the possible answers in plain English, feed twenty real examples to the model, and compare the answers against your own labels. The whole exercise is a three-phase loop — pick, define, test — and you can run it today without hiring anyone or writing a backend. This piece is a coaching walk-through of your first rep.

Most teams spend three months arguing about whether AI “will work for us” and never run the one experiment that would settle the question in an afternoon. The reason is not laziness; nobody has told them how small the first test is allowed to be. By the end of this piece you will know exactly how to run it.

What is the 20-example rule?

The 20-example rule is this: the smallest honest decision-model test is twenty real cases from your own data, scored against a column you labelled yourself. Smaller than that and you are testing vibes; larger than that and you are stalling. Twenty is the sample size at which patterns in the wrong answers stop being coincidence and start being signal.

Twenty is a product-management number, not a statistics number. It is a decision aid, not a confidence interval. The question it answers is not “is this model production-ready?” — a serious production bar needs thousands of examples and a calibration run, which we covered in why “70% sure” is more useful than “yes”. It answers the earlier one: should I invest another week?

Twenty examples fit on one screen. You can see every call the model got right and every call it got wrong without scrolling. Patterns jump out in a way they never do in a thousand-row spreadsheet. The whole point of the first experiment is to make the errors visible to you, in your own voice.

Phase 1: Which task should you pick first?

Pick a repetitive yes-or-no or sorting task your team already does by hand every week. Not the most important task. Not the hardest task. The most repetitive one.

The ideal first-experiment task has five properties, and if yours has four of them you are in good shape:

• Repetitive. The same shape of question gets asked many times a week. Routing a ticket. Tagging a review. Deciding whether a form submission is spam. The reward of success is time multiplied by volume.
• Reversible. The downside of a wrong answer is small — a mislabel, a mis-routed email — not a wire transfer or a legal filing. First experiments belong on low-stakes decisions.
• Already digital. The input to the decision is text or structured data that already exists in a system. If the input is a scanned PDF or a handwritten note, you are running two experiments at once (OCR and decisioning) and both will suffer.
• Owned by you. You can see the ground truth. If the “right answer” requires asking three departments, you cannot score the experiment. Pick a task whose correct answer is obvious to one person — you — in a few seconds.
• A single-shape question. The answer fits one of three forms: choose from a list, rate on a scale, or verify yes-or-no. If the answer needs a paragraph, you are not running a decision-model test yet. For the full taxonomy, see the three questions a decision model can answer and the companion list in five everyday business tasks a decision API can handle.

Common first-experiment tasks that fit all five: tagging the category of inbound support emails, flagging obvious spam in a contact form, scoring the urgency of a Slack message, deciding whether a code of conduct comment is on-topic, classifying a vendor on a receipt. Pick one of these if nothing in your own workflow jumps out.

The failure mode in Phase 1 is picking a task that is operationally important but experimentally bad. “Pricing” is a classic trap: the right answer is contested, inputs are everywhere, and the downside is real money. Save it for experiment six.

Phase 2: How do you write the question clearly?

Write the question in one sentence and list the possible answers underneath it. If you cannot do that in two minutes, the task is not ready for a decision model and no amount of prompting will fix it.

The three decision shapes map directly onto the shape of your question. If the answer is one of several labels you already know, you are writing a Choose question. If the answer is a number on a defined scale, you are writing a Rate question. If the answer is yes or no with a probability you can threshold, you are writing a Verify question. These map onto the Choice, Score, and Noul primitives in TypeSafe AI’s Jev, the enum, number, and boolean fields in OpenAI’s Structured Outputs, and the equivalent tool-input schemas in Anthropic’s strict tool use.

Three Phase-2 rules that save you a round trip:

• One question per experiment. It is tempting to ask the model to categorize, score urgency, flag spam, and extract the customer name in one shot. Save it. Your first experiment is one question against twenty examples; everything else is scope creep in disguise.
• Finite answer list, written out. For Choose, write every label. For Rate, write what each number means (“5 = critical, 1 = ignore”). For Verify, write the exact statement being verified. The model’s answer quality is bounded by the clarity of this list.
• One sentence of input context. Tell the model what the input actually is. “A customer support email body, pasted as text.” “A one-line comment from a user.” Decision models work better when they know what they are looking at, and worse when they have to guess.

The failure mode in Phase 2 is prompt-engineering gymnastics. If you are writing a 300-word prompt with examples and reasoning steps, you are no longer running an experiment; you are negotiating with a model. Keep the question short.

The reusable template you can copy today

Here is the whole experiment on one page. Open a note, paste the block below, fill in the five fields, and you have a plan.

# Decision Model Experiment — [YYYY-MM-DD]

## 1. The task
A one-sentence description of the repetitive job you are testing.
Example: "Tag inbound support emails by queue so routing happens
before a human reads them."

## 2. The input
What are you handing the model, exactly?
Example: "The plain-text body of one support email, no headers."

## 3. The question (one shape only)
Pick one of: Choose / Rate / Verify.
Example (Choose): "Which queue does this belong in?"

## 4. The answers
List the full set, with one-line anchors if it is Rate.
Example:
  - billing
  - technical
  - sales
  - other

## 5. The 20 examples
Grab 20 real cases from the last week. Fill in your own label first
(ground truth), then run the model, then compare.

| # | input (short) | my label | model label | model confidence | match? | note |
|---|---------------|----------|-------------|------------------|--------|------|
| 1 |               |          |             |                  |        |      |
| 2 |               |          |             |                  |        |      |
| … |               |          |             |                  |        |      |
| 20|               |          |             |                  |        |      |

## 6. The decision
After scoring, answer three questions in writing:
  - How many did it get right? (count, not percent)
  - Where did the mistakes cluster? (describe the pattern)
  - Would fixing the top error pattern be a prompt change, a schema
    change, or a different task? (one sentence)

Copy that template into any note tool. The experiment is designed to fit on one screen so the pattern in the errors stays visible to you while you look at them. Teams that put the twenty rows in a scrollable spreadsheet lose the whole point.

Image placeholder

Infographic: a labelled, product-forward spreadsheet-style grid showing a filled-out version of the 20-example scoring table. Columns across the top: '#', 'input (short)', 'my label', 'model label', 'confidence', 'match?', 'note'. Twenty rows beneath. Most 'match?' cells show a tiny emerald green check; three rows show a navy X with a short note like 'ambiguous — mentions both billing and tech' or 'policy edge case'. One row is highlighted with a soft emerald tint to call out an edge case the human got wrong first. A sidebar on the right summarizes '17 right / 3 wrong / 2 confidence < 0.6' in simple tiles. Clean, flat, geometric. Palette: off-white background, pale slate grid lines, emerald green for matches, navy ink for text, one amber highlight for the edge-case row. No AI robots, no neural networks, no faces.

• Aspect ratio: 16:9
• Source size: 1280 × 720 px (sharp on retina at 760 px display width)
• Format: PNG
• Target file size: under 500 KB

Save to /public/images/blog/<slug>/, then set src and alt on this <MediaFigure /> to swap it in. Keep the ratio prop so the browser reserves the right space.

Phase 3: How many examples do you really need?

Twenty is the minimum useful number; a hundred is diminishing returns for a day-one test. Fewer than twenty and random noise swallows the signal. More than twenty and you are deferring a decision you can already make.

Phase 3 is a mechanical pass, not a strategy pass. The sequence is:

• Pull twenty real examples from the last week. Not synthetic ones, not cherry-picked clean cases, not the ones that are “fun” to show a model. Real volume, real messiness, real distribution. If the real distribution is 80% boring and 20% weird, your twenty should reflect that.
• Label the ground truth yourself first. In the “my label” column, write what the correct answer is, before you look at the model output. This step is non-negotiable; without it you are grading on vibes. Spend fifteen minutes.
• Run the model once and paste its answer. Use any decision model with a schema — OpenAI, Anthropic, Google, or a dedicated decision product. One shot per example. Do not iterate on prompts between rows; that smears the experiment.
• Mark match or no-match and jot a one-line note on the mismatches. “Model said billing, I said other, because the email is really about partner access.” The notes are where the next experiment comes from.

Costs for this pass are small enough to ignore. Twenty short-text decisions through a mainstream API typically cost pennies, well below the price of a sandwich; see do you need to be a developer to use a decision API? for the setup logistics. The time cost is the real cost — plan for one focused ninety-minute block, not scattered minutes across a week.

The common error in Phase 3 is to run the model, see 17/20, and declare victory. The headline score is the least interesting output of the experiment. The notes column is the interesting output. Three mismatches with the same reason is a prompt fix; three mismatches with three different reasons is a task-redefinition conversation.

A worked example: tagging demo requests from a web form

Here is the whole experiment, start to finish, applied to a task most companies have.

A B2B company has a “request a demo” form on its site. The form field asks “Tell us about your use case.” Submissions range from one-sentence real buyers to multi-paragraph researchers to obvious students on assignment. Someone on sales ops spends an hour each morning reading them and dragging them into Salesforce lists: “qualified”, “nurture”, “not now”, “not a buyer”.

Phase 1 — pick. Repetitive (yes, 30+ per day), reversible (yes, mis-tagged leads cost minutes), already digital (yes, the form posts text), owned by sales ops (yes), single-shape (yes, Choose from four labels). All five properties present. Good candidate.

Phase 2 — define. The task: “Tag a demo-request submission into one of our four pipeline buckets based on the user’s own description.” The input: “The free-text ‘use case’ field from one form submission.” The shape: Choose. The answers: qualified, nurture, not_now, not_a_buyer, each with a one-line anchor pulled from the current sales SOP.

Phase 3 — test. Pull the last twenty submissions from the form log. Label each in the “my label” column first — sales ops knows the right answers. Run the model once per row. Paste its label and confidence. Mark match or no-match. Jot notes on the misses.

A typical result: 17/20 matches. Three misses — one student who asked in detail about pricing (model said not_a_buyer, sales ops said qualified), one single-word body (model said nurture at 0.52 confidence), one submission naming a competitor (model missed, ops caught it).

Phase 4 — decide. The headline is 85%, but the interesting part is the pattern: two misses are genuine ambiguity, one is a prompt fix (add “mention of competitor by name” to the qualified anchor). A re-run with the revised prompt lands at 19/20 on the same twenty rows. Now the team can have the real conversation about wiring this into Salesforce.

That is a decision-grade result in a day. For the broader frame, see an AI strategy for SMB owners that survives contact with reality.

How do you read the results and decide whether to expand?

Read the results by the shape of the errors, not the size of the score. A model at 85% with clustered, fixable errors is a better bet than one at 92% with random, uncorrelated misses.

Three questions to answer in writing after the experiment:

• Where did the mistakes cluster? If the three misses all involve one specific edge case — mentions of a competitor, non-English input, very short bodies — you have a fixable prompt or schema problem. If the three misses are all different, the task may be under-specified and worth rewriting.
• How many were low-confidence? The three-band rule from why “70% sure” is more useful than “yes” starts here. Count how many answers landed below 0.6 confidence. If that is 2–3 out of 20, your human-review queue is manageable. If it is 10 out of 20, the model is telling you the task is hard.
• What is the next-week investment, concretely? One sentence. Not “we will think about scaling this.” Something like: “revise the qualified anchor to include competitor mentions, re-run on 50 new examples, decide on wiring.” A real next step beats a vague mandate.

The decision point: if the error pattern is fixable and low-confidence is manageable, you have earned the right to another day. Rewrite the question, run 50 examples, and start sketching three-band thresholds. If errors are scattered and confidence is low across the board, you have cheaply discovered this task is not ready — a quarter of engineering time you did not waste. For the broader taxonomy, see models that just decide and chatbot vs decision model.

Frequently asked questions

What task should I pick for my first decision model experiment?

Pick the most repetitive yes-or-no or sorting task on your team right now, provided its input is already digital and its downside of a wrong answer is small. Support email routing, review tagging, spam flagging on contact forms, and expense categorization are all strong starters. For a longer list with decision shapes, see the companion piece on five everyday business tasks a decision API can handle.

How many examples do I need to test on?

Twenty real examples is the minimum useful number — we call it the 20-example rule. Fewer than twenty and random noise swallows the signal. More than twenty for a day-one test is diminishing returns. After a successful first pass, re-run on 50 to 100 examples before wiring the result into any production workflow.

What if the model gets it wrong on some examples?

Expected and useful. The headline score is less interesting than the pattern in the errors. Three mismatches that share a reason is a fixable prompt or schema problem; three mismatches with three different reasons is a task-redefinition signal. Read the notes column, not the match column, when you decide whether to expand.

How much does the first experiment cost?

Pennies in API calls. Twenty short-text decisions through any mainstream decision model API — OpenAI, Anthropic, Google, or a dedicated product like Jev — typically costs less than a dollar at 2026 prices. The real cost is one focused ninety-minute block of your own time to label ground truth and interpret the output.

Do I need to code to try this?

To call the API, yes, lightly. The call itself is a dozen lines of code any junior developer can write in half an hour. The experiment’s hard parts — picking the task, writing the question, labelling the twenty examples — are business work, not coding. See do you need to be a developer to use a decision API? for the division of labor.

Which decision model should I use for a first experiment?

Use whichever API your team already has credentials for. On OpenAI, use Structured Outputs with an enum or boolean schema. On Anthropic, use strict tool use. On Google, use Gemini responseSchema. On a dedicated decision product, start with Choice, Score, or Noul. All four give you the same shape of answer for this experiment; the choice matters more at scale than at day one.

What is a passing score for a first experiment?

There is no fixed number, but a useful rule of thumb: 16/20 or better with clustered, fixable errors earns another day of investment. 12/20 or worse, or 20/20 with scattered errors, is a signal that the task is either under-specified or not a decision-model shape. The pattern in the misses tells you more than the count of the matches.

---

If you are planning your first decision-model experiment and want a sanity check on the task and the question, that scoping call is what a Fractional CTO engagement is designed for. One hour can save a quarter of engineering time. More pieces live in the AI Strategy archive.