ai-ledger

Does This Decision Need an LLM? The Technical Appendix

The technical appendix for _Does This Decision Actually Need a Large Language Model?_: what “decision model” means, a dated provider snapshot, probability vs. confidence vs. calibration, a cost-per-decision example, contract restrictions, how to score a pilot, and a walkthrough from pilot to production

PythonMuse LLC Published October 2026

This page is for readers who want to evaluate a decision model, compare providers, set confidence bands, or run the scoring demo. The article stands on its own; nothing here changes its argument. Think of this as the workpapers behind the memo.

1. Terminology: Three Meanings of Decision Model

The phrase “decision model” gets used three different ways, a bit like “reserve” can mean an allowance, a restricted fund, or an oil and gas estimate depending on who is talking. Each row below is one of those uses: what the person saying “decision model” might actually mean.

When someone says “decision model,” they may mean… Where you’ll see it used this way What it is Did it learn from your company’s past data?
The method a step uses AI Helped Me Build It. AI Does Not Have to Run It. Whatever reaches the step’s result: a written rule, an LLM, a model trained on your history, or a qualified person Depends on the method
The new kind of AI product (the one this article is about) This article A pretrained model that reads text or JSON and returns a bounded, typed answer (a choice, a yes-or-no probability, or a score on your scale) instead of free-form text No. Your categories arrive as instructions with each request
Traditional machine learning The “patterns” category in the four kinds of work (rules, patterns, language, judgment) A predictive model (credit risk, delinquency, fraud scoring) trained on your own labeled history Yes. The logic comes from your data

So the decision models in this article sit between the “language” and “patterns” buckets from the four kinds of work in AI Helped Me Build It. AI Does Not Have to Run It.. They read messy language like an LLM, but they answer like a classifier. They are not the “patterns” tool, because nothing about your history is baked into them. Change the categories in the request and the same model happily sorts into the new ones.

Classification and predictive modeling are decades old. What is new is that decision-oriented models are available through the same AI platforms accountants and developers already use, priced and packaged for high-volume workflow steps.

2. Provider Landscape (October 2026)

A dated snapshot. Microsoft’s and OpenAI’s offerings launched this month, and OpenAI’s is in beta; prices and availability will change. Check the provider’s current documentation before relying on any line here. A note on the third column: Vercel is not a model maker. Its AI Gateway is a service that lets one application call models from many vendors, including these decision models, through a single connection. It is included here because its public documentation is vendor-neutral and unusually specific about how decisions, confidence, and fallbacks behave.

  Microsoft-Decision-1 OpenAI Decisions API Vercel AI Gateway (decision modality)
What it is A decision model, deployed in Microsoft Foundry (also available through OpenRouter). Microsoft says it built it by post-training Qwen3.5-9B. An API endpoint for decisions; gpt-6-luna is the only model currently available A gateway, not a model: one API in front of several decision models (including TypeSafe AI’s Jev, Liquid AI’s d1, Perplexity’s Decider, Microsoft-Decision-1, and OpenAI’s Decisions API)
Release stage Available in the Foundry model catalog; the documentation we checked carries no preview label Public beta; OpenAI says it expects general availability “in the coming weeks” Available; the AI SDK function is still named experimental_decide
Question types noul (yes/no), choice, score predicate (yes/no), choice, score Boolean, Choice, Score (also accepts OpenAI’s and TypeSafe’s formats)
What comes back A probability for yes/no; the selected option plus a probability for every option; a probability-weighted score plus per-level probabilities. No written rationale. A probability for predicates; the choice plus a probability distribution and a separate confidence field; a probability-weighted score Probabilities from the underlying model; Choice and Score confidence only when a native decision model answered
Price (as published) $0.042 per million input tokens; output tokens free $0.10 per million input tokens; no output, cache-read, or cache-write charges The underlying model’s rates; “some decision models price input tokens only.” A triggered fallback bills both stages
Notable caveats “Scores can change based on how you phrase or order questions and options.” Don’t use scores “as the sole basis for decisions about individuals.” “Calibration is strongest on familiar task types.” Use labeled examples from your own application to set thresholds Only native decision models report confidence; a language-model fallback does not (see Trap 2)

Anthropic (Claude). As of October 2026 we found no native decision endpoint and no probability output in Anthropic’s API documentation: the Messages API reference has no log-probability parameter, and nothing in the documentation index describes a decision or classification endpoint. What Claude does have is structured outputs (generally available), which constrain the response to a JSON schema, including an enum of allowed categories. So a Claude model can reliably return one category from your list. It cannot tell you how likely each category was. If you add a "confidence" field to the schema, the model will fill it in, but that number is generated text, not a measured probability. Vercel’s own examples use a Claude model as a language-model fallback for decisions, which fits: it returns a structured answer with no confidence attached.

And the harness. Whichever provider runs the decision, the code around it (the pilot worksheet, the scoring script, the band logic, the logging) is ordinary software you can build with a co-pilot. We built ours with Claude Code in Visual Studio Code; OpenAI Codex or GitHub Copilot would work the same way.

3. Probability, Confidence, and Calibration

These three words get used interchangeably. They are not the same thing.

Three different numbers

The practical rule: neither the probability nor the confidence is calibrated to your data until you have checked it against your data.

Designing the bands

A confidence band is a range of probabilities that gets the same treatment, the way an expense policy’s approval limits work, except the trigger is the model’s probability instead of a dollar amount. A typical design has three:

Where one band ends and the next begins (the cut-offs) is yours to set, from your own results:

  1. Define the allowed answers first, including an escape hatch. Microsoft recommends an abstention option “such as cannot tell, when the model shouldn’t make a forced choice.” Without one, an unclassifiable note still lands in a real category.
  2. Run the model on labeled history (see Scoring a Pilot) and record the probability for every item, right or wrong.
  3. Look at where the wrong answers sit. Vercel’s guidance: “Choose thresholds from your own results and risk tolerance. Start by recording the confidence or Boolean probability ranges associated with incorrect primary answers.”
  4. Set the auto-route line above the errors you can’t live with. Weigh what a wrong answer costs in that band. Microsoft: “Set confidence thresholds based on the cost of false positives and false negatives.”
  5. Freeze the wording. Microsoft warns that “scores can change based on how you phrase or order questions and options.” Test whether shuffling the option order changes results, then treat the question text as a controlled setting.
  6. Write the policy down, including what each band may not do. The Confidence Band Routing Policy template has the structure.

Trap 1: rounding a score

A score is a probability-weighted average of the level numbers. OpenAI’s documentation notes that because of this, it “can fall between levels.” That is useful for ranking, but rounding it to pick a level can land on a level the model barely considered.

Take a three-level scale: 0 = Routine, 1 = Review, 2 = Escalate. If the model puts 50% on Routine, 0% on Review, and 50% on Escalate, the score is (0 × 0.5) + (1 × 0) + (2 × 0.5) = 1.0. Round it and you get “Review,” the one level the model gave zero probability. The model wasn’t saying “review”; it was saying “this is either nothing or serious, and I can’t tell which.” That is an escalation, not a middle answer.

Microsoft’s guidance points the same way: “Prefer scores for relative ordering and thresholds rather than as absolute, calibrated ratings.” If the workflow needs a category, ask a choice question, or look at the per-level probabilities, not the rounded score.

Trap 2: a fallback that returns no confidence

Vercel AI Gateway can rerun an uncertain decision on another model, including an LLM. Vercel is explicit about what comes back: “Only native decision models report confidence. A language-model fallback returns one structured value per question, so its Choice and Score answers have no confidence.”

On Vercel’s TypeSafe-compatible API, that missing value arrives as confidence: 0 with an empty probability map, and Vercel warns that this means the value is unavailable, not that the fallback “measured zero confidence.” A band rule written as “below 0.6 → escalate” would happen to escalate it; a rule written as “0 means certain the answer is wrong” would misread it. Give “no probability” its own band, and route it to a person. The scoring demo in Scoring a Pilot does exactly that.

This is also the clearest illustration of the article’s point: an LLM can choose a label, but it doesn’t tell you how sure it is.

4. Cost per Acceptable Decision: An Example

The article’s measure is:

Cost per acceptable decision = (model cost + review cost + cost of uncaught errors) ÷ decisions that ended up right

In this simplified model, an item a person confirms or escalates ends up right; only auto-routed errors slip through.

Synthetic assumptions: 10,000 collection notes a month, 500 input tokens each, a reviewer at $60 an hour ($1 a minute), and $50 of rework and delay for each error nobody catches. Prices are illustrative, not any provider’s.

  Decision model + bands Cheapest model, loose bands LLM paragraph, every item read All manual
Price per million input / output tokens $0.10 / $0 $0.05 / $0 $2.00 / $10.00 (300 output tokens) n/a
Auto-route / confirm / escalate 70% / 20% / 10% 50% / 0% / 50% 0% / 100% / 0% 0% / 100% / 0%
Minutes per confirm / escalate 1 / 6 1 / 6 1.5 / n/a 3 / n/a
Error rate on auto-routed items 1% 2% n/a n/a
Model cost $0.50 $0.25 $40.00 $0
Review cost 2,000 × 1 + 1,000 × 6 = $8,000 5,000 × 6 = $30,000 10,000 × 1.5 = $15,000 10,000 × 3 = $30,000
Uncaught errors 7,000 × 1% = 70 → $3,500 5,000 × 2% = 100 → $5,000 0 0
Total $11,500.50 $35,000.25 $15,040.00 $30,000.00
Acceptable decisions 9,930 9,900 10,000 10,000
Cost per acceptable decision $1.16 $3.54 $1.50 $3.00

Three things to notice:

Every figure above was calculated by hand, then checked against the cost workbook (recalculated in Excel) and a Python mirror of its formulas that the test suite pins down. Replace the inputs with your own; the formulas are all visible.

5. Contract and Policy Restrictions

A decision model writes fewer words. It processes the same data. Before one goes near customer information, the same review applies as for any AI tool, and a few clauses deserve a specific look. Your AI Workflow Was Approved. Did Anyone Read the Customer Contract? covers the full contract review; this is the decision-model angle.

Look for Why it matters for a decision model
Restrictions on AI, machine learning, or “automated processing” generally They usually cover classifiers too. A decision model is not a loophole.
Restrictions on automated decision-making Auto-routing is an automated decision, even if a person approves later. Know which band triggers the clause.
External processing, subprocessors, and data residency The model may run at a provider or behind a gateway. Know where. OpenAI, for example, documents US and European data residency for its Decisions API; a gateway adds another party to the chain.
Retention and training on customer data Check the provider’s retention terms, and whether zero-data-retention options apply to your account.
Notice to the people affected Microsoft’s guidance: “Tell affected users when AI contributes to a decision.”
Decisions about individuals (credit, collections standing, employee expenses) Microsoft: use the model “for decision support with meaningful human review. Don’t use it as the sole decision-maker.”

Two design habits make the answers easier:

If the only approved tool is a corporate AI tool, the same split (rules automated, bounded decisions on an approved model, judgment with people) still applies; see When Copilot Is the Only Approved AI Tool. And none of this is legal advice: contract interpretation belongs with counsel.

6. Scoring a Pilot

The pilot measures; it does not automate. The model takes no action while you test it.

The method

  1. Pick one decision with known outcomes: old collection notes with final reason codes, or AP exceptions that went to known teams. Use synthetic or approved data only.
  2. Write the allowed choices down first, with an “other” or “cannot tell” option.
  3. Run the model, or a few models, and record each item’s choice and probability. Leave the probability blank when none comes back.
  4. Score by band, not overall. An 80% overall accuracy can hide a band that is 100% right and a band that is a coin toss. The band is what decides the authority.
  5. Read the confusion table. Which pairs get mixed up? Pricing versus purchase order is a wording problem you may be able to fix; pricing versus “other” may be something worse.
  6. Find every high-confidence error. A wrong answer in the auto-route band is the one nobody would have reviewed. Grade each error’s severity: minor (wrong queue, a person catches it), costly (delay, rework), or control failure (cleared with no review).
  7. Ask what a simpler method would have done. Some items never needed a model: a quantity mismatch against the receiving record is a three-way match, and a blank PO field is a rule.

Run the demo

This article comes with a model-agnostic scoring script (also in the article’s GitHub folder, with its sample data). It never calls a provider and needs no API key: it reads a labels file and a predictions file from any source, including answers you typed in by hand. Download or clone the folder, then run from inside it:

python score_predictions.py
python score_predictions.py --auto 0.95 --confirm 0.70
python score_predictions.py --labels my_labels.csv --predictions my_predictions.csv

With the sample data (collection_notes.csv and sample_predictions.csv) and the demo’s illustrative bands (auto-route at 0.90 or above, confirm at 0.60 or above), it reports:

Band Items Correct Accuracy
Auto-route 9 8 88.9%
Confirm 7 6 85.7%
Escalate 3 1 33.3%
No probability 1 1 100.0%
Total 20 16 80.0%

and one high-confidence error: AR-011, where the model said “other” at 93% for a note whose real issue was a pricing dispute.

The sample data is seeded on purpose, and the test file pins every case down so nobody “fixes” the lessons out of it:

Twenty items is a demonstration, not a pilot. In a real pilot you want enough items in each band that a single error doesn’t swing that band’s accuracy by double digits.

The same rows live in the Decision Model Pilot Worksheet, where Correct, Band, and a by-band summary are live formulas and the band cut-offs sit on a Settings tab. The workbook is generated by generate_worksheet.py, which reads the same CSV files, so the worksheet and the script always agree.

7. From Pilot to Production: A Walkthrough

Scoring a Pilot shows how to measure a model on past work. This section follows one workflow the whole way, from “could a decision model help?” to running it with controls, as an accountant would actually experience it.

The example: triaging customer emails in a shared accounts receivable inbox. Each email that explains a late payment gets one reason code (cash flow, pricing dispute, purchase order, quantity dispute, or other), and the code decides which team picks it up.

You are not building an AI system. You are writing a routing policy in plain English, testing it against answers you already know, and letting it act only where it has earned that.

0. Check that it is really a decision. The answers can be listed in advance, and the next step depends only on the code. Both true: it is a candidate. Strip out anything a rule can do first. A blank PO field is a rule. A quantity mismatch against the receiving record is a three-way match. Neither needs a model.

1. Get permission before touching data. An approved provider and environment, and a check of what the customer contracts and company policy allow (see Contract and Policy Restrictions). Start with synthetic emails, or historical emails approved for testing. The questions from If the Data Can’t Leave, Bring the Model In still apply.

2. Write the question like a policy. One sentence for the question, a one-line definition for each choice, and an “other” option. This is the accountant’s job and the heart of the work: the definitions are where accounting knowledge goes in, and vague definitions produce vague probabilities. In the shape shown in Vercel’s documentation (OpenAI and Microsoft use the same idea with their own field names):

{
  "state": "We cannot release payment because the PO number on the invoice does not match our system.",
  "questions": {
    "reason": {
      "type": "choice",
      "instructions": "Why is this customer not paying?",
      "criteria": {
        "cash_flow":        "Customer cannot or will not pay yet for financial reasons",
        "pricing_dispute":  "Customer disagrees with the price billed",
        "purchase_order":   "PO number missing, wrong, or not matching their system",
        "quantity_dispute": "Customer disagrees with the quantity billed or received",
        "other":            "None of the above, or not enough information"
      }
    }
  }
}

The answer comes back as a probability for each choice, for example purchase_order 0.96, pricing_dispute 0.02, ....

3. Prepare the email (intake, not AI). The real input is an email, not a tidy note, and most of the preparation is deterministic:

Shared AR inbox
  -> match to customer and invoice   (rules: sender domain vs. customer master,
                                      invoice or PO numbers in subject and body)
       no match? -> LLM, if approved, suggests customer and invoice references
                    from the messy text -> rules re-check against the customer
                    master and open invoices -> still no match: a person
  -> filter out non-collection mail  (rules: auto-reply headers, remittance advice,
                                      "payment sent"; or a yes/no decision question)
  -> keep only the newest message    (strip the quoted thread, signatures, disclaimers)
  -> decision model: reason code

4. Build the answer key. Pull 100 to 200 old emails that already have final reason codes. Split them: a practice set where you may adjust the wording, and a scoring set you do not look at until the end. It is the same discipline as not testing a control with the evidence used to design it.

5. Run it with no actions. No programming background needed. In Visual Studio Code, ask your AI co-pilot (Claude Code, in our case) something like:

“Write a Python script that reads practice_emails.csv, sends each email body to [provider]’s decision endpoint with the question in question.json, and writes item_id, model_choice, probability to predictions.csv. Read the API key from an environment variable. Do not route or change anything.”

The same request works for OpenAI’s Decisions API, Microsoft-Decision-1, or Vercel AI Gateway; only the endpoint changes.

6. Score it. Run score_predictions.py against your own files (as in Scoring a Pilot) for accuracy by band, the confusion table, and the high-confidence errors.

7. Fix the wording, then freeze it. If pricing and purchase-order disputes get mixed up, sharpen those two definitions on the practice set. When it is good enough, score the held-out set once and record the wording and model version. Wording is now a control setting: changing it means re-testing.

8. Decide what each band may do. A band is a range of probabilities that gets the same treatment: auto-route, confirm, or escalate (see Designing the bands). Fill in the Confidence Band Routing Policy with cut-offs taken from your errors, and check the Cost per Acceptable Decision workbook: if half the emails land in “confirm,” the cheap model may not be saving anything. Get sign-off.

9. Shadow mode. For a few weeks the model suggests a code while collectors work exactly as before. Compare. Nothing depends on the model yet.

10. Go live narrowly.

Who does what

Role Owns
Accountant Choosing the decision, writing the choices, building the answer key, setting the bands, reviewing errors, signing off
AI co-pilot Writing the intake and calling scripts, running the scoring, drafting documentation
IT or system owner Provider approval, the API key, the inbox and AR system connections
People in the loop Every confirm and escalate item, every email still unmatched after the LLM’s suggestion is checked, anything that touches the books

At no point does the model approve, post, or contact the customer. It sorts. That is the entire job.

8. The Toolkit

Every item below opens or downloads from its link. To get them all at once, the complete set is in this article’s folder in the ai-ledger repository on GitHub. All of it uses synthetic data, and all of it is meant to be copied and adapted:

Sources

Every provider-specific statement in this article and its appendix was checked against the provider’s own documentation in October 2026:

Products in beta change quickly. Re-check them before you rely on one.

Back to the article: Does This Decision Actually Need a Large Language Model?

*Related: AI Helped Me Build It: The Technical Appendix From AI Answers to Audit Trails Your AI Workflow Was Approved. Did Anyone Read the Customer Contract?*

© 2026 PythonMuse LLC. Content licensed under CC BY-NC-SA 4.0; code licensed under MIT.

This site is open source. Improve this page.