The technical appendix for _If the Data Can’t Leave, Bring the Model In_: where the model can run, how to choose and test one, the companion test kit, and the controls you still need
PythonMuse LLC Published October 2026
This page is for readers who saw a use for local AI in their own work and want to try it. The article stands on its own; nothing here changes its argument. Think of this as the workpapers behind the memo.
The sections follow the order you would work in: decide where the model runs, run a first test, build a proper benchmark, then put the controls around it. Deal vocabulary is at the end if you need it.
“Local” is one end of a spectrum. Along it you might use:
These options are not interchangeable. Each comes with its own contract, access rules, and settings to review. Start from what your data classification policy and client contracts allow, then choose where the model runs, not the other way around.
The test is the same everywhere: same narrow task, same finished engagements, same pass bar. The worked examples in this series, including the test kit, were built with the Claude extension in Visual Studio Code, but nothing in the method depends on that choice.
A model’s size is quoted in parameters: the billions of numeric settings it learned during training. More parameters generally means more general knowledge, and more memory needed to run it. As a rough guide:
The trade-off is the same one described in Model Selection Is an Accounting Control: pay for the capability you need, not the capability you don’t.
You do not need a server for a first check. A decent laptop and a free desktop app are enough. Ollama, which we first met in Don’t Trust the Model to Find What You Already Know Is There, and LM Studio are two examples, not endorsements.
A free model is not always licensed for business. Qwen3-8B uses Apache 2.0, a standard permissive license. Llama 3.3 uses Meta’s own community license, which includes an acceptable-use policy, requires a “Built with Llama” notice if you distribute it, and requires very large companies to request a separate license.
Neither is good or bad. They are different contracts, and someone should read the one you’re relying on.
A benchmark is a fixed exam: the same documents, the same questions, and the same grading rules for every model you test. Public benchmarks measure general ability, and their questions can leak into a model’s training data. Your completed engagements test your own work, and they are almost certainly not in any model’s training data. Model Selection Is an Accounting Control explains why the public leaderboard has the shelf life of a banana.
This rule matters more than sample size: do not tune your prompts on the same engagements you score. Keep a practice set for adjusting the instructions and a separate scoring set for the verdict. You don’t test a control with the same sample you used to design it. Tune on the scoring set and you’ve measured how well the prompt fits those engagements, not how well the model does the work.
Start with 5 to 10 engagements on one task type. Before anything goes into production, widen the test to 50 to 100 historical engagements where the answers and supporting evidence are already known. Those are practical sizes for building confidence, not a statistical sample.
Historical does not mean nonconfidential. Use only material approved for the test, or redact or synthesize it.
Test several models on the same set: one or two smaller local models, a larger local model, and your company’s approved enterprise AI platform if it has one. For a local model, cost means hardware, power, and staff time to maintain it, not a per-use fee. Compare on that basis, or you’ll be comparing apples to invoices.
If the enterprise model does substantially better, you may have a model or hardware gap, and the answer may be a dedicated AI workstation rather than a server room. Only use test material cleared for sending to that vendor; cleared for internal use is a different approval. If neither model performs acceptably, the workflow may simply not be ready for AI.
The Local LLM Test Kit lets you practice the method before you use your own engagements. It includes:
The scoring set is deliberately booby-trapped with:
No Ollama yet? The kit also replays two sets of example responses, so you can watch the scoring work first. One of them gets every answer right and still fails. The article’s scorecard shows why. Setup steps are in the kit’s README.
Retrieval-augmented generation (RAG) means the AI looks things up in your approved folder first, then answers from what it found, ideally citing the exact document. It’s the difference between asking a colleague what they remember and asking them to pull the file. Bay Point’s Atlas, from IBM’s case study, uses this design. Citations make the answer checkable, which is the whole argument of From AI Answers to Audit Trails.
A model’s context window is its working memory: how much text it can consider at once. It’s the problem behind Why Claude “Forgets” and The PDF Token Trap. A small local model may not hold an entire deal room, which matters much less when retrieval hands it only the documents relevant to the question.
Hallucination is the usual name for a model making things up. The two practical defenses are the ones the article uses: require a citation for every answer, and explicitly allow the answer “I do not have enough evidence to answer this.” A citation is still a claim to check, not proof.
Running the model inside your building changes where the risk sits. It doesn’t remove any of it. You still need:
AI in Accounting Isn’t Just About Efficiency — It’s About Control is the long version of that list. Three items deserve their own note:
Model versions. The model you tested in March is not automatically the model running in September. Pin the version you approved, and retest before replacing it.
Prompt injection. A document can still contain instructions, and a local model can still follow them. When the Invoice Starts Giving Orders covers the controls.
The ethical wall, enforced in retrieval. Document permissions have to decide what the user may retrieve before the model receives anything. Filter at the retrieval step, by engagement and by user, and log what was retrieved for whom. Once a document is in the model’s context, it is too late to ask nicely.
The same ONBOARD checklist used across this series, applied here:
When the pilot works and people start depending on it, You Built an AI Workflow. What Happens If You Win the Lottery? covers how it graduates into a production process someone else can run.
For readers outside deal work:
Every link above was checked in October 2026. Download sizes, licenses, and product names change quickly, so re-check them before you rely on any of them.
Back to the article: If the Data Can’t Leave, Bring the Model In
© 2026 PythonMuse LLC. Content licensed under CC BY-NC-SA 4.0; code licensed under MIT.