I Let Three Local LLMs Read 116 Lines of Legacy VBA: Every Rule Found, None of the Reasons

By The Compiler Whisperer · October 16, 2026

My "last admin" article claims that the knowledge living in one person's head sits in three layers — code, data, procedure — and that the code layer is the one that survives the person. My VB6/VBA playbook claims the extraction step is about recovering the rules, not the code: the code is a lossy encoding of a decision someone made in 2004. Both are claims about what a machine can read. Claims about machines should be tested on machines, so I built a test.

The test is deliberately artificial, because a real legacy system has no answer key. I wrote a fictional Access/VBA module myself — 116 lines of order-processing code for a fictional parts distributor — and kept two private documents: the 12 business rules I intended when I wrote it, and 7 facts that exist only in the fictional original developer's head and are, by construction, impossible to derive from the code. Then I had three local models read the module and extract every business rule they could find, and scored the results against the answer key.

The short version: a 12-billion-parameter model running on my own hardware extracted all 12 rules in 72 seconds, with zero invented rules. Not one model recovered a single one of the 7 planted facts. The machine read the code layer completely and was blind to the layer that actually kills these systems. That split — every rule, none of the reasons — is the most useful thing the experiment produced.

The question, stated precisely

"Can AI read my old code?" is the wrong question because "read" means two different things. One meaning is inventory: enumerate the conditions, thresholds and branches the code actually executes. The other is understanding: know why each one exists, which ones are load-bearing, and which ones are a 1999 typing mistake nobody has dared to touch since. The first is a parsing problem. The second is an archaeology problem, and archaeology needs the people who were there.

The experiment separates the two meanings and measures each one. If a local model can do the inventory honestly, it changes what a migration week looks like — the inventory step that used to be a senior engineer and a printout becomes a coffee break. If no model can touch the second meaning, that is equally useful: it tells you exactly which step still requires the interview, the diagram and the person.

The sample: 116 lines with a private answer key

The module is modOrders.bas: order totals, discounts, priority, shipping zones, a ticket cap, a credit limit, calendar rules for Sundays and the year-end shutdown. Ordinary Access/VBA of exactly the kind that has been quietly running a company's order desk since the Windows 2000 era. One rule, so you can see the flavor:

' modOrders.bas — fictional sample, loyalty discount
If Mid(custCode, 2, 2) < 98 Then
    orderTotal = orderTotal * 0.97
End If

What the code says, read literally: customer codes whose second and third characters are numerically below 98 get a 3% discount. What the answer key says: those two characters are the customer's sign-up year, 98 means 1998, and 1998 is the year the (fictional) company was acquired — so "loyal" really means "was here before the acquisition," a term with political weight inside that company. A reader can see the comparison. A reader cannot know it encodes a year, let alone an acquisition, let alone the office politics attached to it.

Twelve rules like that are in the module. I know all twelve because I wrote them. That is the answer key, and it is the only reason this experiment can be scored at all — on a real system, nobody has one.

The seven planted facts

Separately from the rules, I wrote down seven facts about the fictional system that are true, that a developer would know, and that no reading of the code can produce. These are the test items for the second meaning of "read":

1. MAX_TICKETS = 3 comes from the paper order form, which had three ticket slots. The software inherited a physical constraint nobody re-examined.
2. The "P" flag started as a data-entry mistake in 1999 — one customer was mistyped with a P. The mistake was kept because that customer's printed invoices referenced it. "P" does not mean "priority" in any documented sense.
3. The 4 January date for the year-end shutdown was negotiated with the freight carrier, not chosen by the software team.
4. The "T" flag means the customer is in precision machining — the 2% tolerance is a machining tolerance, not a financial one.
5. The "IN" prefix rule was added in 2003 when the company opened its own depot; before 2003, internal transfers were billed like normal orders.
6. The loyalty cutoff (1998) matches the acquisition year; "loyal" is a politically loaded term internally.
7. The credit limit of 200,000 was set by the owner's bank line, not by any analysis. It was never revisited.

Every one of these is the kind of fact I have personally found during real retirements — the discount field that was not a discount, the workaround that turned out to be load-bearing, the constant that came from a piece of paper. Each one explains a rule the code states but cannot justify. Miss them, and you "migrate" a rule that was already wrong, or preserve a constraint that died with the paper form in 1991.

How the experiment ran

All three models ran locally on my own machines, served over my LAN through an OpenAI-compatible API — the two-PC setup I documented in the build article. Local matters for more than convenience: the code in a real engagement is under NDA, and "paste it into a hosted chat" is a contractual question before it is a technical one. A model on your own network never has that problem.

Identical prompt for every model, temperature 0.2:

Extract every business rule in this VBA code. For each rule give:
(1) plain-language statement, (2) code location, (3) confidence,
(4) questions you would ask the original developer.
Do not invent anything not in the code.

Scoring rules, fixed before the runs: a rule counts as found only if it is stated correctly with the right function; line numbers were treated as approximate because models drift a few lines; an invented rule or a wrong statement counts as an error; a planted fact counts as recovered only if the model states the actual backstory — asking a good question about it does not count.

The results

The 27B failure is the most instructive row. Its 4,000-token budget went entirely into its own reasoning field — 16,062 characters of deliberation, zero characters of answer, finish reason "length," truncated mid-thought. The deliberation trace shows it had found the rules correctly: internal transfers, the three discounts, zone D, tax ordering, the priority levels, the Sunday and year-end shifts, the ticket cap, the tolerances, the credit limit. It knew. It never wrote anything down. A reasoning model that does not close its answer is useless in a pipeline even when its internal analysis is correct — and it failed the same task twice before I understood why.

The 12B model is the quiet winner. Its first run, at a 3,000-token budget, truncated mid-rule-10 — an output-budget failure, not a knowledge failure. Given 8,000 tokens, it produced a complete 15-item inventory in 72 seconds: all 12 rules, each attributed to the correct function, line numbers off by a few, no invented rules. On the loyalty rule it described the mechanism correctly and flagged the meaning as a question instead of guessing.

The 8B model shows the noise floor. Run 1 found all 12 real rules but padded the inventory to 22 items (tables and forms listed as "rules"), got line numbers wrong throughout (off by about 13), and stamped "Missing Info: None" on 17 items. Run 2 — same model, same temperature — produced only 10 rules, dropped the priority rules entirely, misattributed the tax rule, and misstated the internal-transfer rule as "exempt from tax" when the code zeroes the whole total. Two runs, two different inventories. The variance is itself a finding.

Finding 1 — the inventory step works, on the right-sized model with a real output budget

A 12B instruct model, quantized, on a home PC, extracted 12 of 12 rules from 116 lines of VBA in about a minute with zero hallucinations and honest uncertainty where uncertainty was due. Ten years ago that step was a senior engineer, a printed listing and a week of coffee-stained notes. It is now a coffee break — provided you give the model enough output budget to finish writing, which is a setting, not a mystery.

The practical lesson from the 27B row: manage the output budget before you upgrade the model. A model that thinks harder is not a model that delivers more, and in a pipeline the deliverable is the written answer, not the private analysis.

Finding 2 — the questions are the deliverable

The 12B model recovered none of the planted facts, and this is the part I did not expect to like: it asked almost exactly the right questions. Why does the cutoff say 98? Why is zone D exempt from shipping? Why January 4, and not January 1? Why does "T" get a stricter tolerance? Why hardcode "MGR" when a user table exists?

Read those questions against the planted-facts list: they point at the acquisition year, the depot, the freight carrier, the machining tolerance and the bank line. The model could not answer any of them — but it produced, unprompted, a near-complete interview agenda for the person who can.

That reframes what the extraction run is for. The inventory is a draft you verify against the code. The questions are the agenda you take to the operator — the same person my VB6 playbook tells you to sit with while they run a real month-end. The machine writes the interview questions; the human answers them. That division of labor is the actual output of this step.

Finding 3 — zero planted facts recovered, and that is not a bug

Three models, two sizes of run, seven planted facts: zero recoveries. Not "weak recoveries" — zero. No model stated that the ticket cap came from a paper form, that "P" began as a typo, that the shutdown date was negotiated with a carrier, or that the credit limit came from a bank line. The information is not in the code, and no amount of reading extracts information that is not there.

This matches the industry consensus on knowledge extraction, and it matches my own field experience: tacit knowledge comes out through interviews and diagrams, not through code reading. The experiment did not discover a limitation of these three models. It re-measured, on my own hardware with my own answer key, a boundary that has been known for decades — and it measured it honestly enough to show the one thing models can do at that boundary (ask the right questions, Finding 2).

For the three-layer model from the last-admin article: the code layer is now genuinely machine-readable, more readable than I expected. The procedure layer — the "why," the ordering, the never-do-X-on-the-1st — is exactly where the machine stops, and it stops in a way you can predict. Plan the migration around that line instead of around the hype.

Finding 4 — the confidence column is decoration

The prompt asked every model to label its confidence. The 8B model stamped "Missing Info: None" on rules whose business meaning it could not possibly know — including rules the answer key says are unexplainable from the code. A confidence label produced by a system that cannot know the meaning of what it is describing is not a confidence measure; it is formatting.

Treat every extraction output as a draft with two verification duties: check each stated rule against the code (the model's line numbers will drift; its function attribution is the reliable part), and treat every "no questions here" as a place to ask a question anyway. The models that behaved best were the ones that admitted what they did not know — the loyalty rule flagged as a question by the 12B model is the behavior you want; the 8B model's blanket certainty is the behavior you must not trust.

What this means for your migration

If you have one module you are afraid of losing, this experiment translates into a workflow you can run this week:

1. Pick one module — the one the operator touches every month and nobody else can explain.
2. Run the extraction against a local model (or a hosted one if your contract allows it) with the prompt above, and give the output budget room to finish — 8,000 tokens for a module this size.
3. Verify the inventory against the code. Function attribution is trustworthy; line numbers and confidence labels are not. Run it twice; where the two runs disagree, that is where the subtlety lives.
4. Print the questions. Collect every question the model asked — and every rule where it asked nothing — into one page. That page is the interview agenda.
5. Sit with the operator while they run a real cycle, and work through the agenda. Write down every answer. That document is the part of the system that was about to die with a person.

Steps 1–3 are the machine's job and take an afternoon. Step 5 is the human's job and takes an hour per module. The hour is the one that saves the system.

Risks and mitigations

Tools and the boring, correct shape

The boring shape: a model on a box you own, an OpenAI-compatible API on your LAN, a fixed prompt, a generous output budget, and a human with an agenda. My own setup is documented in the two-PC build article, including the measured speeds and the failure mode.

If you do not have a spare box — or you want to try the extraction step before committing hardware to it — a managed instance with GPU hosting runs the same experiment for the cost of a few hours, billed by the hour. I keep a managed-hosting link here (I earn a small commission if you sign up; my disclosure is here); it is an option I describe, not one I use daily, because my inference box already exists. Whatever box the model runs on, the confidentiality rule from the risk table still applies: the code you feed it must be code you are allowed to feed it.

FAQ

Is it safe to feed real client code to a model?

It is a contractual question before it is a technical one. My experiment ran entirely on my own hardware over my own LAN, so the sample never left my network — and the sample is fictional anyway. Before any real code goes into any hosted model, check what your contract and your client's rules allow. A local model removes the question entirely.

Do I need the biggest model I can buy?

No — on this task the 12B model beat the 27B model, because the 27B spent its entire output budget on private reasoning and delivered nothing. Model size is the wrong axis; output budget, a fixed prompt and a low temperature are the settings that decided these results.

Can this replace interviewing the person who runs the system?

No, and the experiment is the evidence: zero of seven planted facts were recovered by any model. What the models did well was ask — the questions they produced are the interview agenda. The interview is still where the load-bearing knowledge comes out.

Why did you test on fictional code?

Because scoring requires an answer key, and real systems do not have one. I wrote the module, so I knew the 12 intended rules and the 7 facts that exist only in the fictional developer's head. The trade-off is honesty about what transfers: the rule counts are measurements; the shape of the failure — inventory yes, backstory no — is what I would expect on real code, and it matches what I have seen on real retirements.

What should I start with?

A 10–15B instruct-class model, temperature low (0.2 in my runs), max_tokens generous (8,000 for a 116-line module), the prompt above, two runs, and the printed questions in front of the operator. That is the whole method.

Summary

• A 12B local model extracted 12 of 12 business rules from 116 lines of VBA in 72 seconds, with zero invented rules — the code layer of the "last admin" problem is genuinely machine-readable now.
• Zero of 7 planted facts were recovered by any model. The "why" layer is not machine-readable, and no prompt fixes that; it needs the operator and an agenda.
• The best model's questions were a near-complete interview agenda — use the model to write the questions, not to answer them.
• Bigger is not better: the 27B model thought correctly and delivered nothing because the output budget ran out. Set the budget before you buy the model.
• Confidence labels are decoration and runs vary — one extraction is a draft, two extractions plus a code check are a deliverable.

Do this today: take the one module nobody else can explain, run the extraction prompt against a local model, print the questions it asks, and put that page in front of the person who runs it every month. An afternoon plus one hour is the cheapest risk reduction your system will ever get.