This one covers October 9 through 18. Two tests, and both of them told me no.

The blind test I promised: the leaner prompt lost, 4 to 2

Last week I said I’d write this one up. The test ran October 6, a few days before this window, but I said I’d tell it, so here it is.

My blog-writing agent runs on a long set of instructions with examples of my voice written right into them. I had a lean rewrite built for the newest model. It followed the official guidance: cut the over-specified process, and move the voice examples out into a separate reference file the agent could open when it needed them.

On paper, the rewrite was the better prompt.

So I tested it blind. Both versions wrote drafts from the same 6 frozen briefs. I read them in pairs without knowing which version wrote which, and picked the one I liked better each time. Every measured stat stayed hidden until I’d picked, because some of them would have given the game away.

The old version won, 4 to 2.

It lost on voice. The agent never once opened the reference file with my voice examples, so the lean version wrote without them. It wasn’t cheaper either. It cost about 10% more to run, because it spent more steps checking its own work.

The automated checks couldn’t tell them apart. Quality warnings came in at 15 against 16. AI-tell hits, 1 against 0. Drafts inside the target length, 5 of 6 against 6 of 6. If I’d let the scores pick, it was a coin flip. My read called it.

The rewrite didn’t ship, and neither did the plan to rewrite two more agents the same way. The test bench stays put, so the next time a prompt has a real quality or cost problem, I can run it again.

Cut the procedure if you want, but keep the voice examples where the agent can’t skip them. An example sitting in a file it never opens is an example it doesn’t have.

I tried to keep lead data on my own machine. It wasn’t good enough.

Every new contact that comes into an account gets sorted into one of six buckets: real lead, spam, vendor, existing customer, internal, or nothing to go on. A cloud model does that sorting today.

I wanted it running on my own hardware, so lead data never leaves the machine. This was about privacy, not cost.

Before running anything, I wrote down what passing looked like:

• It handles at least 60% of leads on its own.
• When it’s confident, it agrees with the cloud model at least 95% of the time.
• It buries zero real leads.

Then I made the test hard to fool. 700 leads, sampled across every account. The examples the model learned from never came from the account it was being tested on. Before the run, I checked the scoring logic against 2,999 past verdicts to make sure it matched production exactly.

First attempt: a small local model with examples in the prompt. It handled 36% of leads, agreed 86.9% of the time when it was confident, and buried 1 real lead. Spam was its worst bucket, right 48.5% of the time. It also took 30 to 40 seconds a lead on a machine with no usable GPU.

Second attempt: local text embeddings plus a simple classifier. Better on its top guess, 73.8% against 66.7%, but it barely trusted itself. It handled 3.7% of leads on its own and still buried one.

Then I gave both of them the best case I could. I picked the ideal confidence cutoff after the fact, the one that made each look as good as possible at 60% coverage. The first agreed 75.7% of the time and buried 16 real leads. The second agreed 85.2% and buried 8.

Neither came close to 95%. So it’s dropped, and the cloud model stays.

I asked the obvious follow-up: would a faster machine fix it? No. A faster machine runs the same models faster. It doesn’t make them more accurate. The one lever left is a much bigger model on a GPU box, and what I’d save didn’t justify buying one.

Write the pass line down before you run the test. If I’d set the bar after seeing 86.9%, I’d have talked myself into shipping a model that buries real leads.