This is Part 4 of the Modern Software for Small Businesses and Nonprofits series, and Part 1 of a new sub-topic inside it: providing affordable SEO for small business, starting with local business-listing profiles, and starting with Jev doing the narrow work it’s actually good at.

The short version

SEO tools that can manage more than five or six sites at once can easily cost $1,000 a month. For an agency to make a living reselling that, a small business ends up paying several thousand dollars a month — money that is not affordable for most of them. So I’m building a small, human-assisted tool that creates one approved business record once and prepares it for ten listing sites, as an experiment. It is semi-automated on purpose, because the sites themselves don’t all allow the same thing, and I’d rather be honest about that than ship something that gets a client’s listing suspended.

The problem: SEO pricing locks small business out before it starts

Here’s the plain fact I keep running into: the popular SEO tools that can manage more than five or six client sites can easily cost $1,000 a month. An agency buying that tooling has to recover it across its client base, and the bill that lands on an individual small business is several thousand dollars a month. That’s not a business decision a solo tea shop or a two-truck pet-waste company gets to make — it’s simply out of reach.

Local listing profiles (Google, Bing, Yelp, Nextdoor, and the rest) are one of the cheapest, highest-leverage pieces of local SEO there is, and they’re exactly the kind of repetitive, multi-site data entry a small agency charges for. So the first experiment in this sub-series is narrow: can one approved record be reused across ten listing platforms without an agency retainer, and without quietly breaking any site’s rules to do it?

What Jev is, in plain terms

I don’t expect anyone to already know what Jev is, so here it is simply: Jev is an AI model I reach through TypeSafe’s official SDK (docs.typesafe.ai), and I use it only for narrow, bounded decisions — not for writing code and not for driving a browser.

Jev shows up in exactly one place in the running application: a loopback-only “runtime assist” endpoint. A human supplies compact, visible page state and a closed set of candidate answers; Jev picks among them; the endpoint validates the choice against the candidates it was given and always returns executionAllowed: false. It cannot navigate, fill a field, upload anything, submit anything, or expand what an adapter is allowed to do.

Why semi-automated, not fully automated

Before writing a single adapter, I decided to validate, not assume, what each site’s rules actually say — so here’s what I found, and where I still don’t know.

The common framing is this: some sites simply won’t let a bot click “submit,” and stricter sites won’t let a bot populate the form fields at all. That’s real, but it undersells how granular the actual permission question is. I ended up tracking each platform’s permission on several separate dials — business eligibility, free-profile availability, read/navigate, fill, upload, submit, account/verification requirements, and address visibility — and every one of them defaults to unknown means deny. A Jev answer cannot grant a capability; only documented, official evidence can.

What that produced, platform by platform:

• Foursquare documents that service API keys are explicitly intended for internal automation, scripts, and agents — the only platform of the ten with an affirmative automation grant. (authentication, place search, suggest place)
• Bing Places documents a free claim/add/update flow and lets an eligible service-area business hide its public address, but nothing in the official help establishes permission to automate its web interface. (Bing for Business help)
• Nextdoor documents a free Business Page for local, casual-service, and home-service providers, requiring an owner or authorized representative to claim it. (Create a Business Page)
• Data Axle documents a free route for one to ten listings with duplicate matching and verification. (Local Listings)
• Yelp documents that claiming a page is free but requires owner or representative verification. (Creating a Yelp Page)
• BBB separates a free Business Profile from paid accreditation — the pilot is never allowed to drift into the accreditation funnel. (Get Listed)
• Angi is the one I actually blocked: its signup funnel asks for a mobile number and explicitly authorizes marketing calls and texts, and I could not find a free, no-lead-funnel profile route. Until that changes, it’s blocked outright, not “manual.” (Angi pro join)
• Apple Business, Yellow Pages, and Facebook Pages all have a documented free or claimable path, but none of them establish enough about address visibility, eligibility, or current terms to automate anything beyond preparing values for a human.

So “semi-automated” isn’t a hedge — it’s the honest description of ten sites that each permit something different, with the default on every undocumented permission set to deny.

The architecture

The diagram above is the real shape of the system: one approved, source-attributed business record feeds four possible levels of automation, Jev sits in a narrow assist loop that any of those levels can call into but that can never act on its own, and nothing reaches a live platform without passing a validation gate and a second, adversarial review from a stronger model first.

1. API automated — Foursquare. The only platform with a documented automation grant. The adapter does an authenticated duplicate search, an explicit dry_run=true preview, and then one human-approved dry_run=false suggestion write — never a browser interaction with Foursquare’s business portal.
2. Prefill + automated submit. Architecturally supported by the adapter contract, but currently unused by every one of the ten platforms, because none of them has granted fill-and-submit permission. This is the level where “unknown means deny” actually bites: Jev being confident about a field does not open this path either.
3. Prefill + human submits — Bing. A fixture-tested draft review is prepared from the approved record, checked against known duplicates, and handed to a person to claim, review, and submit. No browser drives Bing’s site.
4. Assist the user — the other eight platforms. Values and instructions are prepared; the person does every step themselves, in their own browser, with their own login.

This is also the path with the most unused room for Jev — ideas I want to try, not finished work, and every one of them still ends with the person clicking submit themselves:

• Clipboard assist — the tool notices which field the person has focused (its label and the text around it), Jev picks the matching value from the approved record, and it lands on the clipboard for one paste.
• Label matching per site — Jev maps each platform’s own wording (“Contact number,” “Business phone”) back to the record’s fields, since no two sites use the same labels.
• Category picking from each site’s own taxonomy — the same narrow job Jev already does for Foursquare’s category list, applied to whichever list the next platform uses.
• Page recognition — Jev classifies which step the person is looking at (claim, verification, address options, done) so the tool can show the right instruction instead of a generic one.
• Duplicate judgment — before the person claims anything, Jev looks at a search result and judges whether it’s plausibly the same business.
• Address-visibility choice — Jev reads a site’s own address and service-area options and suggests the setting that matches how the record’s address is classified.

Jev’s runtime assist loop sits underneath all four paths, not inside any one of them: it takes compact page state and a closed candidate list, returns one validated choice, and never sees a screenshot, raw HTML, a credential, or a private email.

Status

The project is still in progress, and it’s a ten-site experiment, not a finished product: one platform (Foursquare) has a real, tested API adapter; one (Bing) has a fixture-tested human-submit workflow; eight are deliberately manual-guidance or, in Angi’s case, blocked. Live dry_run and live writes against the real Foursquare endpoint for the pilot business are still untested, pending the client’s approval of the location and category facts. If it keeps working, I’ll open source it.

Part 2 of this sub-series will pick up once there’s a live, human-approved publication to report on.

Previous: Part 3 — My Ten-Year-Old Built a Video Game From Four-Word Prompts · Series: Modern Software for Small Businesses and Nonprofits

Part 6 of The End of Software Scarcity. Previously: Software Is Getting Cheaper. But By How Much?

Two camps have formed around AI and software development, and I don’t trust either one.

The first camp says nothing has really changed. Senior engineers still gatekeep every merge. Five years from now looks like today, just with better autocomplete.

The second camp says code is already free. Software will be generated instantly, forever, for everyone, at zero marginal cost.

Neither extreme survives contact with evidence. Believing either one is exactly the mistake this series keeps warning against: picking a comfortable story and skipping the part where you check it.

So here are three specific, checkable hypotheses about what actually changes in the next five years — not “AI changes everything,” but three separate claims, each with its own evidence, and its own reason to doubt it.

The Short Version

Coding is getting cheaper, but the developer’s job doesn’t disappear. It moves upstream into defining the problem correctly and downstream into validating what an agent produced. Software’s price doesn’t fall in a straight line either — the same budget increasingly buys more capability instead of a lower invoice. And the whole shift is bottlenecked by something more mundane than any of that: getting an agent enough of the right context about one specific, messy, real business without it confidently building the wrong thing.

Three hypotheses. Here’s the evidence for each, and the counterevidence that should keep you honest.

Hypothesis 1: The Developer Becomes a Conductor

The clearest version of this future is already shipping. Conductor is a Mac app that runs “parallel Claude Code, Codex, and Cursor agents in isolated workspaces” — each agent gets its own copy of your codebase, works a separate task, and reports back to a single dashboard. You stop writing every line yourself and start deciding what gets built, in what order, by which agent, and whether the result is good enough to merge.

That’s the shape of the hypothesis: human value moves upstream to specifying the task well enough that an isolated agent can’t misunderstand it, and downstream to reviewing four diffs instead of writing one function. The scarce skill stops being “can you write a for-loop” and becomes “can you write a spec an agent can’t misread” and “can you tell good code from confident code.”

Here’s why I’m not popping champagne over it.

The reason is in the last article: the METR study found experienced developers got slower with AI on codebases they already knew, while believing the opposite the whole time.

If conducting several parallel agents is harder than writing code yourself — and reviewing four unfamiliar diffs at once is a plausible way for that to be true — the “conductor” role could easily produce the same illusion: busier, more confident, and not actually faster. Being the conductor raises the validation burden precisely where METR shows humans are worst at judging their own performance.

So Hypothesis 1 isn’t “developers become conductors and everything gets better.” It’s “developers become conductors, and whether that’s actually better depends entirely on whether validation keeps up” — which is an open question, not a settled one.

Five years is roughly the timeline I’d bet on for this role to become the default for professional developers, not because the tooling needs five years to get better — it’s already usable today — but because validation habits take longer to build than the tools that make them necessary. Learning to write code and learning to distrust code you didn’t write are different skills, and most developers currently have only practiced the first one.

Hypothesis 2: Software’s Purchasing Power Explodes

The second hypothesis is one I’ve already made the case for in this series, so I’ll be brief and point you to the receipts.

The naive version of “AI makes software cheaper” imagines a flat discount: the same product, 25% off the invoice. The evidence doesn’t support that shape. What it supports is purchasing power — for the same $1,000, a business gets custom workflows, integrations, and features that used to be completely out of budget. The invoice doesn’t necessarily shrink. What you can get for it grows.

I laid out the mechanism and the strongest supporting signal — a McKinsey build-vs-buy survey — in the previous article in this series. The short version of the thesis: for decades, the relationship was Business → adapts to → Software, because changing software was expensive and changing your own process was cheap. If the marginal cost of building and modifying software keeps falling, that reverses into Software → adapts to → Business. Custom stops being the expensive option by default.

I’m not restating the percentages here, because restating a number outside the analysis that produced it is how a caveated finding turns into a headline. Go read the receipts if you want them. The point for this article is narrower: hypothesis 2 is about what a dollar buys, never about the price printed on the invoice. Collapsing that distinction is the single error this whole series is built to catch.

Hypothesis 3: Context Is the New Bottleneck

Here’s the part that doesn’t get said enough.

Writing new code from a blank file is the easy case for a coding agent. Give it a well-specified feature in a clean repo, and current models are genuinely good at it. The hard case is everything else: a fifteen-year-old schema nobody fully documented, a security rule that exists because of an incident in 2019, an integration held together by a Slack thread and one engineer’s memory. An agent doesn’t know what it doesn’t know, and it will confidently generate plausible, syntactically correct code that quietly violates a rule it was never told about.

This isn’t a hypothetical concern. A recent study of AI coding agents on a real 108,000-line production codebase found exactly the failure mode you’d expect from an agent without durable memory of the project: it “loses coherence across sessions, forgets project conventions, and repeats known mistakes” — not because the model can’t code, but because nothing was carrying the accumulated context of the project between sessions. The fix the researchers built was infrastructure for persisting that context — specs, conventions, and domain knowledge encoded outside any one conversation — because without it, capability alone wasn’t enough.

A major vendor is already betting on this being the real problem. In December 2025, Google shipped Conductor, a Gemini CLI extension (open source, Apache 2.0, gemini-cli-extensions) that makes a human-reviewed spec and plan — not the prompt — the unit of work: context first, then a written spec, then a plan you approve, then implementation. It’s one vendor’s tool, not proof of an industry trend, but it’s a concrete, checkable data point that a major AI lab thinks context infrastructure, not raw code generation, is where the next increment of value sits.

METR’s slowdown wasn’t a syntax problem — those developers already knew their codebases; what cost them time was feeding the model enough of what they already knew, correctly. That’s the context bottleneck showing up in the one dataset in this article that actually measured real developers doing real work.

Context is the new code. Not “prompting is a skill” — actually building and maintaining the structural knowledge (schemas, constraints, history, conventions) that lets an agent act safely inside a specific business is turning into the harder engineering problem, right as writing the code itself gets easier.

This is also why “five years” is doing real work as a horizon in this article, rather than being a round number I picked for the title. Greenfield capability is improving on a timeline measured in months. Context infrastructure — the unglamorous work of encoding a specific organization’s schemas, history, and unwritten rules somewhere an agent can actually use them — is an organizational problem, not a model problem, and organizations change slowly. I’d expect the gap between “AI can write good code” and “AI can be trusted with your specific legacy system” to close in years, not months, and to close unevenly: greenfield teams first, complex regulated brownfield systems last.

Where the Three Meet

Put the three together and a pattern shows up that none of them states alone.

If developers become conductors (Hypothesis 1), and software’s purchasing power explodes because custom work gets cheap enough to be worth it (Hypothesis 2), the thing that makes both of those actually pay off is solving the context bottleneck (Hypothesis 3). An agent that can’t safely absorb a business’s real structure can’t be trusted to run unsupervised, which caps how much conducting-not-coding can scale, which caps how much purchasing power actually explodes. The three hypotheses aren’t independent bets. Two of them are conditional on the third.

And solving the context bottleneck has an uncomfortable requirement baked into it: an agent that’s genuinely useful at this needs deep, ongoing access to the private, messy, specific data that makes a business what it is. Not a sanitized API. Not a scrubbed demo dataset. The real schemas, the real customer records, the real internal Slack history that explains why the weird rule exists. Extreme software abundance and extreme data access turn out to be the same request.

That should make you pause. It made me pause. If the payoff of this whole shift depends on agents holding that much intimate context about how a business actually runs, a new question becomes unavoidable:

What happens when massive third parties or competitors try to buy, trade, or steal that intimate agent context? We will tackle the high-stakes world of Agentic Data Privacy in our next article.

Part 5 of The End of Software Scarcity.

One of the first things you learn when working with AI is simple: ask for evidence.

I’ve spent the last few months arguing that AI is making software cheaper.

Then I realized I should take my own advice.

Where is the evidence?

Simple Economics

Here’s the mechanism: when supply rises faster than demand, price falls.

Code supply has visibly risen since coding agents arrived. GitHub’s own Octoverse 2025 report puts a number on it: developers pushed nearly 1 billion commits in 2025, up 25.1% year-over-year — the platform’s biggest jump on record, including almost 100 million commits in August alone. Commits aren’t code exactly, but they’re the closest supply meter we have, and it’s spinning faster than ever.

So: does that mean software cost is about to come down?

Not evenly. Price isn’t set by supply alone — it’s set by what buyers can substitute for what they already pay. Routine, well-specified work is already getting repriced: demand for AI-substitutable freelance skills is down 20–50% relative to trend, exactly where a price collapse hits first (see References). Bespoke work that used to sit outside the budget becomes affordable — that’s the build-vs-buy shift below. And demand for software was never fixed: cheaper production unlocks buyers who never bought custom software at the old price, eating part of the supply increase. The invoice doesn’t fall uniformly. Software purchasing power beats a price index because it survives demand that moves too.

The Short Version

There is no single number for “how much cheaper software has become.” The honest evidence is messier and more interesting than that. Developer productivity, labor cost, production cost, market price, and what I’ll call software purchasing power can all move in different directions at once. Below, two strong supporting signals, one important counterexample, and the concept I think actually matters.

What the Evidence Actually Shows

Developers really can get more done

The strongest evidence I found is a Microsoft Research field experiment covering 4,867 professional developers. Randomized access to an AI coding assistant increased completed tasks by roughly 26%.

That’s real. But it measures productivity — output per developer — not what anyone paid for the finished software. A 26% productivity gain does not automatically mean a 26% cheaper product.

Businesses are starting to build instead of buy

The second signal is a build-vs-buy shift. In McKinsey’s State of AI survey, about 32% of respondents said their organization had decided against purchasing a software product or feature because agentic coding tools let them build it internally.

That’s a meaningful early signal, not proof. McKinsey didn’t verify that these companies rebuilt an identical commercial product for less. What they reported is a changed decision: build now looks viable where buy used to win by default. That’s still worth taking seriously, because it’s exactly the mechanism my broader thesis depends on.

The Counterevidence, Front and Center

Here’s where it gets uncomfortable for my own argument.

METR studied 16 experienced open-source developers completing 246 real tasks in mature codebases they already knew well. Using early-2025 AI tools, they took about 19% longer — even though they believed AI had made them faster.

This is not a small caveat. It suggests the Microsoft/Accenture gains may not generalize to experienced developers doing complex work in codebases they already understand. Greenfield work, junior developers, and well-bounded tasks may behave completely differently from brownfield maintenance on a system an expert already knows cold.

So: productivity gains from AI are real in some settings and absent, or negative, in others. Anyone telling you it’s universal is skipping the METR result.

Software Purchasing Power

Here’s the reframe I think is more useful than any single percentage.

Instead of asking “how much cheaper is software?”, ask: how much more software can I buy for $1,000 today than I could two years ago?

The evidence above doesn’t point to invoices falling by some clean number. It points somewhere else: for the same money, you can plausibly get more functionality. A GoodFirms survey of software companies found roughly 61% expect AI to cut development costs 10–25% — but it also surfaces something more interesting: some of that saving doesn’t show up as a lower invoice at all. It shows up as scope that used to be out of budget getting included instead.

Same money. More software. That’s technological deflation of the kind we’ve seen before in other industries, where capability rises faster than the sticker price falls.

Back to the Thesis

This connects to the claim I made when I started this series: that software scarcity is ending.

The old model was straightforward. Software was expensive to build, so businesses adapted themselves to it:

Business → adapts to → Software

If the marginal cost of building and modifying software keeps falling, that relationship can start to reverse:

Software → adapts to → Business

The McKinsey build-vs-buy number is an early signal of exactly this — companies choosing custom over generic because custom got cheap enough to be worth it. I’m not declaring SaaS dead. I’m saying the calculation is shifting, and I can point to a specific data point instead of just asserting it.

So, How Much Cheaper Has Software Become?

The honest answer is that we don’t have a single number.

In controlled settings, developer output rose by roughly 26%, and other field research has found gains as high as 55%. Software companies expect AI to cut development costs 10–25%. Some freelance categories are already under pricing and demand pressure. Some companies are now building what they used to buy. And experienced developers on complex, familiar codebases saw no gain — some got slower.

None of those numbers means “software is 26% cheaper” or “55% cheaper.” Forcing them into one figure would be the opposite of asking for evidence.

The more interesting possibility is that we’re measuring the wrong thing. The price of software may not collapse. The amount of software we can afford might explode instead.

Historically, software was expensive and ideas were cheap. If software production keeps becoming abundant, that relationship may flip: software becomes cheap, and good ideas become the expensive part.

That, I think, is what the end of software scarcity actually means.

Footnote: Five Things That Move Separately

Before any of the evidence above makes sense, keep five things separate: developer productivity (output per developer), labor cost (cost of engineering time), production cost (total cost of creating the software), market price (what customers pay), and software purchasing power (functionality per dollar). They don’t move together — an AI-skilled developer can charge more per hour while finishing in far fewer hours, so labor gets pricier per hour while the software gets cheaper to produce. Illustration, not data: before AI, 40 hours × $60/hour = $2,400; with AI, 10 hours × $100/hour = $1,000. Nobody measured this exact project, but it shows productivity, labor cost, and production cost moving in three directions in one paragraph.

References

Additional evidence reviewed for this article, not covered in the body above:

• BIS / Ant Group CodeFuse field experiment — a field experiment found generative AI increased code output by roughly 55%, concentrated among less-experienced developers.
• Ramp Economics Lab — businesses with high AI exposure saw roughly $0.03 of increased AI spending for every $1 of reduced freelance spending. This is spending substitution, not proof that AI output equals freelance output.
• Upwork Future Workforce Index 2026 — AI-using freelancers earn about 34% more per hour, while lower-complexity generative-AI contract work saw starts up ~90% and per-contract earnings down ~13%.
• GoodFirms Custom Software Development Cost Survey — survey of 100+ development companies on AI’s expected effect on cost and scope.
• Freelance-market research on generative AI — analysis of 3M+ freelance postings found demand for AI-substitutable skills fell 20–50% relative to trend, while complementary AI-skill demand rose.

Part 4, and the last part of Round 1. Part 1 prompted a small generalist and watched it miscalibrate. Part 2 trained a specialist that won in-domain. Part 3 adapted a generalist with LoRA and found the only arm that did not degrade out of domain. All three ran on one 16 GB laptop. This round we finally leave it - same problem, same labels, a real datacenter GPU - to answer the question the series has been circling since Part 1.

The Short Version

Nothing in this experiment has broken 0.5 QWK against independent human labels, and every learned or prompted arm has landed somewhere between 0.24 and 0.49. That is the whole shape of Round 1: a lexical floor at 0.222, a prompted 3B at 0.288, a fine-tuned cross-encoder at 0.360, a LoRA-adapted generative model at 0.353 - and the single best number anywhere, 0.486, out of domain. Four methods that share nothing all landed under the same 0.5 ceiling, spread across roughly a quarter of the scale rather than clustered together.

Which leaves one obvious suspect. Every one of those arms was laptop-bound by design. Maybe the ceiling was never the problem. Maybe it was the machine.

So this round changes exactly two things - the model and the hardware - and nothing else. A prompted Qwen2.5-7B-Instruct on a datacenter NVIDIA GPU, served by vLLM, grading the identical 3,000 held-out ESCI pairs and 3,000 held-out WANDS pairs, with the same system prompt (verified byte-for-byte by hash), the same parser, and the same QWK implementation Round 1 used.

The answer:

• ESCI QWK 0.361 (95% CI 0.333-0.389), WANDS QWK 0.354 (95% CI 0.326-0.382).
• Versus the 3B it scaled up from: +0.073 on ESCI, +0.110 on WANDS. Real, and not small.
• And landed inside the 0.24-0.49 band the learned laptop arms already occupied. It did not beat Part 3’s 0.486. It did not break anything.

The ceiling is the task, not the model size. Pointwise relevance grading against independent human labels is genuinely hard, and a 2.3x bigger prompted judge on datacenter hardware does not change that. That finding is what the whole round was for, and it is now a poster at the Science and Cyberinfrastructure for Discovery (SCD) 2026 at Georgia State University - more on that at the end.

Where we left off: a ceiling nobody could break

Round 1’s scorecard after three parts, all of it on one laptop:

Two honest readings were available at the end of Part 3, and they point in opposite directions:

1. The task is hard. Four very different methods - lexical, prompted, discriminative fine-tune, generative adapter - all landed under the same 0.5 ceiling, with the three learned or prompted methods filling the 0.24-0.49 range above the lexical floor. When methods that share nothing land under the same ceiling, the thing they share is the problem.
2. The machines are small. Every arm fit in 16 GB of unified memory because that was the constraint we set, not because it was the right size for the job. A 3B model is a small model. Nobody serious grades relevance with a 3B model if they have a choice.

You cannot settle that from inside the laptop. You have to go get a bigger machine.

The experiment: change two things, hold everything else

This is the part that makes the number mean anything. If you change the model and the prompt and the parser and the scoring, a QWK movement tells you nothing about which change caused it. So the round was built as an ablation against Part 1’s Arm B, with the changed surface reduced to model + hardware:

The model-delivery mechanism is matched too: Ollama applied llama3.2’s own instruct chat template for Arm B, and vLLM’s chat() applies Qwen’s. Same shape, different model.

The model, and an honest confound

The model I wanted was Llama-3.1-8B-Instruct - same family as Arm B’s Llama-3.2-3B, so 3B to 8B would isolate size and only size. It is gated on HuggingFace and no access token was available in the cluster environment, so it could not be pulled.

The model I used was Qwen/Qwen2.5-7B-Instruct - open, ungated, and a strong instruct model at roughly the same scale. That means this run changes model family as well as size. So the +0.073 / +0.110 lift over Arm B is “a bigger, different-family prompted model,” not a clean size ablation, and I am going to say that every time I quote the number. The clean version needs one more run with a token in the environment.

It is worth being clear about what the confound does and does not threaten. It muddies the attribution of the lift. It does not touch the headline, which is a ceiling claim: a strong 7B on datacenter hardware still landed at ~0.35.

Getting onto a datacenter GPU (the part nobody writes about)

The compute came from the National Research Platform’s Nautilus cluster - a shared academic Kubernetes GPU pool. Writing the judge took an afternoon. Getting a pod onto a card took three separate fixes, and I think they are more useful to read than the YAML:

1. Resource ratios. The cluster enforces that CPU, memory, and ephemeral storage limits sit within 1.2x of their requests. The fix is to set requests equal to limits and stop being clever: 3 CPU, 20 GiB memory, 40 GiB ephemeral.
2. The big cards are tainted. 48 GB cards carry a large-gpu hardware taint, which is a hardware label, not a preemption one - so tolerating it does not make the job preemptible. Without the toleration you simply never schedule.
3. The pool was too narrow. The manifest asked for an RTX A6000 by node affinity, and every A6000 in the cluster was allocated. But Qwen2.5-7B at fp16 is about 15 GB - it fits any card with 24 GB or more. Widening the affinity to the pool of non-quota-gated 24-48 GB products scheduled it immediately, because the 24 GB tier had dozens of free cards sitting idle while I queued for the 48 GB one.

That third one is the practical lesson of the round: ask for the GPU your model needs, not the biggest GPU you can name. The queue is at the top of the range.

The job itself is deliberately boring and safe for a shared cluster: a vllm-openai image with the entrypoint overridden to the judge script, code and data mounted from ConfigMaps (the repo was private, so no clone), restartPolicy: Never, backoffLimit: 0, and a 2-hour activeDeadlineSeconds so a hung pod can never idle on a card it is not using. Results come back on stdout between marker lines and get collected with kubectl logs.

The result: the ceiling did not move

Model load took 103.8 seconds. After that, greedy generation ran at 23.5 pairs/sec on ESCI and 32.8 pairs/sec on WANDS - 3,000 pairs in about two minutes per set.

Here is the full Round-1 scorecard with the datacenter arm in it:

Read down a column, never across - ESCI and WANDS use different label spaces.

The bigger prompted model on the bigger machine scored 0.361 and 0.354. Arm C, a fine-tuned cross-encoder that runs in about a gigabyte of RAM on a laptop, scored 0.360 on ESCI. Those two numbers are the same number. The 95% confidence intervals - [0.333, 0.389] and [0.326, 0.382] - sit right on top of the band the laptop fine-tunes were already in.

So the answer to the question that closed Part 3 is: the ceiling is the task. Scaling the prompted judge 2.3x and moving it onto a card with more memory than the entire laptop moved it from one part of the 0.24-0.49 band to another part of the same band. Pointwise relevance grading against independent human labels is hard in a way that more parameters do not fix.

Where the bigger model genuinely did help

I do not want the headline to flatten a real result. Against Arm B - identical method, identical prompt, identical parser, just 3B instead of 7B and a laptop instead of a cluster - QWK rose +0.073 on ESCI (0.288 to 0.361) and +0.110 on WANDS (0.244 to 0.354). Scale helps inside the band, and it helps by a lot.

The more interesting consequence is what that buys you with no training at all. The prompted 7B:

• Ties the fine-tuned cross-encoder on ESCI - 0.361 vs 0.360, in-domain, on the data the cross-encoder was fine-tuned on.
• Beats it out of domain on WANDS - 0.354 vs 0.299.

A model that never saw a single labeled example from this task matched a purpose-built fine-tune on its home turf and generalized better off it. If you have GPU access and no labeled data, that is a genuinely useful place to start, and it is the strongest practical argument in this round for reaching for the bigger machine.

Where it loses (because it does)

• It did not catch the LoRA out of domain. WANDS: 0.354 vs Part 3’s 0.486. This is the sharpest loss in the round and the clearest evidence the ceiling held - the best number in the whole experiment still belongs to a 3B model that was adapted on a laptop before dinner. A bigger prompted judge on better hardware did not get near it.
• It loses badly on exact accuracy to both fine-tunes. 0.343 / 0.332, against bge’s 0.517 / 0.473 and the LoRA’s 0.483 / 0.603. Its QWK is respectable only because its mistakes are ordinally close: off-by-one accuracy 0.79 on ESCI and 0.87 on WANDS, MAE 0.91 and 0.80. It agrees with the humans about rank far better than about label.
• You can see why in the prediction distribution: it hugs the middle. On WANDS it predicted grade 3 only 148 times out of 3,000, and grade 1 1,469 times. On ESCI it used grade 3 only 479 times out of 2,997. A judge that will not commit to “exactly relevant” gets the ordering roughly right and the label consistently wrong.
• Deployment cost is in a different universe. This needs a 24 GB-or-larger datacenter GPU and a 104-second model load. Throughput is excellent once it is warm - 23 to 33 pairs/sec batched - but Arm C does about 28 ms/pair in ~1 GB on a laptop. For a shipping on-device judge, the cross-encoder still wins the efficiency trade by a wide margin, and it is not close.
• It wrote three pieces of invalid JSON. Three parse failures on ESCI, zero on WANDS, all the same failure mode: the model put a literal unescaped " inside its reason string - an inch mark in 40", or quoting the query "cream and sugar" back at itself. A prompted judge that writes free-text reasons will occasionally produce unparseable output. The parser rejected all three rather than inventing a grade, which is the correct behavior and is exactly why they are reported here instead of silently becoming zeros.

Why I did not run a 14B

The plan going in was explicit: run ~8B first, go to 14B only if the ceiling visibly moved, and do not jump to 70B or A100-class hardware unless the trend was clear.

It did not move. The 7B landed on the same ~0.35 plateau as two 3B fine-tunes, and the 3B to 7B trend is already showing diminishing returns inside the band. A 14B prompted judge would very likely land in the same band at more cost, on a shared academic cluster where that GPU-hour is somebody else’s queue. So I did not spend it. The manifest is one --model change away if the datapoint is ever worth having.

Reporting a negative result and then not buying a bigger one to dress it up is, I think, the whole point of keeping an honest scorecard.

Honest caveats (they travel with every round)

• Compare arms within a dataset, never across. ESCI and WANDS use different label spaces - WANDS has no grade 1 at all - and are not on the same scale. Read down a column.
• Family confound, restated. Qwen2.5-7B vs Llama-3.2-3B mixes family with size. The lift is “bigger and different,” not “bigger.” The ceiling claim is unaffected.
• This is still an on-device contest at heart. The shipped production default is cloud-first (a hosted Gemini model), which was not benchmarked in this experiment and would very likely beat every arm here, including this one. What Round 1 asked was narrower: what is the best relevance judge you can get without an API call, and does leaving the laptop change the answer.
• Agreement with the production system’s own history is reproduction, not accuracy - the circularity trap from Part 1 holds for every round, which is why every number in this series is scored against independent labels.
• One run per arm, 3,000 pairs per set. The confidence intervals are bootstrap intervals over that sample, not variance across repeated runs. Greedy decoding makes the run reproducible; it does not make the sample bigger.

Where this went: a poster at SCD 2026

This finding - that the quality ceiling is a property of the task rather than of model size - was written up with Jiho Noh (Department of Computer Science, Kennesaw State University) as “The Quality Ceiling Is the Task, Not the Model Size: A Reproducible CPU/GPU Benchmark of Search-Relevance Judges,” and it was accepted as a poster at SCD 2026 at Georgia State University, with a 3-minute lightning talk in the Artificial Intelligence and Machine Learning session on September 22.

The poster is the compressed version of all four parts: one pipeline, one label mapping taken from the production system’s own source rather than retyped, one metric, and every arm scored identically across three hardware tiers. If you want the argument on a single page instead of across four posts, that is the page.

Next: Round 1 closes, the battle does not

Four parts in, Round 1 has an answer, and it is a negative one - which is usually the kind worth having. The ceiling on pointwise relevance grading is the task. It survived a lexical baseline, a prompted small model, a discriminative fine-tune, a generative adapter, and a 2.3x bigger prompted judge on a datacenter GPU. Five methods, three hardware tiers, one ceiling nobody broke.

What that leaves you with is not “which model is best” but which arm is best for which goal, and the series has a clear answer to that now:

• Shipping an on-device judge for one domain? The Part 2 cross-encoder. 28 ms, 1 GB, best in-domain accuracy of anything here.
• Grading queries from domains you did not train on? The Part 3 LoRA. Still holds the best number in the experiment, and it holds it on a laptop.
• No labeled data, but you have GPU access? This round’s prompted 7B. It ties a fine-tune in-domain with zero training.
• Need to actually break 0.5? Nothing here does it. That needs a different problem formulation - pairwise or listwise comparison instead of pointwise grading, or richer inputs than a query and a product title - not a bigger model.

Two things Round 1 earned but did not run, and I would rather list them than pretend they are done: the clean same-family size ablation (Llama-3.1-8B, once there is a token in the cluster environment), and the reverse generalization test Part 3 asked for - train on WANDS, test on ESCI - which is the real proof that adapting a generalist travels.

The battle itself is open-ended. Round 1 took search relevance. Round 2 takes a different problem, and the CPU gets to swing first again.

Same rules every round: independent labels, held-out numbers, and the losses reported next to the wins.

Code and reproduction details: github.com/Northvalley-Intelligence/cortex — journals 00-08 and results/*.json on main; this round’s journal 09, the Kubernetes manifest, and the raw gpu_prompt_qwen2.5-7b__round2.json on the round2-gpu-arm branch.

Part 3 of the series. Part 1 prompted a generalist and watched it miscalibrate. Part 2 trained a specialist and watched it win in-domain but travel less well. This round we try the third thing: keep the generalist, but stop prompting it - LoRA-tune it on the laptop GPU and read the grade straight off the logits.

The Short Version

Part 2’s cross-encoder was the first arm to beat the shipped on-device judge, but it was a specialist - trained on Amazon ESCI product search, and you could see it leaning on that domain when it met furniture (WANDS). This round asks the opposite question: instead of building a purpose-made classifier, can we take the generative model from Part 1 - Llama-3.2-3B - and adapt it into a grader with a small LoRA, still entirely on the laptop’s GPU?

Two changes make that work on a 16 GB machine:

• LoRA on a 4-bit base. We freeze a 4-bit quantized Llama-3.2-3B (~1.9 GB on disk) and train only small low-rank adapters on top - QLoRA-style, via Apple’s mlx_lm. Training peaked at about 5.5 GB of RAM, comfortable on a 16 GB laptop.
• Read the logits, don’t generate text. At inference we don’t ask the model to write JSON. The digits 0 1 2 3 are single tokens; we read the model’s logits for those four tokens at the one position where the grade goes and take the argmax. No free generation, no parsing, zero parse failures.

The result is the interesting one of the series so far. In-domain, the adapted generalist lands about even with the trained specialist (ESCI QWK 0.353 vs the cross-encoder’s 0.360). But out of domain it is the only arm that does not fall: on WANDS it scores QWK 0.486, while BM25, the prompted model, and the cross-encoder all drop from their ESCI numbers. That is a lead worth chasing, not a victory lap - and below I am careful about why.

(Numbers: results/mlx_lora__{esci,wands}_eval_sample.json; journal 07-arm-d-lora.md. n = 3,000 pairs per set, held-out, independent human labels.)

Where we left off: a specialist that showed its seams

Part 2’s bge-reranker-v2-m3 cross-encoder won the deployment tradeoff outright - more accurate than the prompted judge and ~200x faster - but it ended on a crack. Its edge on WANDS furniture (QWK 0.299) was real but thinner than on its home turf, ESCI (0.360). That is the classic shape of a model that fit its training domain well and travels less well. A cross-encoder with a bolted-on 4-class head has no choice: it only knows what it was trained on.

So the Part 3 move is to keep the broad prior knowledge a general chat model already has - it has “seen” furniture, electronics, and negation queries in pre-training - and just teach it this scale with a light touch, rather than building a classifier from the domain up.

The idea: adapt a generalist instead of training a specialist

New to LoRA? LoRA (Low-Rank Adaptation) is a way to teach a model a new skill without retraining all of its billions of weights. You freeze the original model untouched and slip small pairs of low-rank matrices - the adapters - into its layers; only those tiny matrices ever get trained. Here it’s the QLoRA flavor: the base Llama is 4-bit quantized and frozen (~1.9 GB), and the adapters train on top (peaking ~5.5 GB on a 16 GB Apple-Silicon laptop, 16 layers adapted). The payoff is that the trainable part is tiny and quick to train, and because the base model stays intact you can keep it and swap a different adapter in for a different task.

There are three genuinely different ways to turn a (query, product title) pair into a grade, and this series has now built all three. Side by side:

The same contrast in words:

• Approach 1 - prompt a generalist (Part 1). Take llama3.2:3b off the shelf, hand it a two-line rubric, and ask it to generate a grade as JSON. No training. It improvises the rubric on every call, and you have to parse free text back out.
• Approach 2 - train a specialist (Part 2). Take a cross-encoder, replace its head with four class logits, and fine-tune the whole thing on ESCI. It is discriminative - it does not generate anything, it scores. One forward pass, an argmax, no text.
• Approach 3 - adapt a generalist (this round). Take the generative Llama-3.2-3B, add LoRA adapters, and train only those on ESCI so the frozen base keeps its general knowledge. Then, at inference, skip generation entirely and read the label-token logits. A generative model turned into a judge by adaptation, not by prompting and not by rebuilding.

The point of Approach 3 is to get the reliability of a classifier (no parsing, no malformed output) without throwing away the broad prior that a general model carries into domains the training set never covered.

How you read a grade off the logits

This is the mechanic that makes the arm honest, so it is worth being concrete. In this tokenizer, "0", "1", "2", "3" each encode to a single token (ids 15, 16, 17, 18 - a leading space would split them into two tokens, so the prompt is built to avoid one). The chat template ends the prompt right where the assistant’s answer begins, with no trailing space, so the very next token position is unambiguously the grade digit.

That means we never let the model talk. We run one forward pass over the prompt, look at the logits for exactly those four token ids at that final position, and take the argmax. For a given pair the model emits four label-token logits and we take the largest - they might look like [26.77, 24.55, 25.44, 24.77] (illustrative, not measured values), where the argmax lands on grade 0.

Two consequences fall out for free:

• No parse failures, ever. The results file records parse_failure_count: 0 with the note “not applicable - argmax over the 4 label-digit token logits at one forward-pass position, no free-generation or text parsing.” Part 1’s prompted arm had to strip <think> tags and fences and occasionally threw a pair away; a logit read cannot malform its output, exactly like Part 2’s classifier.
• It is a fair comparison of method, not data. Arm D trains on the identical 36k-row balanced ESCI split the Part 2 cross-encoder used (a 4k dev split held aside), so Approach 2 vs Approach 3 is a clean comparison of cross-encoder classifier against generative LoRA logit-read, not of who got the better training rows.

Training it on the laptop GPU

The GPU does real work again this round. On the Apple-Metal GPU via mlx_lm, we LoRA-tune the 4-bit base for one full epoch (9,000 iterations over the 36k rows, batch size 4, --num-layers 16, learning rate 1e-5), computing the loss only on the completion digit (--mask-prompt), not on the instruction text. On a 16 GB machine the run peaked at about 5.5 GB of RAM and took a handful of hours - the kind of job you start before dinner and read the next morning, no datacenter, no API.

The 4-bit base is a deliberate choice, not a shortcut: an fp16 3B model is ~6-7 GB in weights alone, which leaves almost no headroom for activations and optimizer state on a 16 GB laptop. Freezing a 4-bit base and training fp32 LoRA adapters on top is mlx_lm’s well-supported path and the reason this fits at all.

As always in this series, every score below is measured against independent human-derived labels (Amazon ESCI and WANDS), never the service’s own past judgments - roughly 92% of those were written by the same local model, so agreeing with them would measure reproduction, not accuracy. That circularity trap from Part 1 still governs every number here.

Does it win? In-domain it ties; out of domain it holds

Here is the whole board, all four arms this series has built, on the same seeded held-out samples. Compare arms down a column (within one dataset) - the ESCI and WANDS columns are not on the same ruler, and I say why right after the table.

(Arm D: ESCI QWK 0.353, 95% CI 0.320-0.385, accuracy 0.483, off-by-one 0.722, mean nDCG 0.840; WANDS QWK 0.486, 95% CI 0.453-0.517, accuracy 0.603, off-by-one 0.816, mean nDCG 0.884 - all from results/mlx_lora__{esci,wands}_eval_sample.json, inference peak RSS ~2.67 GB, warm median ~0.87 s/pair on ESCI. Arms A-C from Parts 1-2 and results/bge_reranker__*.json.)

Read the last two columns together, because that is where the story lives:

• In-domain, the adapted generalist ties the specialist. On ESCI, Arm D’s 0.353 and Arm C’s 0.360 sit inside each other’s confidence intervals (D: 0.320-0.385). Building the grader by adapting a generalist got essentially the same in-domain accuracy as training a purpose-built classifier. That alone is a result: you did not have to give up the general model to get classifier-grade reliability.
• Out of domain, it is the only arm that goes up. Every other arm loses ground from ESCI to WANDS - BM25 -0.067, the prompted model -0.044, the specialist cross-encoder -0.061. Arm D moves the other way: +0.133. It is the single arm in the whole experiment whose ordinal agreement improves on a domain it never trained on.

Why “0.486 > 0.353” is a lead, not a trophy

I want to be exact about what that +0.133 does and does not prove, because it is the easiest number in this series to over-claim.

• You cannot compare the two datasets directly. WANDS uses a coarser 3-class label space (Exact / Partial / Irrelevant, no grade-1 bin), which is simply an easier target than ESCI’s four grades. So “0.486 on WANDS beats 0.353 on ESCI” is not a valid sentence - they are different rulers. The claim is not that Arm D is better on WANDS than on ESCI in absolute terms.
• What is comparable is the direction, across arms. Every arm was scored on the same two rulers. Three of the four degrade going out of domain; one does not. That cross-arm pattern - who holds up and who slides - is a fair comparison even when the absolute cross-dataset numbers are not. Arm D holding up (indeed improving) is the honest signal.
• One out-of-domain collection is not proof of generalization. WANDS is a single domain (furniture). A model that travels well to one new domain has earned a hypothesis, not a law. The real test is a reverse run - train on WANDS, test on ESCI - and see whether the adapted generalist still holds up when the home and away fields are swapped. That is deliberately left as future work; I would rather flag it than quietly bank the win.

So the fair summary is: adapting a generalist looks like it may travel better than training a specialist - a lead strong enough to chase, on the strength of being the only arm that did not degrade out of domain, and honest enough to admit it rests on one collection and a coarser label space.

Where it loses (because it does)

• It did not beat the specialist in-domain. On ESCI, Arm C is still nominally ahead (0.360 vs 0.353), even if the gap is inside the noise. If ESCI product search were the only domain you cared about, Part 2’s cross-encoder is still the arm to ship - and it is far cheaper to run.
• It is slower and heavier than the specialist. A generative 3B forward pass - even reading a single position - runs at roughly 0.4-0.9 s/pair and ~2.7 GB of RAM at inference, versus the cross-encoder’s ~28 ms and ~1 GB. That is much faster than Part 1’s prompted arm (seconds, and it actually generated text), but it is still one to two orders of magnitude off the specialist. Adaptation buys generalization; it does not buy the classifier’s speed.
• The absolute ceiling is still modest. QWK in the low-to-mid 0.3s (ESCI) is better, not good. Every arm in this experiment lives in the ~0.3-0.5 neighborhood; a lead over the baseline is not a solved problem, and I am not going to dress it up as one.

Honest caveats (they travel with every round)

• Compare arms within a dataset, never across. Restating it because it is the one that makes the headline honest: the ESCI and WANDS columns use different label spaces and are not on the same scale. Read down a column.
• This is an on-device contest, not “we beat the cloud.” The shipped service default is cloud-first (a hosted Gemini model), which was not benchmarked here and would very likely beat every laptop-bound arm. The win we chase is narrow and specific: the best relevance grader that runs on this machine with no API call - for privacy, cost, and offline reasons.
• The GPU is the Apple M-series integrated GPU (Metal), not a discrete card. No hand-written CUDA or Metal kernels - mlx_lm handles device dispatch. The whole thing, training and inference, fits on a 16 GB laptop, which is the entire point.
• Agreement with the service’s own history is reproduction, not accuracy - the circularity trap from Part 1 still holds, which is why every number here is against independent labels.

Next: leave the laptop

Three rounds in, the on-device picture is clear enough to name a winner per goal: if you only serve ESCI-shaped product search, the Part 2 cross-encoder is the arm - fastest, lightest, best in-domain. But if you care about grading queries from domains you did not train on, the adapted generalist is the one that did not fall over, and that is the more interesting lead.

The open question the whole series has been circling is still open: is the ~0.3-0.5 QWK ceiling the task - pointwise relevance grading is just hard - or is it the size of a model that fits on a laptop? Every arm so far has been laptop-bound by design. In Part 4 we finally leave the machine and hand the same problem, the same held-out labels, to a real datacenter GPU - to find out whether the ceiling moves when the hardware does. And somewhere in there is the reverse generalization test this round earned but did not run.

Same rules every round: independent labels, held-out numbers, and the losses reported next to the wins.

Code and reproduction details: github.com/Northvalley-Intelligence/cortex/tree/main/experiments/relevancy-replacement — journals 00-08 and results/*.json.

Part 2 of the series. In Part 1 we prompted a general-purpose model to grade search relevance and watched exactly where it fell over. This time we stop asking a generalist to guess and train a specialist instead - on the laptop’s own GPU. It becomes the first arm to actually beat the shipped on-device judge.

The Short Version

Part 1 left us with a prompted llama3.2:3b that grades e-commerce search relevance on the 0-3 scale, is fast enough, and is honest-to-goodness okay - but misses mostly by miscalibration, not by misunderstanding. The fix this round is not a bigger prompt. It’s a different tool: a cross-encoder, bge-reranker-v2-m3, fine-tuned on Amazon’s ESCI data with a small 4-class head bolted on. Training runs on the MacBook’s Apple-Metal GPU (PyTorch’s MPS backend) - the GPU finally does real work, on the same machine, no datacenter, no API.

The result: it’s the first arm to beat the shipped on-device judge on both independent test sets - ESCI QWK 0.360 vs the prompted model’s 0.288, WANDS 0.299 vs 0.244 - while running at ~28 ms/pair instead of seconds, in ~1 GB of RAM instead of 5-6. That’s roughly two orders of magnitude faster and ~5x lighter, for better accuracy.

And - because this series reports the losses too - it’s a specialist. It was trained on ESCI product search, and you can see it leaning on that: its edge shrinks on WANDS (furniture), a domain it never saw. Holding that thought is the whole setup for Part 3.

(Numbers: results/bge_reranker__{esci,wands}_eval_sample.json; journals 06-arm-c-bge.md, 08-final-comparison.md.)

Where we left off: a calibration problem, not a comprehension problem

The Part 1 diagnosis was specific. The prompted model mostly understood the query - it rarely called a perfect match irrelevant - it just put the boundary in the wrong place, sliding a “2 - Relevant” to a “1 - Marginal” and back. A calibration problem. When the failure is calibration, you don’t need a smarter conversationalist. You need something that has seen the labels and learned where the lines actually are.

That’s the pivot for this round: stop prompting a general model and start training one. Prompting asks a model that was built to chat to improvise a grading rubric on every call. Fine-tuning bakes the rubric in.

The idea: a cross-encoder, not a chatbot

A cross-encoder is the unglamorous, right tool for pointwise relevance. Instead of generating text, it takes the (query, product title) pair together, lets every query token attend to every title token, and emits a score. We take BAAI/bge-reranker-v2-m3 - a reranker already good at “does this passage answer this query” - and replace its scalar head with a 4-way classification head, one logit per grade: 0 Irrelevant, 1 Marginal, 2 Relevant, 3 Perfect. Inference is a single forward pass and an argmax over four numbers.

One quiet but real consequence: there is no text to parse, so there are zero parse failures - the field in the results file literally reads “not applicable - argmax over a 4-way classification head, no free-text parsing” (results/bge_reranker__esci_eval_sample.json). Part 1’s prompted arm had to strip <think> tags and fences and occasionally threw a pair away when the JSON came back malformed. A classifier can’t malform its output. That reliability is part of why it’s ~200x faster: no generation, no retries, no salvage.

Training it - this is where the GPU actually shows up

Part 1’s “GPU” was mostly Ollama quietly using Metal to run a 3B model. Here the GPU does the thing GPUs are for: training. On the laptop’s Apple-Metal GPU (CrossEncoder(model_dir, device="mps") - scripts/train_bge.py), we fine-tune on a balanced 36k-row ESCI train split with a 4k dev split, for 3 epochs. Best dev QWK peaked at 0.471 (06-arm-c-bge.md).

Everything is measured against independent human-derived labels - Amazon ESCI and WANDS - never against the service’s own past judgments, because ~92% of those were produced by the same local model, so “agreeing” with them would measure reproduction, not accuracy. (That’s the circularity trap from Part 1, and it still governs every number here.)

Does it actually win?

Yes - and this is the first time in the series I get to write that without a caveat swallowing the sentence. Held-out, on the same seeded test samples every other arm was scored on:

(ESCI QWK 0.360, 95% CI 0.325-0.391; WANDS 0.299, CI 0.262-0.334; warm median 27.9 ms/pair, p95 40 ms; peak RSS ~1.06 GB - all from results/bge_reranker__esci_eval_sample.json / …wands_eval_sample.json.)

Read it carefully, because the two axes both move the right way:

• Accuracy up on both sets. It beats the prompted judge on ESCI and on WANDS - and it never trained on WANDS. So it isn’t just memorizing ESCI; the calibration it learned carries to a domain it hasn’t seen. That’s the direct fix for Part 1’s failure mode.
• Speed and memory down by orders of magnitude. ~28 ms vs seconds is 1-2 orders of magnitude; ~1 GB vs 5-6 GB is ~5x lighter. For a service that grades thousands of pairs, that’s the difference between a batch job and a coffee break.

If you only wanted the deployment answer - more accurate AND faster than the on-device judge, on this machine - this arm is it.

Where it loses (because it does)

• The absolute ceiling is still modest. QWK 0.360 is better, not good. Pointwise relevance grading is genuinely hard - landing near ~0.3-0.5 is the neighborhood every arm in this experiment lives in. A win over the baseline is not a solved problem, and I’m not going to dress it up as one.
• There’s a real train/test gap. Dev QWK peaked at 0.471; held-out ESCI came in at 0.360 (06-arm-c-bge.md). The honest number is the held-out one. The 0.471 is what it looked like from the inside during training, and the gap is exactly the kind of thing that’s tempting to quote and wrong to.
• On the service’s own past grades it “disagrees” - and that’s correct. On the Cortex-validation set bge scores only 0.305. But that set is ~92% grades from the prompted model, so a genuinely independent judge should diverge there. Low agreement is the arm behaving honestly, not failing.
• It’s a specialist, and the seams show. Its edge on WANDS (0.299) is real but thinner than on its home turf, ESCI (0.360) - the classic shape of a model that fit its training domain well and travels less well. That crack is the entire premise of Part 3: a differently-trained arm that gives up some in- domain polish to generalize better. Watch this space.

Honest caveats (they travel with every round)

• Compare arms within a dataset, never across. WANDS uses a coarser 3-class label space (no grade 1), so a WANDS number and an ESCI number aren’t on the same ruler.
• This is an on-device contest. The shipped service default is cloud-first (a hosted Gemini model), which was not benchmarked here and would likely beat every laptop-bound arm. The claim is “no API, runs on this machine, and it beat the local judge” - not “beats the frontier cloud models.”
• Agreement with the service’s own history is reproduction, not accuracy - see the 0.305 above.
• The GPU here is the Apple M-series integrated GPU (Metal), not a discrete card. No hand-written CUDA or Metal kernels - PyTorch’s MPS backend handles device dispatch. The whole thing fits on a laptop, which is the point.

Next: read the logits

The cross-encoder wins the deployment tradeoff, but it’s a purpose-built classifier - and we just saw it lean on its training domain. What if we keep a generative model - the same Llama-3.2 family from Part 1 - but instead of prompting it, we fine-tune a LoRA adapter and read the label-token logits directly, no free generation at all? Does a generalist-that-was-taught generalize better out of domain than a specialist-that-was-trained? That’s Part 3 - still on the laptop’s GPU. Then, in Part 4, we finally leave the machine and hand the problem to a real datacenter card to ask a different question entirely: is the ~0.3-0.5 ceiling the task, or just the size of a laptop model?

Same rules every round: independent labels, held-out numbers, and the losses reported next to the wins.

Code and reproduction details: github.com/Northvalley-Intelligence/cortex/tree/main/experiments/relevancy-replacement — journals 00-08 and results/*.json.

The Short Version

We want to grade how relevant a product is to a search query, on a 0-3 scale, for an e-commerce search team. This is Round 1 of an open-ended series we call CPU vs GPU battle: take a real problem, solve it first the easy, CPU-friendly way, and keep an honest scorecard - a different problem each round, the same tension every time.

This round covers the two things you would reach for first, both of which run entirely on a laptop with no cloud API:

• A lexical baseline (BM25). Instant, tiny, and a genuinely useful floor.
• A prompted local LLM judge (llama3.2:3b via Ollama) - which is essentially what our relevance service already does on-device today.

Both are cheap and neither is embarrassing. The prompted model is the better grader of the two. But the interesting finding of this round is not a leaderboard - it is two traps that anyone doing this work will hit: the model is systematically miscalibrated (which makes raw agreement understate it), and the “ground truth” we were tempted to grade against was 92% produced by the same model we were testing - so agreeing with it measures reproduction, not accuracy.

Code and full method: github.com/Northvalley-Intelligence/cortex — experiments/relevancy-replacement (the method and the input/output contract are written out in full below, so this round stands on its own without the link).

The Problem: turning “is this relevant?” into a number

An e-commerce search team lives or dies by one question, asked millions of times a day: for this query, is this product any good? You need the answer as a number - three systems consume it: ranking (a score per candidate), evaluation (graded judgments for metrics like NDCG), and guardrails (a cheap flag for “this does not belong here”).

So we grade each (query, product) pair on a 4-point ordinal scale - the exact scale the relevance service uses, and the single source of truth every approach here is measured against:

The hard part is the 1-vs-2 gap: a phone case for the query “phone” is category-adjacent but doesn’t satisfy the intent - a 1, not a 0 or a 2. Most of the difficulty lives there.

The contract is deliberately minimal - query and title in, an integer out:

INPUT   { query, product_title }
OUTPUT  integer grade in {0, 1, 2, 3}

The judge prompt is two lines, so query plus title alone is enough to grade. (An unparseable model response is surfaced as a failure, never silently coerced to 0 - a detail that bites us later.)

It’s easy, and it’s not bad

Standing this up is genuinely a weekend’s work. You pull a small instruction-tuned model with Ollama, write the two-line prompt plus the rubric above, and you have a relevance grader running on your own machine with no API key and no per-call bill.

And on the easy pairs it behaves the way you would hope. These are the kinds of pairs it handles well (illustrative, generic - no client data):

Sourcing note: the query/title shapes above are drawn from public evaluation data and a live smoke-test of the service; the grades shown are illustrative of how the rubric is meant to read, not a per-pair transcript of one model’s output. The hard evidence for “not bad” is the aggregate score below, not any single row.

Across a held-out sample of 3,000 Amazon ESCI pairs, the prompted llama3.2:3b judge lands within one grade of the reference label 77% of the time (its mean absolute error is about 0.95 of a grade). For a 3B model running on a laptop, grading e-commerce relevance it was never trained on, that is a reasonable place to start. It is also, notably, better on ordinal agreement than the pure lexical baseline - more on that in the scorecard.

So far, so encouraging. Then you look at where it disagrees with the labels, and the story gets more interesting.

Then it misclassifies - and that’s the interesting part

Look at where the prompted model disagrees with the independent labels and a pattern emerges. Here are three real held-out cases - the query, the product, the human gold grade, and the grade the current approach gave (✗ marks a miss):

The current approach fails in two opposite directions:

• It overgrades on word overlap (row 1): a book whose title contains “cat” scores Perfect for a query about a cat. The model latched onto the shared word - off by three whole grades on something a human dismisses instantly.
• It undergrades real matches (rows 2-3): a protector that fits the iPhone 8 is called Marginal because the model over-weighted “not plus”; a valid cross-brand TV substitute is called Irrelevant. Both are genuine matches it was too harsh on - and negation queries (“without”, “not plus”) are a recurring weak spot.

These are not cherry-picked oddities - they are the everyday failure surface of the easy approach: word-overlap false positives and negation-driven false negatives, over and over. This round stays on the arms you would actually reach for first, and on being honest about where they break.

Effect 1: it’s mostly a calibration problem

Look at the two easy arms side by side on the held-out ESCI sample:

The prompted model gets fewer grades exactly right than a bag-of-words baseline, yet scores meaningfully higher on Quadratic Weighted Kappa - the metric that rewards being ordinally close and punishes being far off. That combination points at a systematic problem rather than random noise: most of its errors are near-misses that land next to the right grade (which is why QWK beats BM25 despite the lower exact accuracy), even though the word-overlap traps like the cat book show it can still miss badly. Near-miss, systematic error is a calibration problem - the kind of error that is fixable in principle, not a sign the model fails to understand. Low raw agreement does not mean a bad model; it often means an uncalibrated one.

(Why QWK and not accuracy as the headline? On an ordinal scale, calling a 3 a 2 is a small slip and calling a 3 a 0 is a disaster - plain accuracy scores them identically. QWK weights the miss by the square of the distance, which is what you actually care about for ranking and guardrails.)

Effect 2: the circularity trap - don’t grade a model against itself

The bigger, more dangerous trap is about what you grade against. We also had 4,576 of our own past relevance judgments sitting in a database, and it was tempting to treat those as “gold” and measure each approach against them.

Then we checked where those judgments came from: about 92% of them were produced by the same local llama3.2:3b model we were now testing. On that set the prompted model “agrees with the gold” enormously well - but that is just the model reproducing its own cached grades at temperature: 0. It is self-consistency, not accuracy. Grading a model against labels it largely authored tells you whether it is reproducible, not whether it is right.

The fix is to insist on independent labels. Every accuracy number here is measured against human-derived datasets the models never authored:

• Amazon ESCI - real query/product relevance labels (Exact / Substitute / Complement / Irrelevant, mapped to 3 / 2 / 1 / 0).
• WANDS - furniture search relevance, used held-out only as an out-of-domain generalization test (Exact / Partial / Irrelevant, mapped to 3 / 2 / 0).

Our own historical judgments are kept only to illustrate this trap - never as the accuracy verdict. If you take one thing from this round: the labels you are auditing are not automatically ground truth, and if your model helped write them, agreement is a mirror.

The scorecard for the two easy arms

Both arms run entirely on the laptop, no API. Here is the honest read on each.

Arm 1 - BM25 (the lexical floor)

Classic bag-of-words scoring: how much do the query’s terms overlap the title, weighted by term rarity. Thresholds were calibrated on ESCI-train only, then frozen.

Where it wins: nothing to train, nothing to load, effectively free, and a real floor to beat. Where it loses: lexical overlap is not intent. It does worst on WANDS furniture, where the right product often shares few words with the query. Keep it as the baseline; it is not a contender.

Arm 2 - prompted llama3.2:3b (what we run on-device today)

The two-line prompt plus the rubric, one model call per pair, on Ollama.

Where it wins: better ordinal agreement than BM25 on both datasets, no training required, and it produces a human-readable reason alongside each grade. Where it loses: it is the slowest on-device option (seconds per pair with the full rubric prompt), it is heavy (the model weights sit at 5-6 GB of RAM), it occasionally emits JSON we cannot parse (about 0.2% of pairs on ESCI - remember that parsing detail), and it is miscalibrated as shown above.

Honest caveats (they travel with every round)

Credibility is the point, so the fine print is not buried:

• Absolute scores are modest. Pointwise relevance grading is genuinely hard - even the better of the two easy arms lands well short of a perfect grader. Treat these as a floor to beat, not a solved problem.
• Compare arms within a dataset, never across. WANDS numbers look higher than ESCI for the same arm partly because WANDS is a coarser 3-class label space (no grade-1 bin), which is simply an easier target. A cross-dataset comparison would be meaningless.
• Don’t over-read the speed numbers. Latency was measured with different methods per arm (concurrency here, serial there), so treat speed as order-of-magnitude, not a stopwatch photo finish.
• This is an on-device contest, not “we beat the cloud.” A hosted cloud model (GPT- or Gemini-class) would very likely beat every laptop-bound approach here on accuracy - we did not benchmark one. The win we are chasing is a specific, constrained one: the best relevance grader that runs on this machine with no API call - for privacy, cost, and offline reasons. Not a claim about beating the cloud.

Next: bring in the GPU

The two easy arms told us something precise: the prompted model’s problem is calibration, not comprehension, and the honest way to measure any fix is against independent labels, not our own history. Both of those point at the same move - stop prompting a general model and start training one to output this exact scale.

That is where the GPU in the laptop earns its keep. In part 2 we fine-tune a cross-encoder reranker to emit the 0-3 grade directly - and find out whether learning the labels fixes the calibration gap (and what it does to that seconds-per- pair latency). In part 3 we take the generative model itself, LoRA-tune it, and read the label-token logits instead of free-generating JSON - and see whether a tuned generalist beats a specialist out of domain.

The open question I most want answered: can a GPU-trained model fix both failure modes at once - the word-overlap overgrades like the cat book and the harsh undergrades on negation queries - without trading one for the other, and at what cost in speed and memory? How many of the misclassifications from this round survive contact with a GPU? We will put the numbers on the board next week.

Code and reproduction details: github.com/Northvalley-Intelligence/cortex/tree/main/experiments/relevancy-replacement — journals 00–08 and results/*.json.

Part 7 ended on a hard verdict: the classical retail-search toolkit was measured end to end, and almost none of it earned its place. But almost none is not none. One component shipped in Phase 3, and it is the most interesting result of the phase — because it did not come from a bigger model, a training pipeline, or more data. It came from reading the failures.

The phase opened with a funny one. A search for chikko — a misspelling — was returning coffee, and the hybrid seemed to quietly fix it. Encouraging. But “seemed to” is not a number, so the agent stopped trusting the anecdote and measured a thousand queries. The fuller truth was less flattering, and much more useful.

The Short Version

The agent built a failure taxonomy: it took 1,000 queries that each have a known-correct Exact product, and asked a precise question — when the right answer exists, how often does the system fail to rank it first, and why?

That taxonomy exposed a residual the million-vector embedding model could not touch: negation. Queries like refrigerator without freezer failed just as often with dense retrieval as without it — because an embedding model does not encode the word without. The fix was not a bigger model. It was a single linguistic rule, and it shipped: +4.8 nDCG on negation queries, no regression, live.

And the counterpoint that proves the discipline: a second, similar fix for attribute constraints was measured and deliberately not shipped, because the taxonomy predicted — correctly — that it had almost no headroom left.

Reading a Thousand Failures

Most retrieval evaluation reports one aggregate number. A failure taxonomy asks a sharper question, and it needs provable failures to be honest — cases where you can prove the system got it wrong because a correct answer demonstrably existed and was not ranked first.

Across the 1,000 queries that have an Exact match,¹ plain BM25 fails to rank that Exact first 37.2% of the time (372 queries). The hybrid brings that to 30.4% (304). But the interesting part is what kind of failure each one is:

Read that last row twice. Every other category improved when dense retrieval was added — brand mismatch was nearly eliminated, substitutes were cut. Negation did not move at all. Forty failures before, forty after. And underneath the flat total, it was not stable — the hybrid fixed 5 negation queries and broke 5 others. That is not improvement. That is a coin toss. The single most powerful technique in the entire project — dense embeddings, the thing that broke every ceiling from Cranfield to a million products — is blind to negation.

Why the Obvious Fix Is Wrong Twice

The obvious reaction to refrigerator without freezer is a must_not filter: exclude any product whose text contains “freezer.” It is wrong, and it is wrong in two different directions.

Trap one: the word is part of a name. “Without” often appears in product titles that have nothing to do with a constraint — Cemetery Without Crosses (a film), a book, a band’s album. A must_not on the excluded term throws these out for containing their own title.

Trap two: the correct product advertises the absence. The single best answer to “refrigerator without freezer” is often a product whose description literally says “no freezer compartment” or “freezer-free design.” A filter that removes anything mentioning “freezer” deletes exactly the product the shopper wanted, because it named the very thing it lacks.

So a naive filter fails the movie and the freezer-free fridge — the two populations negation splits into. An embedding model, meanwhile, sees “refrigerator” and “freezer” as highly related words and cheerfully returns fridges with freezers. Neither scale nor a filter solves this. It needs to be read.

The Rule the Agent Shipped

The fix is one observation stated as a rule: demote a product only when it makes a positive mention of the excluded term — not a mention of its absence.

A fridge whose description says “spacious freezer” is a positive mention of “freezer” → demote it. A fridge that says “no freezer” or “freezer-free” is a mention of the term’s absence → protect it. A film called Cemetery Without Crosses never positively asserts the excluded attribute at all → untouched. One rule, both traps closed.

On the 61 real negation queries in the set, it moved nDCG² from 0.7784 to 0.8267 — up 4.8 points — and fixed 10 of the 28 top-1 failures, with no regression elsewhere. It costs nothing at query time and needs no model. It shipped to production, and it is the “hybrid + negation” row that sat at 0.8528 in Part 7’s scoreboard — the lean stack the full training cascade could only beat by a tenth of a percent.

You can watch it work: refrigerator without freezer — freezer-equipped units drop, and the “without freezer” products hold their place.

The Fix the Agent Refused to Ship

Here is the part that makes it a discipline and not a lucky guess. Negation has a sibling: attribute constraints, like a color or a size that must match. The agent built the same kind of reranker for attribute conflicts and measured it exactly the same way.

On its own 44-query subset it looked fine — nDCG 0.8679 to 0.8855. But across the full thousand queries the effect was +0.0008, and it fixed exactly one of eleven top-1 failures. The taxonomy had already predicted this: attribute failures were a small, mostly-handled category. So it was not shipped. The measurement existed only to answer “is this worth building?” — and the honest answer was no.

That is the whole method in one contrast. Two nearly identical fixes, built and measured the same way. One found a residual nothing else could touch and shipped. The other confirmed there was no residual left and was set aside. The taxonomy is what told them apart before either became a maintenance burden.

The Point

Phase 3’s biggest, most portable win was a million-vector embedding model. Its last, shipped win was a single sentence about the word without — something that model, for all its scale, does not encode. The residual that survived more data and more model fell to reading the data.

That is where structural query understanding earns its keep: not as a layer you add because the textbook says so (the textbook cost more than it returned), but at the exact point where a bigger model stops helping and a human observation starts. You find that point by measuring failures, not by measuring averages.

Closing Phase 3

Three articles, one arc. Part 6: the academic stack transferred to a million real products at production latency — parity, on a free-tier box. Part 7: the full classical toolkit was the ceiling, and not worth its cost. Part 8: the one thing worth shipping came from understanding a failure, not adding scale.

What Phase 3 ships is exactly that lean: BM25 + a BGE hybrid + one negation rule, all inside 700 ms on one small box. Everything else was measured, mapped, and left on the shelf with its evidence attached.

Phase 4 is where the last thing academic data cannot provide arrives: customers. Clicks, add-to-cart, purchases — the only signal that separates a search engine that is technically correct from one that actually sells. That is where the learning-to-rank infrastructure kept in the portfolio, dormant since Phase 2, may finally get the signal it was built for.

Takeaways

1. Measure failures, not just averages. The aggregate nDCG hid the fact that dense retrieval does nothing for negation. A per-category failure taxonomy is what surfaced the one residual worth fixing.
2. Some meaning is not in the vectors. A retrieval-tuned embedding model that beat every benchmark still cannot read the word “without.” Scale did not close that gap; a rule did.
3. The obvious fix can be wrong in two directions at once. A must_not filter on negation kills both product names and products that advertise the absence. The correct rule demotes only a positive mention of the excluded term.
4. Knowing what not to build is the return on measuring. The attribute reranker worked on its own subset and was still correctly refused, because the taxonomy showed it had no headroom. That refusal is the discipline, not a failure of it.

Phase 1 proved the method. Phase 2 found out what it learned. Phase 3 took it to real products — and the last word belonged not to the biggest model in the stack, but to a careful reading of where it broke.

References

Reproduction figures are measured on this project’s own free-tier stack, over the 1,000-query ESCI evaluation subsample. Phase 3 experiment artifacts and the architecture ledger are public in the project repository.

Part 6 established that the lean academic stack — BM25 blended with BGE embeddings — reaches parity with the published baseline on 1.2 million real products, inside the phase’s two rules: one free-tier box, 700 ms per query. That is the floor the rest of Phase 3 has to beat.

And retail hands the agent something no dataset before it did: real graded relevance labels, assigned by humans, at scale.¹ That matters for a specific reason. Through two phases, a whole toolkit of classical retail-search techniques sat unused — field weighting, learning-to-rank, query understanding — not because they were rejected, but because there were no real labels to train or tune them against. ESCI removes that excuse. This is their first fair audition. The question is simple: does any of the textbook earn its keep?

The Short Version

The honest answer is measure everything, ship almost nothing.

• Field weighting — the single most canonical retail-search lever — is dormant: tuned by cross-validation it scores 0.8237, below the 0.8252 default. The textbook’s first move makes things slightly worse.
• Learning-to-rank reactivates for the first time in the entire project: with real labels it finally beats the fixed hybrid, 0.849 to 0.8472. Then you read the weights it learned and find it rebuilt the hybrid almost exactly. A +0.2% win that is inside the noise, for a technique that rediscovered what already ships.
• The full classical cascade — query-understanding signals fed as features into a retrained ranker — is the single best configuration measured, 0.8539. The textbook, run end to end, genuinely wins the benchmark.
• And it beats the lean stack already in production by +0.13%. A complete training pipeline, to gain eleven ten-thousandths of an nDCG point.

The cascade is the ceiling. The lean stack is already touching it. You do not ship a training loop to imitate a result you have for free.

Field Weighting: The Canonical Lever That Does Nothing

If you asked any search engineer for the first retail-search improvement to try, most would say field weighting — boost the title, weight the brand, tune how much the description counts. It is the textbook’s opening move. On ESCI, cross-validated over the evaluation queries, it scored 0.8237 against the 0.8252 default — very slightly worse.

The reason is mechanical and worth knowing, because it is a trap. The default is OpenSearch best_fields, which scores a document by its single strongest-matching field and adds only a fraction of the rest (a tie-breaker). Learned field weighting, as tested here, pushed toward an additive sum across fields — and summing lets several weak, partial field matches outvote one strong, coherent one. For products, where the title is usually the one field that matters, “best field plus a tie-breaker” is simply a better model of relevance than “add everything up.” The canonical lever is not just unhelpful here; it points the wrong way.

Learning-to-Rank Wakes Up — and Relearns the Hybrid

Now the interesting result. In Phase 2, learning-to-rank was classified portfolio, not core: it beat plain BM25 everywhere but only tied the hybrid, so by the project’s Occam rule it did not ship. The standing hypothesis was that LTR was dormant for lack of real labels, not for lack of value — that given genuine graded relevance it would come alive.

It did. Trained with query-grouped cross-validation on ESCI’s real labels, LTR scored 0.849, beating the fixed hybrid’s 0.8472 — the first time in the entire project that learning-to-rank beat the shipped configuration. The hypothesis was right.

Then you look at the weights it learned:

Handed every structured field and a free hand to weight them, the model leaned almost entirely on the two signals that already constitute the hybrid — BGE and BM25 — and set the field weights to near-zero, the title literally to zero. It did not discover a better ranking function. It rediscovered the hybrid, and its +0.2% edge is comfortably inside cross-validation noise. A technique that reactivates only to rebuild the thing it was meant to beat has proved the hypothesis and failed the audition in the same breath. Unpromoted.

The Classical Cascade: The Textbook, Validated

There is one more configuration, and it is the one the textbook actually prescribes: query understanding → retrieval → learning-to-rank. Don’t just hand LTR the retrieval scores — hand it query-understanding signals too, as features, and retrain. Specifically, two signals the agent built for the failure analysis in the next part: whether a product violates a negation in the query, and whether it conflicts on a stated attribute.

This is the best number in all of Phase 3: 0.8539. And unlike field weighting, LTR genuinely values the new features — the learned weights put negation-violation at 2.8 and attribute-conflict at 3.4, right alongside BGE at 3.4. The cascade is not noise. The textbook, run end to end on real labels, is validated: it is the ceiling.

…and Loses the Trade

Here is the scoreboard, every configuration cross-validated on the same evaluation queries:

Read the bottom two rows together, because they are the whole point. The full classical cascade — a feature pipeline, a trained model, and the retraining loop to keep it fresh — reaches 0.8539. The lean production stack — the same hybrid, plus a single hand-built negation fix that is the subject of the next part — sits at 0.8528. The cascade’s entire advantage over the thing already running in production is +0.0011 nDCG, or 0.13%.

That is the trade. To bank a tenth of a percent, you take on a training pipeline: labeled data to maintain, a model that drifts, a retraining schedule, and a feature store — all of it forever, on a project whose first rule is one free-tier box. The textbook wins the benchmark. It loses the trade badly. The lean stack was already at the ceiling for free.

Measure Everything, Ship Almost Nothing

This is the discipline that has governed every promotion decision in the project, stated plainly here because retail is where it bites hardest. The value of an audition is not what you ship out of it. Most of the time you ship nothing. The value is knowing, with evidence, where the ceiling is — and whether you are already at it.

Phase 2 demoted LTR because it only tied the hybrid. Phase 3 gave LTR real labels, watched it wake up exactly as predicted, and demoted it again — this time for winning by too little to be worth its cost. Same rule, applied twice, to a technique that got stronger in between. That consistency is the point. A method that says yes to everything that improves a number would have shipped the training pipeline for its 0.13% and called it progress. This one measured the whole textbook, learned exactly where the ceiling sits, confirmed the lean stack is already there, and shipped none of it.

Who Did What

As in every phase: the operator kicked off the audition and approved (or declined) each promotion; the agent built the field-weighting experiment, trained the rankers, ran the cross-validation, and read the learned weights back. The human set the Occam rule and made the ship/no-ship calls. The agent produced the evidence those calls were made on.

What’s Next

Almost nothing shipped from the textbook. But one thing did ship in Phase 3 — the negation fix sitting at 0.8528 in that scoreboard — and it did not come from a bigger model or a training loop. It came from reading a thousand failures and noticing a pattern about language that no embedding encodes. That is the next part, and it is the one where the agent actually wins.

Takeaways

1. Real labels were the missing ingredient, exactly as predicted — and it still wasn’t enough. LTR reactivated on ESCI’s graded labels, vindicating a two-phase-old hypothesis, then relearned the hybrid and won by noise.
2. The canonical lever can point the wrong way. Field weighting, the textbook’s first move, scored below the default — because additive field-sum is a worse model of product relevance than best-field-plus-tie-breaker.
3. The textbook is the ceiling, not the product. The full cascade is genuinely the best configuration measured. It beats the shipped stack by 0.13%, which is not worth a training pipeline on a free-tier box.
4. Measure everything, ship almost nothing. The point of the audition was to learn where the ceiling is and confirm the lean stack is already at it. Knowing what not to build is the return on measuring it.

Phase 1 proved the method. Phase 2 found out what it learned. Phase 3 — retail — is where the toolkit finally got its fair hearing, and mostly earned a polite no.

References

Reproduction figures are measured on this project’s own free-tier stack, cross-validated on the ESCI evaluation queries. Phase 3 experiment artifacts and the architecture ledger are public in the project repository.

For five parts, a series called Self-Learning Agent-Based Retail Search has contained no retail. Phase 1 proved the method on 1,400 aeronautics abstracts. Phase 2 took the winning techniques across fifteen academic benchmarks to find out which ones travel. Both were rehearsals. Neither had a product in it.

Phase 3 does. The agent is now searching 1.2 million real Amazon products from the ESCI Shopping Queries Dataset¹ — real titles, brands, colors, attributes, and real human relevance judgments. This is the test the whole project was built to reach, and it opens with the only question that matters at the door: does the academic stack survive contact with real retail?

The Short Version

It survives. On 1.2 million ESCI products, the BM25 + BGE hybrid the agent carried out of Phase 2 lands at nDCG@10 0.847, within 0.4% of the published single-model baseline of 0.850 — essentially parity — at production latency, with no cross-encoder running at query time.²

Two things made that possible and honest. First, int8 quantization is lossless here (0.8473 vs 0.8472 at full precision), which shrinks the vectors 4× and lets a live million-vector index fit on the free-tier box — 4 GB of RAM, 20 GB of storage. Second, the claim is deliberately modest: parity with the single-model baseline, not the KDD Cup 2022 leaderboard winner at 0.904 — that number is a query-time ensemble of large cross-encoders, a different latency class entirely, and never the target under a 700 ms budget.

You can search the corpus yourself: retail-search.feroshjacob.workers.dev/phases/esci/search.

Two Rules, Set at the Start

Two constraints were fixed before Phase 3 began, and they are what make the word parity mean anything here.

One box. Everything runs on a single free-tier OpenSearch instance — 4 GB of RAM, 20 GB of storage, one node. No cluster, no GPU, no scaling out. If a technique cannot run on that box, it does not ship.

700 milliseconds. Every query has a production latency budget of under 700 ms. A method that is more accurate but slower than that is not more accurate in any sense a shopper experiences — it is a different product.

These two rules are the reason the numbers below are worth reading. Anyone can buy accuracy with a bigger machine and a slower query. The question this phase asks is what the agent can do when it is not allowed to.

Why a Product Is Not a Document

Every dataset before this one, academic or not, was flat text: a query, a passage, a relevance judgment that was essentially yes-or-no. A product breaks all three assumptions at once, which is exactly why Part 1 argued retail search is a harder problem than it looks.

A product is structured — title, brand, color, bullet points, attributes — not a bag of words. And relevance stops being binary. ESCI grades every query-product pair as Exact, Substitute, Complement, or Irrelevant, with graded gains (Exact = 1.0, Substitute = 0.1, Complement = 0.01, Irrelevant = 0). A substitute — a different product that would still satisfy the shopper — carries real credit. Sometimes the substitute is the best answer available, and a search engine that only chases exact matches leaves the customer with nothing. Ranking quality here is genuinely multi-objective, and for the first time in the project it is measured against labels a human actually assigned.

Two Modes, Never Averaged

Before any number means anything, one discipline has to be stated, because getting it wrong is the easiest way to publish a lie by accident. ESCI can be evaluated two completely different ways, and they must never be mixed.

• Official rerank mode. The dataset hands you up to 40 candidate products per query; you re-rank them. This is directly comparable to published numbers, because everyone is ranking the same list. It is the mode where “parity with the baseline” is a meaningful claim.
• End-to-end retrieval mode. You index the full 1.2M-product corpus and retrieve from scratch — the real problem a shopper poses. But ESCI’s judgments are incomplete: only a fraction of what you retrieve was ever labeled. At full scale, judged coverage is about 27%, so the absolute nDCG (~0.26) is floored by missing labels, not by the system. That number is real, but it can only be read alongside its coverage, and it can never be compared to a published figure.

Averaging these two would produce a meaningless middle number that flatters nothing and describes nothing. Every result below states which mode it is.

The Baselines Reproduce

The BM25 baseline in official rerank mode scores 0.8254 on a 1,000-query subsample and 0.8268 across all 8,956 English test queries. The published single-model baseline for this task is 0.8503, so plain keyword matching over structured product fields already gets most of the way there. Good — the harness is sane, and the headroom the hybrid has to prove is small and real, not an artifact of a broken baseline.

The Hybrid, at 1.2 Million Products

The agent embedded all 1.2 million products with the same retrieval-tuned BGE model³ that won Phase 2, and blended dense vector scores with BM25. Here is the transfer test, in official rerank mode across all 8,956 queries:

The hybrid reaches parity with the published single-model baseline — and it does it at roughly 0.45 seconds with no cross-encoder at query time, where the baseline it matches runs a transformer over every candidate. The same architecture the agent discovered on aeronautics abstracts and validated across fifteen academic domains transfers, intact, to a million real products. That is the whole point of the two phases of rehearsal: the win was never Cranfield-shaped or BEIR-shaped. It was real.

What it does not do is reach the 0.904 KDD winner, and it is worth being precise about why that is not a failure — it is the second rule doing its job. That number is an ensemble of large cross-encoders scoring candidates at query time. Our budget is under 700 ms per query; running a stack of cross-encoders over every candidate blows past that by orders of magnitude, and it will not run on a 4 GB free-tier box at all. It is a different latency class entirely, and never the target. Chasing it would mean optimizing for a leaderboard at a latency no shopper would tolerate. Parity with the single-model baseline, inside the 700 ms budget, is the honest claim.

Making a Million Vectors Fit

This is where the first rule bites. There was a hardware wall between “the hybrid works” and “the hybrid runs”: 1.2 million full-precision vectors are 3.74 GB, and Phase 2 had already established the ceiling — on this same 4 GB box, no corpus above roughly half a million documents could hold a live nearest-neighbor index. So the agent tested whether the vectors could be compressed without losing the win.

int8 quantization is lossless at 4× compression — the +0.0001 is rounding. That single result broke the Phase 2 ceiling. The 1.2M int8 index loaded on the free-tier box at 4.8 GB — well under the 20 GB storage limit, and the nearest-neighbor graph fit inside the 4 GB of RAM — and the cluster stayed green. It is wired into the live endpoint (Workers AI query embedding → int8 vectors → kNN blended with BM25). Binary quantization trades about 1% nDCG for 32× compression — kept in the portfolio for a scale where even int8 will not fit, but not needed yet.

The Honest Limit

Parity does not mean shipped-as-default, and here is why. Reranking 40 candidates is fast (~0.45s), but retrieving from scratch across 1.2 million int8 vectors on a free-tier instance takes 2.4 to 7 seconds warm — the nearest-neighbor step alone is about 2.5s, far over the project’s 700ms production target.

So the agent did not pretend the problem away. The default /api/esci/search stays fast BM25 (~0.43s, production-shaped), and the hybrid is opt-in via ?mode=hybrid — the semantic wins are fully demonstrable without making the default search slow. A production-latency hybrid needs a bigger instance, which is an approval-gated cost decision, not an engineering surprise. The system tells you exactly what it can and cannot do at the latency it actually runs at.

Try the difference yourself: ?mode=hybrid against the default.

Who Did What

As in every phase, the division of labor is stated plainly. The operator’s role in Phase 3 was direction and approvals — kicking off the phase, the “go” decisions, approving the deploy, catching UI-parity issues. The measurement, the embedding pipeline, the quantization experiment, and the implementation were the agent’s. The human decided what to test and whether to ship; the agent did the testing and the building.

What’s Next

Parity is the entry ticket, not the destination. Retail is the first dataset in this entire project with real graded labels — which means it is the first place the classical retail-search toolkit gets a fair audition. Field weighting, learning-to-rank, query understanding: techniques that sat dormant through two phases for lack of labels finally get to prove whether they earn their keep on real product data. The next part is that audition — and the answer is more surprising than “the textbook works.”

Takeaways

1. The academic core transferred. BM25 + BGE hybrid hits parity with the published single-model baseline on 1.2M real products — the two phases of academic rehearsal paid off exactly as designed.
2. Claim the right baseline. Parity at production latency with the single-model baseline is honest; chasing a query-time ensemble leaderboard at 7-second latency is not.
3. int8 was free. Lossless 4× compression is what turns “the hybrid works in an experiment” into “the hybrid runs on the hardware we have.”
4. State the mode, and state the coverage. Official rerank and end-to-end retrieval are different questions; conflating them, or comparing an incompletely-judged retrieval score to a published number, is how honest projects accidentally lie.

Phase 1 proved the method. Phase 2 found out what it learned. Phase 3 hands the agent products — and the first thing the products confirmed is that nothing so far was wasted.

References

Reproduction figures are measured on this project’s own free-tier stack; comparison targets are published numbers, cited below. Phase 3 experiment artifacts and the architecture ledger are public in the project repository.