How we replaced gpt‑4o‑mini: test on your own data
OpenAI is retiring gpt-4o-mini, which our events pipeline relied on. Here is how we picked a replacement using a test set built from real parent complaints, and what that means for AI work we do for clients.
OpenAI’s email came with a deadline: GPT-4o, gpt-4o-mini, GPT-4.1 and GPT-4.1 mini retire on January 14, 2027. The AI pipeline behind Tailend used gpt-4o-mini to read event pages, so we had to pick a replacement.
Earlier rounds had not found one that matched it on both cost and accuracy. This time one did. The way we found it is the same way we approach AI work for clients, so it is worth walking through.
Generic benchmarks did not answer our question
Our job is narrow. Read a messy local page and pull out each event: its dates and times, whether and how it repeats, price, ages and location. A model that scores well on a general test can still struggle with a church-hall flyer that gives no year, or a page where the schedule hides in a heading.
So the question was not “which model wins?” It was “which model gets our pages right, at a cost and speed we can live with?”
The test set came from parents
We built our test set while running TinyJawns. Parents reported listings that were wrong, and we turned those reports into test cases: real local pages, labelled by hand, with the answer we expected.
That gave us the awkward cases a generic test would never include:
- dates without a year
- recurring schedules stated only in a section heading
- dense pages with many similar items
- date ranges
- prices that say “inquire” or vary by package
We tested in two stages. A quick screen of many models on about 15 hand-labelled pages per task told us which were worth a closer look. The finalists then ran on the full set, 32 pages with 87 events, several times each.
What we found
| Setup | Events found | Details right | Cost vs. gpt-4o-mini |
|---|---|---|---|
| gpt-4o-mini (today) | 92% | 92.2% | 1.00x |
| GPT-6 Luna, low reasoning | 95% | 92.0–92.2% | about 0.72x |
| GPT-6 Luna, plus a second look at dates | 95% | 93.3–93.7% | about 0.77–0.85x |
GPT-6 Luna on low reasoning found more of the events, got the details right about as often, and cost about 28% less. A targeted second pass that rechecks only dates and repeat schedules lifted accuracy a little more for a small extra cost.
A few other results shaped the decision:
- Reasoning effort mattered more than the model. gpt-5-nano went from 71% to 91% detail accuracy as we raised its effort, but its typical time per page went from about 1 second to 79, and its cost from 0.30x to 2.04x.
- Fix the weak spot, not everything. Dates and repeat schedules were where cheaper settings slipped. Rechecking just those beat running every page at high effort.
- Slow can be expensive. One open-weight model looked cheap per token but took 24 to 51 seconds per page in our test. We pay for compute while a page is in flight, so at scale that would cost more, not less.
- Check on pages you did not tune on. One promising tweak looked good on the first batch and did not hold up on held-out pages. We did not ship it.
The full numbers and charts are in the original write-up on tailend.dev.
What this means for AI work we do for clients
When a client asks for an assistant that drafts quotes, sorts an inbox or answers customer questions, we start the same way. Before choosing a model we collect a few dozen real examples from the business, write down the right answer for each, and test candidates against them. We look at accuracy, cost per task and speed together, because a cheap model that makes a customer wait a minute is not cheap.
That test set stays with the project. Models will keep being retired and replaced, and when that happens you rerun the test instead of guessing.
If there is a task in your business you would like AI to handle, tell us what it is. Bring the messy examples. They are the useful part.