When Experience Becomes the Problem
A clever AI search architecture worked exactly as designed - until newer capabilities made it unnecessary. A real-world lesson in how yesterday’s expertise can become today’s complexity.
I built the right solution for the way software used to work.
By Prateek Sharma, MCA
I had a good solution.
Actually, I was quietly proud of it.
And then someone asked a question I really didn't want to hear.
We were building Bradley, a conversational facilities management assistant. A user could report something like a water leak in “the court building,” and Bradley needed to figure out which building they meant before creating the service request.
I had been writing software for 16 years.
I knew how to solve this problem.
So I did.
I Did What Experience Taught Me
I loaded the building data into memory, used Fuse.js for fuzzy search, ranked the results in JavaScript, and handed the five best candidates to the model.
Clean.
Fast.
Predictable.
Good software engineering.
Then Brian, our CEO, challenged the architecture.
He ran the same task through a general-purpose model with access to a search tool.
Given “the court building,” the model reasoned that the user might mean court, courthouse, court house, judicial centre, or something else entirely.
It generated variations and searched for them itself.
My carefully constructed ranking algorithm couldn't do that.
I'd built a very small, very stubborn brain and put it between a much better brain and the data.
Ouch.
Yesterday's Best Practice
The problem wasn't that I didn't know what I was doing.
That's what made the lesson uncomfortable.
I had done exactly what years of traditional software development taught me to do.
Constrain the inputs.
Narrow the possibilities.
Preprocess the data.
Hand the next component a clean, predictable result.
Those instincts had served me well for years.
But the technology underneath the architecture had changed.
My experience hadn't caught up yet.
So we deleted my clever solution.
Fuse.js went away. The local caching went away. We gave the model direct access to a search API and let it control the query.
We went from 49 buildings in the demo to more than 55,000 buildings and 250,000 floors in production.
And the code got smaller.
Expertise Has an Expiration Date
Not all of it, obviously.
Sixteen years of software engineering didn't suddenly become useless because AI showed up. I still needed everything those years taught me about APIs, state, performance, failures, debugging, architecture, and what happens when software meets real users.
But I learned something else.
Experience can tell you what worked before.
It can't guarantee that it's still the best answer.
And right now, “before” doesn't necessarily mean ten years ago.
It might mean last year.
Sometimes it might mean last month.
That's uncomfortable when you've spent a career becoming good at something. Expertise is supposed to make you better at recognizing the right solution.
Sometimes it makes you faster at recognizing an old one.
That's why I don't think learning AI engineering is mostly about throwing away what experienced developers already know.
It's about being willing to question which parts still apply.
Even when the code works.
Even when the architecture makes sense.
Even when you're proud of it.
Especially then.
Want to know more? See the White Paper here: https://blog.protovate.ai/we-stopped-paying-per-token-and-put-the-models-on-our-own-machine/
Originally published on Protovate.AI
Protovate builds practical AI-powered software for complex, real-world environments. Led by Brian Pollack and a global team with more than 30 years of experience, Protovate helps organizations innovate responsibly, improve efficiency, and turn emerging technology into solutions that deliver measurable impact.
Over the decades, the Protovate team has worked with organizations including NASA, Johnson & Johnson, Microsoft, Walmart, Covidien, Singtel, LG, Yahoo, and Lowe’s.
About the Author
