Stop Treating the Model Like a Chatbot

Production AI is more than a chatbot. Models work best as components within larger systems, using structured outputs, search, APIs, databases, and other tools to get the job done.

Share
Bradley the AI robot selects from tools for search, databases, APIs, calculations, and documents.
The model doesn’t need to know everything. It needs the right toolbox.

The model is one component. The application is the system.

By Prateek Sharma, MCA

Most people meet an AI model through a chat window.

You type something.

It types something back.

So when developers first start building with AI, it’s natural to carry that same mental model into the application.

Ask the model a question.

Get an answer.

Do something with the answer.

That works for a prototype.

But eventually, I had to stop thinking about AI as a chatbot and start thinking about it as another component in a software system.

That changed how I built everything around it.

Prompts Aren’t Just Questions

When you’re using ChatGPT, a prompt can be conversational.

When you’re building a production application, the prompt starts looking a lot more like a specification.

What is the model responsible for?

What information will it receive?

What decisions is it allowed to make?

What should it return?

What happens when it doesn’t know?

Those aren’t really conversation questions.

They’re interface questions.

On Bradley, our conversational facilities management assistant, a user might say:

There’s water leaking in the court building.

Bradley needed to determine what the user meant, identify the correct location and service category, gather whatever additional information was required, and eventually create a work order.

A pleasant answer wasn’t the goal.

The application needed usable information.

Output Becomes Data

That distinction matters.

If another part of the application needs a building ID, giving it a beautifully written paragraph about the building isn’t particularly helpful.

Software needs structure.

So instead of asking the model to produce prose and then writing increasingly creative code to figure out what the prose meant, we used structured output.

The model could return defined fields that the application could validate and use.

Now the model wasn't simply answering.

It was participating in a workflow.

That also made failures easier to see.

A missing field is visible.

An invalid value can be rejected.

A beautifully confident paragraph containing the wrong thing is considerably harder to deal with.

Give the Model Tools

The next shift was realizing the model didn't need to carry everything around in its prompt.

It needed access to the right capabilities.

Search is a good example.

Instead of trying to predict every variation a user might use for a building name and coding all that logic ourselves, we could give the model a search tool.

Now it could reason about what the user might mean, generate useful search variations, inspect the results, and continue from there.

The same idea applies to APIs and other systems.

The model doesn't need to be the database.

It needs a controlled way to query the database when appropriate.

It doesn't need to memorize every business rule.

Some rules belong in deterministic code where they can be enforced predictably.

And it certainly doesn't need permission to do absolutely everything just because it can reason about what should happen next.

The Model Isn't Bradley

This was probably the biggest mental shift for me.

The model isn't Bradley.

Bradley is the whole system.

It's the model, prompts, search, retrieval, APIs, tools, guardrails, business rules, logging, databases, work-order integration, user interface, and ordinary application code surrounding all of it.

And that distinction matters when something goes wrong.

We've had situations where Bradley appeared to misunderstand a user when the model never received the correct search result.

We've had guardrails reject legitimate input.

We've had ordinary production infrastructure fail while the AI itself was working perfectly.

If you think AI = model, you debug the wrong thing.

Build the System

Once I stopped thinking of the model as the application, the architecture became easier to reason about.

The model didn't need to know everything.

It didn't need to do everything.

It needed a clearly defined job, the right information, and the right tools.

Which, come to think of it, is pretty much what we ask of every other component in a software system.

The chatbot was never the architecture.

It was just the first interface we met.


Want to know more? See the White Paper here: https://blog.protovate.ai/we-stopped-paying-per-token-and-put-the-models-on-our-own-machine/


Originally published on Protovate.AI

Protovate builds practical AI-powered software for complex, real-world environments. Led by Brian Pollack and a global team with more than 30 years of experience, Protovate helps organizations innovate responsibly, improve efficiency, and turn emerging technology into solutions that deliver measurable impact.

Over the decades, the Protovate team has worked with organizations including NASA, Johnson & Johnson, Microsoft, Walmart, Covidien, Singtel, LG, Yahoo, and Lowe’s.

About the Author

Author

Prateek Sharma

AI engineer at Protovate

Prateek Sharma is an AI engineer at Protovate with over a decade of experience building and integrating intelligent applications across AI, mobile, and full-stack development. He brings a hands-on, practical approach to turning advanced technology into reliable, production-ready systems.