AI model routing means choosing the right AI model for each piece of work instead of sending every request to the same one.
That sounds obvious, but it is not how many business AI systems are currently designed.
A company selects OpenAI, Anthropic or another provider, connects its software to one model, and gradually sends more and more work through the same route.
Internal questions go there. Document processing goes there. Email classification goes there. Complex reasoning goes there. Automated workflows go there.
The model may be extremely capable.
That does not mean it is the right model for every one of those jobs.
As smaller open-weight models become more capable, local inference becomes practical and frontier models continue to improve, we think businesses should increasingly stop asking:
“Which AI model should our company use?”
and start asking:
“Which model should handle this particular task?”
One company does not have one type of AI workload
The fundamental problem with a one-model architecture is that company workloads are not uniform.
Compare these requests:
- extract an invoice number from an email;
- classify a support enquiry;
- summarise an internal meeting;
- answer a question from an employee handbook;
- review a complex software architecture;
- analyse a lengthy contract;
- reason through an unfamiliar operational problem;
- process a confidential internal document.
All of them can involve AI.
They do not have the same requirements.
Some need speed.
Some need low cost.
Some need access to private company information.
Some require deeper reasoning.
Some may need a model capable of using tools or processing a particular type of media.
Sending all of them to exactly the same model means making one technical decision cover several different problems.
That is convenient.
It is not necessarily efficient.
AI model routing is already becoming normal infrastructure
The idea of routing AI requests between models is not speculative.
Major cloud platforms are already building it directly into their AI infrastructure.
Microsoft Foundry’s model router is designed to inspect an incoming prompt and select an appropriate model in real time. Microsoft describes the objective as maintaining performance while reducing unnecessary compute cost.
The current router supports models across several capability levels and providers, including multiple OpenAI models, Anthropic models, DeepSeek, Meta Llama and OpenAI’s gpt-oss-120b.
Microsoft also offers different routing modes.
- Balanced attempts to balance quality and cost.
- Cost favours lower-cost models more aggressively.
- Quality prioritises the highest-quality response regardless of cost.
The logic behind this is important.
Microsoft’s own documentation says that not every coding question, summary or conversation requires the same reasoning depth. Its router attempts to identify relatively straightforward requests that can be handled by faster, cheaper models while reserving frontier models for prompts that require deeper reasoning or more complex tool use.
Amazon Bedrock Intelligent Prompt Routing follows a similar principle. AWS routes requests between models according to predicted response quality and cost rather than automatically sending every prompt to the larger model.
That is essentially the architectural argument businesses should now consider for their own AI systems.
How AI model routing can work inside a business
A business does not need Microsoft’s or Amazon’s managed router to adopt the principle.
Routing can happen inside the application itself.
The software can examine what a request needs and decide which model should receive it.
| Requirement | Possible route |
|---|---|
| Simple classification | Small local model |
| Routine document extraction | Small local model |
| Internal company Q&A | Local model + company knowledge |
| Confidential internal content | Private/local model only |
| High-volume repetitive task | Efficient local model |
| Complex reasoning | Frontier model |
| Difficult coding problem | Frontier model |
| Local model produces insufficient confidence | Escalate to stronger model |
This is only an example. The correct routes depend on the organisation.
The important idea is that the application makes a model-selection decision before inference takes place.
Local AI can become the default instead of the exception
This is where AI model routing becomes particularly relevant to the move towards local AI.
In Does Every Employee Really Need the Most Powerful AI Model?, we argued that a large amount of routine business AI work does not necessarily require frontier-level capability.
If that is true, there is little reason for the strongest external model to be the default route.
The architecture can be reversed.
Instead of:
Send everything to the frontier model unless there is a reason not to.
a company can design for:
Process work locally unless there is a reason to escalate it.
That is a meaningful difference.
The local model becomes normal infrastructure.
The frontier model becomes a specialist resource.
Why pay for intelligence a task does not use?
The economic argument follows naturally.
In Self-Hosted AI: Is It Cheaper Than OpenAI or Anthropic?, we looked at the difference between paying for external inference request by request and operating dedicated inference capacity.
Model routing adds another layer to that calculation.
Even if a company retains access to a frontier API, it does not have to pay frontier-model rates for every request.
Imagine an internal AI system receiving thousands of requests.
Some may be simple lookups.
Some might involve extracting structured data.
Some may require little more than categorising a short piece of text.
A smaller model may handle those tasks perfectly well.
A much smaller proportion might involve genuine ambiguity, difficult reasoning or complex tool use.
Those requests can be escalated.
We should not assume a particular percentage of company requests will fall into either category without measuring the workload.
But the underlying principle is straightforward:
if only some requests require expensive intelligence, only those requests should pay for it.
Routing can consider more than difficulty
Model capability is only one reason to choose one route over another.
A useful AI router can consider several characteristics at once.
1. Data sensitivity
A request containing confidential company information may be restricted to locally controlled infrastructure regardless of whether an external model would produce a slightly better answer.
2. Task complexity
A straightforward classification job may go to a small model while an ambiguous multi-stage problem is escalated.
3. Cost
High-volume tasks can favour models with lower inference requirements.
4. Latency
A fast local model may be preferable for interactive workflows where response time matters more than marginal gains in reasoning quality.
5. Model specialisation
Different models may perform better on coding, structured extraction, language tasks, tool use or particular internal workflows.
6. Availability
If one provider or deployment is unavailable, another model can become a fallback rather than taking the entire AI system offline.
This is why model routing is ultimately bigger than cost optimisation.
It becomes part of the application’s architecture.
Private information can have its own route
One of the strongest uses for routing is separating sensitive and non-sensitive work.
A business might decide that requests involving:
- customer records;
- source code;
- contracts;
- financial documents;
- product roadmaps;
- internal research;
- employee information;
- or other proprietary material
must remain inside company-controlled infrastructure.
Other requests involving public information can use an external frontier model when there is a genuine advantage.
That gives privacy a technical enforcement mechanism rather than relying entirely on employees to remember which AI tool they should use for which type of information.
OpenAI’s gpt-oss documentation is relevant here because those models can run on infrastructure controlled by the organisation, including on-premises or in a private cloud. OpenAI states that it does not receive the data sent to genuinely self-hosted gpt-oss deployments unless the user explicitly shares it with OpenAI or uses a managed hosting provider.
This means a model router can make decisions based not only on capability, but also on where the data is allowed to go.
Open-weight models make hybrid architecture much more practical
A few years ago, building a serious multi-model architecture often meant routing between several external APIs.
That is still possible, but open-weight models add another option.
A business can operate part of the model pool itself.
OpenAI says its gpt-oss-20b model can run with around 16GB of memory, while gpt-oss-120b can fit within 80GB.
Other open-weight model families provide different sizes and capability levels.
That means a company’s AI architecture can include:
- a lightweight local model;
- a larger locally hosted model;
- a specialist model for a particular task;
- and one or more frontier APIs.
The system does not have to treat them as competitors.
They can be different tools inside the same application.
A specialised model may beat a larger general model at the job that matters
There is another reason not to reduce model selection to size.
The largest general-purpose model is not automatically the best model for every tightly defined business workflow.
A smaller model can be:
- fine-tuned for a narrow task;
- combined with a controlled company knowledge base;
- given access to specific tools;
- constrained to a structured output;
- or optimised around a repeatable workflow.
That means the question is not simply:
“Small model or large model?”
It can also be:
“General model or specialised model?”
Nort has previously covered some of the technical methods behind reducing inference requirements in AI Model Optimization: Techniques for Faster and Efficient Inference.
As AI becomes part of ordinary software architecture, these decisions become increasingly similar to other engineering decisions: choose the component that performs the required job reliably without allocating unnecessary resources.
AI model routing does not have to be intelligent on day one
The word “router” can make this architecture sound more complicated than it needs to be.
A company does not necessarily need another AI model deciding which model to call.
Routing can begin with simple deterministic rules.
For example:
If confidential_data = true:
use local_private_model
Else if task = "classification":
use small_local_model
Else if task = "document_extraction":
use specialist_model
Else if complexity = high:
use frontier_model
Else:
use default_local_model
That type of architecture can be easier to understand, audit and test.
More advanced routing can come later.
Microsoft’s current model router demonstrates the more sophisticated version: its routing model analyses the request and estimates which underlying model is best suited to it.
But even Microsoft recommends evaluating the router against representative workloads rather than assuming automatic routing is always optimal.
Its evaluation guidance recommends comparing quality, cost, latency and policy requirements against a real workload baseline.
That is good advice whether the router is managed by Microsoft or built specifically for the business.
The router itself can make mistakes
There is an obvious weakness in model routing.
The system has to correctly decide which requests need more capability.
If it sends a difficult task to an inadequate model, quality can fall.
If it sends every borderline task to the most expensive model, much of the cost advantage disappears.
Routing therefore needs monitoring and evaluation.
Useful measures include:
- which model handled each request;
- whether the answer passed the required quality threshold;
- how often requests were escalated;
- cost per successful task;
- latency;
- failure rate;
- and how often human intervention was required.
The aim is not to minimise model cost at any price.
The aim is to use the least expensive and least resource-intensive model that still produces an acceptable result.
Fallbacks are another reason not to depend on one model
Multi-model architecture can also improve resilience.
If an entire application depends on one provider, one endpoint and one model, problems with any of them can affect the whole system.
A routing layer can provide controlled fallbacks.
If a local model is unavailable, a secondary local deployment may take over.
If an external provider fails, another approved model can handle appropriate requests.
If a model produces an unacceptable result, the application may retry with a stronger model.
That does not mean blindly swapping models. Outputs, tool behaviour and prompt compatibility can differ significantly, so failover needs to be designed and tested properly.
But having more than one route gives the application options that a single-model architecture does not.
The business should own the routing decision
This is perhaps the most important part.
A business should not need to restructure its entire AI system every time a new model becomes the industry favourite.
The application should define what it needs:
- minimum quality;
- acceptable cost;
- maximum latency;
- privacy requirements;
- required tools;
- allowed providers;
- and escalation rules.
Models can then compete for the workloads they are suited to.
A new local model can be introduced and tested.
A new frontier model can become the escalation option.
An expensive model can be removed from routine tasks.
A sensitive workflow can be locked to local infrastructure.
The application remains the stable layer while individual models change underneath it.
One model for the whole company is convenient, but probably temporary
Using one AI model for everything made sense when businesses were experimenting.
Choose a capable API, connect it to some software and find out whether the idea works.
That is still a sensible way to prototype.
But production AI is a different problem.
As usage grows, the differences between workloads become harder to ignore.
Some tasks are cheap and repetitive.
Some are sensitive.
Some require specialist knowledge.
Some genuinely need the best reasoning available.
Those requirements should not automatically produce the same technical answer.
Our view at Nort Labs is that AI model routing will increasingly become a normal part of business AI architecture.
Local models can handle routine, private and high-volume work. Specialist models can be assigned to tasks they perform particularly well. Frontier models can remain available for requests where their additional capability justifies the cost.
The result is not necessarily less access to powerful AI.
It is more selective access to it.
And for companies moving from AI experiments into serious operational use, that is likely to be a much more efficient architecture than sending everything through the same model simply because it was the first one they connected.
Nort Labs can design local and hybrid AI architectures that route work according to capability, cost, privacy and workload requirements, while retaining frontier AI access for the tasks that genuinely need it.
