Local AI for businesses: what is already possible

Local AI is still often associated with a single language model running on a powerful computer. For real applications, that definition is now too narrow.

An AI system can classify emails, search internal documents, retrieve information from CRM or ERP systems and process voice channels. The language model itself is only one component. The important part is how models, data sources, tools and infrastructure work together.

Such a system also does not have to run entirely inside the same building. Between a model on a local workstation and a public AI service, several deployment models offer different levels of control and operational responsibility.

This article explains how local or controlled AI infrastructure can be designed, why smaller models can be sufficient for clearly defined tasks and how RAG, fine-tuning and data protection fit into the architecture.

Local AI is not a single deployment model

In the strictest sense, local AI means that a model runs directly on a device or inside an organisation’s own network. The model itself does not need to send prompts and outputs to an external AI provider.

For organisations, however, physical location is only one part of the decision. It also matters who controls the infrastructure, which data leaves the environment and which components are operated by external providers.

Deployment	Model location	Direct control	Operational effort
Local workstation	user device or workstation	very high	medium
Internal server	organisation’s network	very high	high
Self-managed server	data centre	high	high
Managed model infrastructure	selected region or data centre	depends on service	lower
Public AI service	provider infrastructure	depends on provider	very low

The language model is only one component

A bare language model accepts text and produces text. Additional data sources and tools turn it into a system that can participate in a real workflow.

Email
Web form
Phone
internal application
      |
      v
prepare input
      |
      v
language model
      |
      +---- document search
      +---- CRM
      +---- ERP
      +---- calendar
      +---- internal APIs
      |
      v
response, classification
or defined action

The model does not need to contain all required information itself. It can identify what information is needed and retrieve it from the appropriate system.

Why smaller models can be sufficient

Large general-purpose models are expected to solve many different tasks. A model embedded in a defined workflow often has a much smaller job.

It may only need to classify an incoming request, process information from approved sources and generate a structured output.

For such workloads, general benchmark performance is not the only important metric. Reliability on the specific task can matter more.

RAG: business knowledge does not have to be trained into the model

Retrieval-augmented generation, or RAG, keeps documents and knowledge sources outside the language model. Before the model answers, the system retrieves relevant information and provides it together with the request.

Question
  |
  v
search internal data
  |
  v
relevant information
  |
  v
language model
  |
  v
answer based on
provided context

This is especially useful for information that changes regularly. Updating a product description or internal policy does not require retraining the language model.

The German Federal Commissioner for Data Protection and Freedom of Information notes that RAG can enable smaller language models because relevant information is supplied from an external database. Information stored outside the model can also be changed or deleted more easily.

RAG does not guarantee correct answers

If the retrieval layer selects the wrong documents, the data source is outdated or the model misinterprets the supplied context, the result can still be wrong.

System quality therefore depends on the model, the source documents and the retrieval process.

When does fine-tuning make sense?

Method	Typical purpose	Changes the model?
Prompting	describe task and rules	No
RAG	supply current or internal knowledge	No
Fine-tuning	adapt recurring behaviour	Yes
Training from scratch	create a new foundation model	Yes, completely

RAG is often a better fit when the system needs current internal knowledge. Fine-tuning becomes useful when the desired change concerns recurring behaviour, classification or output structure.

Example: processing customer requests from several channels

A system can first identify what an incoming request is about and then retrieve the information required for that specific case.

An order-status request can use current ERP data, while a technical question can use relevant documentation through RAG.

Voice can use the same underlying workflow

Phone calls add speech recognition and speech synthesis around the same processing pipeline.

Call
  |
  v
speech recognition
  |
  v
text
  |
  v
language model
  |
  +---- internal data
  +---- documents
  +---- tools
  |
  v
response text
  |
  v
speech synthesis

Local AI does not automatically mean fully offline

A local language model can still use external speech services, cloud databases or web search. In that case the model execution remains local while other data leaves the controlled environment.

Each component therefore has to be evaluated separately.

A server in Germany does not automatically make the system GDPR-compliant

Infrastructure location can matter for data protection and international transfers, but it does not by itself determine whether a specific processing activity is lawful.

Purpose, legal basis, data categories, access rights, retention periods and external processors still need to be considered.

The European Data Protection Board explicitly takes a case-by-case approach to personal data processing in the context of AI models.

Questions to answer before implementation

• What exact task should the AI perform?
• Which data does it need?
• Which data must it not access?
• Which systems does it need to read?
• Which actions may it execute?
• Which steps require human approval?
• Which components must remain local or within a specific region?

How I would approach a prototype

A first prototype should test one narrow task with representative example data rather than automate an entire workflow at once.

Only after that task works reliably should additional data sources, tools and automatic actions be added.

Local and cloud infrastructure can be combined

A smaller internal model can handle standard cases while more difficult requests are escalated to a stronger external model or to a human.

In a hybrid architecture, the important question is which data is allowed to follow that external route.

Conclusion

Local AI is no longer just a language model generating text on a single computer. Real systems combine models with internal data sources, retrieval, voice and other applications.

Smaller models can be sufficient for clearly defined tasks, while RAG provides current domain knowledge without embedding that knowledge permanently in the model.

Whether the system runs locally, on a self-managed server or partly on external infrastructure is ultimately an architecture decision. The complete data flow matters more than the physical location of a single model.

The best starting point is therefore not the question of which model is largest, but which concrete task has to be solved and which data the system actually needs.

If you want to evaluate a concrete AI workflow or test it as a small prototype first, we can define the required architecture together. Book a first conversation.