A local AI is still a long way from a secure AI

A local AI first sounds like a fairly simple answer to many questions companies are dealing with right now.

The language model runs on hardware of your own. The data does not leave the company. No external APIs, no third-party cloud, no data transfer you cannot control.

Technically that is comparatively simple to do today. With projects such as Ollama, current language models can run locally within a few minutes.

But one decisive question is still unanswered:

What is this AI actually allowed to do?

In my view that question still gets too little attention in many discussions about local AI.

Because as soon as a language model becomes more than a chatbot and is supposed to work with company data and company systems, it is no longer only about the right model. It is about identities, permissions and controlled actions.

The problem starts with the first access

Take a simple example.

A company runs a local AI on a Linux server. The AI should be able to search internal documents.

Technically you could just give the process access to a folder:

/data/knowledge

The documents are indexed, stored in a vector database and can then be searched with a RAG approach. That is often how the start of an AI knowledge base looks in practice.

That works.

It gets interesting as soon as the AI should also access other data or run actions.

Perhaps:

• /data/knowledge
• /data/projects
• /data/templates
• /data/crm

Perhaps it may create documents, but not overwrite existing ones.

Perhaps it may only read certain data.

Perhaps it should later even run actions in an ERP system.

At that point the sentence „The AI runs locally“ is no longer enough.

Local language model

Ollama, vLLM or comparable inference

/data/knowledge
Documents

/data/projects
Projects

/data/templates
Templates

/data/crm
CRM data

ERP
Actions

Without clear rights the AI potentially sees everything the process is technically connected to.

An AI should not automatically be allowed to see everything

An obvious approach is to run the AI under a technical user of its own.

On Linux, for example, a dedicated service account could be used:

user: ai-reader

That user gets only the rights needed for its task.

For example:

Path	Right
/data/knowledge	read
/data/drafts	read/write
/data/finance	no access
/etc	no access
/home	no access

The idea is not new. It matches a basic principle of classic IT security: least privilege.

A service gets only the permissions it actually needs.

For a local AI that principle matters especially. A model that runs locally can be given overly broad permissions just like any other process.

But the service account alone is not enough

So far we have only answered one question:

What may this AI service do in principle?

In a company environment there is a second dimension:

What may the concrete employee do?

Take two people.

Person A works in sales.

Person B works in HR.

Both use the same internal AI.

The sales person may access offers, customer data and sales documents. HR may access personnel files.

The AI should therefore not simply say: „I am installed locally and may read everything.“

It has to take the user’s context into account.

That creates two separate permission layers.

Human

Employee identity

Active Directory, Entra ID or LDAP. Sales may see CRM, HR may see personnel files.

AI

Technical identity

ai-reader, ai-editor, ai-crm, ai-erp. Each agent has only the rights for its task.

User permission + agent permission = effective permission

Humans and AI need different identities

The human user has an identity, for example via Active Directory, Entra ID or LDAP.

The AI, or a single AI agent, also has a technical identity.

For example:

• ai-reader
• ai-editor
• ai-crm
• ai-erp

Those technical identities can each have different rights and capabilities.

That lets you state a relatively simple base model:

The employee brings their permissions. The AI agent brings its own capabilities and limits. A concrete action should only be possible if both layers allow it.

Simplified:

User permission
+
Agent permission
=
effective permission

That is deliberately simplified. In a real environment extra factors such as data classification, the concrete resource, purpose, approvals or the risk of the action can also play a role.

That is why the question of a service account increasingly becomes a question of an access layer of its own.

The AI should not work directly with the file system

Taken one step further, I would give the language model itself as few direct rights as possible.

The model should not simply run shell commands or change arbitrary files.

Instead it should use clearly defined tools.

For example:

• search_documents()
• create_draft()
• read_customer()
• create_offer()
• create_ticket()

The model then cannot run arbitrary actions. It can only request a defined action.

A further layer then decides whether that action is allowed.

1. Employee makes a request
2. AI agent requests an action, for example create_offer
3. Policy engine checks user, agent, data, action, approval and logging
4. Tool runs the action only if it is approved
5. CRM, ERP or file system remains the actual source of the data and its permissions

That could look like this, for example:

Employee
    ↓
AI agent
    ↓
"create_offer"
    ↓
Policy engine
    ↓
Is this user allowed to do that?
Is this agent allowed to do that?
Are the required data released?
Is a human confirmation required?
    ↓
Tool
    ↓
CRM / ERP / file system

The language model itself then becomes a relatively small part of the system.

The actual control sits outside the model.

And suddenly a chatbot becomes a permission problem

At that point, in my view, the perspective on company AI changes.

As long as an AI only answers questions, the permission problem is comparatively manageable.

As soon as the AI may act, the situation changes. That is where AI automation starts: not with chat, but with defined actions in existing systems.

An AI might be supposed to:

• Read documents.
• Create documents.
• Prepare emails.
• Create tickets.
• Update CRM data.
• Create offers.
• Fetch information from an ERP system.
• Or at some point even run administrative tasks.

Each of those actions has its own risk and needs its own permissions.

An AI that may summarise a document does not automatically need the right to delete that document.

An AI that may draft an offer does not automatically need to send offers.

An AI that may read server logs should not automatically be able to change server configurations.

We have built those boundaries into classic software for years. For AI agents we have to start applying the same principles consistently.

Before an agent is supposed to run such actions, it is worth asking first whether AI is even the right path for the process. That is what the AI check is for.

The operating system already supplies part of the solution

The interesting part is that we do not have to start from zero.

Linux has had a sophisticated permission model for decades.

Users, groups, file rights, ACLs, containers, systemd, AppArmor or SELinux can be used to isolate processes and restrict access.

On Windows, Active Directory, service accounts and NTFS ACLs also give powerful options.

macOS also brings matching building blocks with Unix permissions, ACLs, sandboxing and further security mechanisms.

The concrete technical implementation differs by operating system.

The basic principle stays the same:

The AI does not get blanket access to the company environment. It gets exactly defined capabilities.

But what about company systems?

This is where the classic file permission model ends.

An ERP system does not work like a folder on a Linux server.

A CRM system does not work like a local file system.

And a SaaS application cannot simply be secured with a service account and an NTFS ACL.

That is why such an architecture also needs a layer above the operating system. A kind of policy or access layer.

That layer could check for every action:

• Who is the user?
• Which agent wants to act?
• Which data is needed?
• Which action should run?
• Is this action allowed?
• Is an approval needed?
• Which technical identity runs the action?
• What has to be logged?

That creates a translation layer between the language of the AI and the language of company IT.

Perhaps AI needs a new permission model

I currently find that question more interesting than which local language model you should use.

Because models become more interchangeable.

Today a model from Meta, Alibaba, Mistral or another vendor may run. Tomorrow another model is better.

The basic question remains:

Which capabilities may an AI agent have inside a company?

That reminds you of classic operating system permissions, but it goes further.

An AI agent does not only have access to resources. It can make decisions and derive actions from them.

So besides the question „May this process read this file?“ we increasingly also have to ask:

„May this agent run this action on behalf of this user?“

That is the point where classic permission models and AI agents overlap. The agent needs not only access to data, but clearly defined capabilities and limits.

A possible structure

Such an architecture could, in simplified form, consist of five layers.

Human

↓

Identity

↓

Policy layer

↓

AI agent
Reader

AI agent
Editor

↓

Tools / APIs / services

↓

Files

CRM

ERP

Mail

The human brings their identity and their company permissions.

The AI agent brings its own capabilities and limits.

The policy layer decides whether a concrete action is allowed under the given conditions.

The tools run the action.

The company systems remain the actual source of the data and its permissions.

The language model itself stays as far away from direct system rights as possible.

Local AI is therefore only part of the story

For especially sensitive applications it can make sense to run the language model fully inside your own infrastructure.

That can happen, for example, with Ollama, vLLM or other local inference setups. More context is on the page Local AI for companies.

That reduces dependence on external AI APIs and can make sense for certain data classes.

But „local“ should not be treated as the same as „secure“.

A model that runs locally and has access to all company data is not automatically better secured than a cloud model.

Security comes from the combination of infrastructure, identities, permissions, isolation, policies and actions you can trace.

Local execution is only one building block in that.

Perhaps that is exactly the interesting part

I do not believe every company necessarily needs an AI platform of its own.

And I also do not believe you should invent an AI use case for every client at all costs.

The more interesting question is the other way around:

Which processes would companies like to support with AI today, if they could trust the AI while doing so?

Perhaps it is internal knowledge search.

Perhaps it is handling email.

Perhaps it is creating offers.

Perhaps it is analysing production data.

Perhaps it is something else entirely.

The concrete use case can change.

The requirements for identity, permissions, isolation and control remain.

That is why such a security and access layer could be more interesting in the long run than another specialised AI chatbot.

Because the models get better.

The company data stays.

The processes stay.

And the question of what an AI may actually do inside that environment becomes more important with every new agent.

Perhaps companies therefore do not need more AI. Perhaps they first need a better way to let AI work securely inside the company.

If you want to check which data, identities and approvals an AI agent should actually get in your environment, I clarify that in 15 minutes without obligation. Book a first conversation.