Ollama: how to run local AI over your own company data

Ollama is a program that lets a firm run a language model (LLM) on its own server or computer, so questions and documents stay in your network. You install it with one command, download a model with another, and Ollama opens an API on port 11434 that your own programs can call too. Ollama is free (MIT licence); you pay for hardware, electricity and an administrator's time. What decides it is the size of the graphics card's memory: the model has to fit entirely. For summaries, e-mails and short answers, a model with 8 to 14 billion parameters is enough by our estimate. What a model can do on its own in a firm after launch is covered in the article on AI agents.

What Ollama is and what you can run in a firm with it

Local AI is a language model that runs on a computer under your control, not at an outside provider. Ollama downloads it, runs it and exposes it through an API. According to the guide it runs on macOS, Windows and Linux, and there is also a Docker image.

On Linux, curl -fsSL https://ollama.com/install.sh | sh is enough. Then ollama run gemma4:e2b downloads a model (about 7.2 GB according to the guide) and opens a chat in the terminal. Nobody in a firm will sit in a terminal, so you'll use the API, covered further down. With Docker, you start the server with docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama, and for an NVIDIA card with the switch --gpus=all.

Ollama is open source under the MIT licence. Model licences differ, so always check them for the specific model.

How much memory and what hardware a local LLM needs

The main rule: the model has to fit into the graphics card's memory (VRAM). If it doesn't, the command ollama ps shows in the Processor column how the model is split between the card and the computer's memory, for example 48%/52% CPU/GPU. We didn't measure speed, but expect slower answers. Ollama supports NVIDIA cards with compute capability 5.0 and newer (driver 550 and newer), AMD Radeon with the ROCm v7 driver on Linux, and Apple through Metal. For scale: the RTX 4090 has 24 GB of memory and the RTX 5090 32 GB.

The table shows file sizes from the library on ollama.com and licences from the model cards. Add a reserve for context to the file: the default window is 4,096 tokens, and a longer context takes extra memory. The "Where it fits" column is our estimate from the file size, not a manufacturer's figure, unless stated otherwise.

The Llama 3.1 licence requires a special request to Meta only when a firm's products have more than 700 million monthly active users. Small and medium firms aren't touched by it.

An example for a firm of 20 people where only a few ask at the same time: the qwen3:14b model has a 9.3 GB file and the bge-m3 embedding model (see below) 1.2 GB. Together that is 10.5 GB. With a reserve for context, a card with 16 GB is enough. A bigger model or more simultaneous queries raise the demands fast.

Which open model suits which task

Summaries, e-mails, short answers: models of around 8 to 14 billion parameters are enough, for example qwen3:14b or gemma4:12b. They run on one ordinary card.

Czech: Llama 3.1 officially supports eight languages (English, German, French, Italian, Portuguese, Hindi, Spanish and Thai), and Czech isn't among them. Qwen3 claims support for more than 100 languages and Gemma 4 support for more than 35 languages with pre-training on more than 140. These figures don't say how much of that is good Czech. Try the models on your own texts.

Answers from documents: you need an embedding model as well. In its blog post of 8 April 2024, Ollama names mxbai-embed-large, nomic-embed-text and all-minilm, and the library also offers bge-m3 (1.2 GB, more than 100 languages according to its page). It is small and runs alongside the main model.

Harder reasoning and longer analyses: bigger models have more capacity, but gpt-oss:120b needs 65 GB and llama3.1:70b 43 GB.

How to run local AI over your own company data

A model knows nothing about your firm by itself. To make it answer from your documents, you use RAG (retrieval augmented generation). Ollama describes it in its blog post on embeddings.

1. Install Ollama on a server in your network and check that the model runs on the card (ollama ps).
2. Download both a generative and an embedding model with ollama pull.
3. Split the documents into chunks, turn them into vectors and store them in a vector database (Ollama's example uses Chroma).
4. On a query, find the nearest chunks and send them to the model together with the question.
5. Your programs call the API. Ollama offers an OpenAI-compatible interface at /v1/ (chat completions, embeddings and responses), so existing tools can often just be pointed at your server's address. The key is entered as a formality; Ollama ignores it.

Watch the permissions. Simple RAG doesn't know who is asking and returns a chunk from a document the asker must not see. The permission filter has to run before the chunks reach the model. That is our recommended practice, not part of Ollama.

How to make sure the data doesn't leave your network

• The API address. According to the FAQ, Ollama by default listens only on 127.0.0.1 and port 11434. If you open it to the network through OLLAMA_HOST, put a firewall and user sign-in in front of the API, for example through a reverse proxy. Watch out with Docker: the option -p 11434:11434 exposes the API on the host's port.
• Cloud features. Ollama also offers cloud models and web search. The variable OLLAMA_NO_CLOUD=1 turns them off. With purely local running, according to the manufacturer's FAQ, it sees neither prompts nor data.
• Cut the server off from the internet. Once you've downloaded the models, block outbound traffic from the server on the firewall and check that everything still works. Then you have proof, not just a promise.
• Logs and backups. Conversations and the vector database contain company data. Control access to them the way you do for the documents they came from.

How a local model differs from ChatGPT and Claude

We found nothing in open sources on how good local models are against the cloud, so we don't state it. Take 20 real tasks from your firm and run them on a local model and in the cloud. You'll see the difference better than from any leaderboard. What of this runs in practice in firms in 2026 is covered in the article on AI agents for business.

Running it has another side. By default Ollama handles one query per model at a time (OLLAMA_NUM_PARALLEL has the value 1), the others wait in a queue, and a model is unloaded from memory after 5 minutes without a query. When the queue grows past 512 requests, Ollama rejects further ones with a 503 error. You have to tune, update and watch the server yourself. On the other hand, you don't pay for every token and costs are predictable.

Is local AI compliant with GDPR

Running locally doesn't meet GDPR for you. You still need a legal basis, to inform people and to secure the data. In Opinion 28/2024 of 17 December 2024, the EDPB states that no legal basis outranks the others and the controller has to pick the suitable one. For legitimate interest it has to go through three steps: identify the interest, check that you really need the data, and weigh whether people's rights override it. The EDPB says the same about models trained on personal data: they can't always be considered anonymous.

The advantage of local running lies elsewhere. When the model runs at your site and nobody else has access to the data, there is no processor contract, which under EDPB Guidelines 07/2020 Article 28(3) GDPR requires for the work of a processor. The technical side of data protection is covered in the article on the technical side of GDPR.

Where xOffice fits in

xOffice is Lumenax's ERP on open source Odoo. The AI assistant in it is optional. According to the xOffice page you have four routes: none, your own account with a provider (you pay it for usage), a model on Lumenax's infrastructure in the EU from 4,900 CZK a month, or an AI server in your network from 5,999 EUR one-off plus installation. The page recommends the last variant only for cases with the highest protection needs, where data must not leave the firm. The assistant sees only what the asking user may see, and every change to the data has to be confirmed by a person.

Sources

• Ollama, FAQ, viewed 25 Oct 2026: https://docs.ollama.com/faq
• Ollama, supported GPUs, viewed 25 Oct 2026: https://docs.ollama.com/gpu
• Ollama, Quickstart, Linux and Docker, viewed 25 Oct 2026: https://docs.ollama.com/quickstart
• Ollama, OpenAI API compatibility, viewed 25 Oct 2026: https://docs.ollama.com/api/openai-compatibility
• Ollama, Embedding models, 8 Apr 2024: https://ollama.com/blog/embedding-models
• Ollama, model library (llama3.1, qwen3, gemma4, gpt-oss, bge-m3), viewed 25 Oct 2026: https://ollama.com/library
• Ollama source code, MIT licence, viewed 25 Oct 2026: https://github.com/ollama/ollama
• Hugging Face, model cards Qwen3-14B, Gemma 4 12B and Llama 3.1 8B, viewed 25 Oct 2026: https://huggingface.co/Qwen/Qwen3-14B
• NVIDIA, GeForce RTX 4090 and RTX 5090, viewed 25 Oct 2026: https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/
• EDPB, Opinion 28/2024, 17 Dec 2024: https://www.edpb.europa.eu/system/files/documents/2024-12/edpb_opinion_202428_ai-models_en.pdf
• EDPB, Guidelines 07/2020 on the concepts of controller and processor, version 2.1, 7 Jul 2021: https://www.edpb.europa.eu/system/files/2023-10/edpb_guidelines_202007_controllerprocessor_final_en.pdf
• Lumenax, xOffice ERP system, verified on the page 12 Oct 2026, viewed 25 Oct 2026: https://www.lumenax.cz/erp-system