Ollama is a tool that lets a company run a language model (LLM) on its own server or computer, so questions and company documents stay inside your network. One command downloads the model, and Ollama exposes an API on port 11434 that your own applications can use too. The program is free (MIT licence); you pay for hardware and the time you put into running it.
Local AI is a language model that runs on a device you control, not at a third-party provider. Ollama is an open source program that downloads such models, runs them and exposes them through an API. It runs on macOS, Windows, Linux and in Docker.
What is Ollama and what can a company run on it?
The library on ollama.com holds model families such as Llama, Qwen, Gemma and gpt-oss. Sizes range from hundreds of millions of parameters (gemma3 270M) to hundreds of billions (qwen3 235B). The bigger the model, the more memory it needs, and the better it tends to be at demanding tasks.
Installing on Linux and macOS is one line: curl -fsSL https://ollama.com/install.sh | sh. Then the command ollama run gemma4 downloads the model and opens a chat in the terminal. In a company nobody will work in a terminal, though. You will use the API, covered below.
How much memory and what hardware does a local LLM need?
The main rule is simple: the model has to fit into the memory of the graphics card (VRAM). If it doesn't, Ollama leaves part of the calculation to the processor and answers arrive more slowly. The command ollama ps shows how the model is split, for example 48% CPU and 52% GPU. Ollama supports NVIDIA cards (compute capability 5.0 and above), AMD Radeon through ROCm and Apple through Metal.
The table lists download sizes straight from ollama.com. Add headroom for context: the default window is 4,096 tokens, and memory grows with its length and with the number of parallel requests (OLLAMA_NUM_PARALLEL, default 1). For reference, the RTX 4090 has 24 GB of VRAM and the RTX 5090 has 32 GB.
| Model | Download | Licence | Where it fits |
|---|---|---|---|
| llama3.1:8b | 4.9 GB | Llama 3.1 Community License | a card with 8 GB VRAM |
| gemma4:12b | 7.7 to 8.0 GB | Apache 2.0 | a card with 12 GB VRAM |
| qwen3:14b | 9.3 GB | Apache 2.0 | a card with 12 to 16 GB VRAM |
| gpt-oss:20b | 14 GB | Apache 2.0 | 16 GB of memory (figure from ollama.com) |
| llama3.1:70b | 43 GB | Llama 3.1 Community License | 48 GB or more, typically two cards |
| gpt-oss:120b | 65 GB | Apache 2.0 | a single 80 GB GPU (figure from ollama.com) |
The Llama 3.1 licence only requires a separate agreement with Meta above 700 million monthly users, so small and mid-sized companies are not affected. Apache 2.0 models can be used without such conditions. Always check the licence of the specific model before you put it into production.
Which open model fits which task?
- Summaries, e-mails, short answers: models of roughly 8 to 14 billion parameters are enough, for example qwen3:14b or gemma4:12b. They run on one ordinary card.
- Czech: Llama 3.1 officially supports eight languages (English, German, French, Italian, Portuguese, Hindi, Spanish and Thai), and Czech is not among them. Qwen3 claims more than 100 languages and Gemma 4 claims more than 35 languages with training on more than 140. Test Czech quality on your own texts anyway.
- Searching documents: you also need an embedding model, such as nomic-embed-text, mxbai-embed-large or bge-m3 (1.2 GB, more than 100 languages). It is small and runs next to the main model.
- Harder reasoning and longer analyses: larger models such as gpt-oss:120b or llama3.1:70b have more capacity, but need 65 and 43 GB of memory respectively.
How do you run local AI over your own company data?
A model knows nothing about your company by itself. To answer from your documents you use a pattern called RAG (retrieval augmented generation), which Ollama describes in its embeddings guide.
- Install Ollama on a server in your network and check that the model runs on the GPU (ollama ps).
- Download a generative model and an embedding model with ollama pull.
- Split documents into chunks, convert them to vectors and store them in a vector database (Ollama's example uses Chroma).
- When a question comes in, find the nearest chunks and send them to the model together with the question.
- Applications call the API. Ollama offers an OpenAI-compatible interface at /v1/ (chat completions and embeddings), so existing tools often only need to point at your server.
Watch the permissions. A simple RAG setup doesn't know who is asking and will return a passage from a document the asker is not allowed to see. The permission filter has to run before the passages reach the model.
How do you keep data from leaving your network?
- API address: by default Ollama listens only on 127.0.0.1, port 11434. If you set OLLAMA_HOST to, say, 0.0.0.0, you open it to the whole network. Put a firewall and user authentication in front of it, for example through a reverse proxy.
- Ollama cloud features: Ollama also offers cloud models. You turn them off with the variable OLLAMA_NO_CLOUD=1. In purely local use, the vendor's FAQ says it does not see your prompts or data.
- Cutting the internet: once the models are downloaded, block outbound traffic from the server at the firewall and check that everything still works. Then you have proof, not just a promise.
- Logs and backups: conversations and the vector database contain company data. Back them up and control access the way you control the documents they came from.
How does a local model differ from ChatGPT and Claude?
Models that fit on one server are smaller than the best cloud models. You notice it in complex reasoning, long documents and writing in Czech. For clearly defined jobs (summaries, sorting, answers from internal documentation) they are often enough. We recommend taking 20 real tasks from your company and running them on the local model and in the cloud. You will see the difference better than from any leaderboard.
Local operation has an operations side too. By default Ollama processes one request per model at a time while the others wait in a queue, and a model is unloaded from memory after 5 minutes of inactivity. Tuning, updates and monitoring are on you. On the other hand you don't pay per token and costs are predictable.
Is local AI GDPR compliant?
Running locally does not satisfy the GDPR for you. You still need a legal basis, information for people, data minimisation and security under Art. 32. The advantage lies elsewhere: you do not pass personal data to another processor (Art. 28) or to a third country (Chapter V).
In December 2024 the European Data Protection Board (EDPB) said in Opinion 28/2024 that legitimate interest is not a default legal basis for AI and always has to be assessed with a test. The technical side of data protection is in the article GDPR and data security; for the AI regulation see the AI Act page. What an AI agent does in a company is described in What is an AI agent.
Where does xOffice fit in?
xOffice is the Lumenax ERP system built on open source Odoo 19. The AI assistant in it is optional and you have four choices: none, your own provider account (for example Anthropic or OpenAI, which you pay directly), a model on our infrastructure in the EU from 199 EUR a month, or an AI server in your own network from 5,999 EUR one-off plus installation. The last option is exactly what this article is about, just ready made.
The assistant answers in Czech from your data, sees only what the asking user may see, every data change has to be confirmed by a person, and a monthly budget applies per user and per company. See the xOffice page or try the demo (after entering a code from an e-mail).
Frequently asked questions
Is Ollama free?
Yes, the program is open source under the MIT licence. You pay for hardware, electricity and administration. Licences of individual models differ, so check each model.
Do I need a graphics card for Ollama?
You don't. Ollama can keep a model in ordinary memory and compute on the processor, but answers are slower. For several users a graphics card that holds the whole model is practical.
Does Ollama send my data to ollama.com?
Not when running locally; the vendor's FAQ says it does not see your prompts or data. Ollama's cloud features can be turned off with OLLAMA_NO_CLOUD=1. Verify the setup by watching outbound traffic.
Which model is best for Czech?
We don't have a single answer. Vendors state different numbers of supported languages (Llama 3.1 has eight and Czech is not among them). Try qwen3, gemma4 and gpt-oss on your own texts.
Can Ollama connect to applications that support OpenAI?
Often yes. Ollama offers an OpenAI-compatible interface at /v1/ including chat completions and embeddings. You only change the server address. The key is entered as a formality; the local server ignores it.
