Building a local AI assistant no longer requires a complex machine-learning platform. With Ollama to run models and Open WebUI to provide the browser experience, you can create a private assistant that works on an XLC dedicated server. This guide uses the current Ollama-compatible Gemma 4 and Qwen3 model families and the current Open WebUI Docker workflow. The software stack can be used without a per-message API fee, although the dedicated server, storage, bandwidth, and administration still have a cost. You will go from server selection to a working chat interface, then add local documents, security controls, and an update plan so the result is useful beyond a quick demo.
What you will build
- Ollama will run the local language model and expose a local API; Open WebUI will provide chats, file uploads, model selection, and administration.
- An XLC dedicated server gives you single-tenant CPU, memory, NVMe, network, and optional GPU choices that can be matched to the model and expected traffic.
- Gemma 4 E4B or 12B is a practical multimodal starting point; Qwen3 8B or 14B is a strong multilingual and reasoning-oriented option.
- Larger Qwen3 30B or Gemma 4 26B/31B models need materially more memory and should be selected only after checking the current model tag and hardware capacity.
- Private documents are best added through retrieval-augmented generation, with careful embedding, access control, and evaluation rather than blindly fine-tuning the model.
- Free local inference removes per-request API charges, but reliable production use still requires server, security, backup, monitoring, and maintenance planning.
Choose the right XLC dedicated server
Start with the workload, not a model name. A single-user assistant that summarizes notes can run comfortably on a smaller CPU or GPU configuration, while a team assistant with long documents, several simultaneous conversations, tool calling, and background jobs needs more headroom. CPU inference is simple and can be cost-effective for smaller models; a compatible GPU can reduce response latency and improve concurrency. If you choose a GPU, confirm the current NVIDIA CUDA or AMD ROCm support and driver combination for the exact hardware before deployment.
RAM is often the first constraint because the model file is only part of the working set. The runtime also needs space for the context window, key-value cache, embeddings, the operating system, Open WebUI, and other services. As a planning guide, 16GB of system memory suits smaller E4B or 8B experiments, 32GB or more gives 12B and 14B models better headroom, and 64GB or more is more comfortable for 30B-class models, longer contexts, or multiple users. These are practical starting points rather than hard requirements: actual performance depends on quantization, context length, concurrency, and GPU offload.
Use fast NVMe storage for model files, the Open WebUI data volume, document indexes, logs, and backups. Leave free capacity for new model tags and temporary downloads. Choose the server location according to where users and source systems are located; XLC availability in Los Angeles, Tokyo, and Hong Kong can help reduce application round trips for different regions. Also decide whether the assistant will be private to a team or exposed through a public application, because that choice changes firewall rules, bandwidth planning, and the need for a reverse proxy.
Tip: Reserve headroom for the context window, document index, operating-system updates, and at least one additional model before ordering the server.
Install Ubuntu and Ollama
Use a current supported Ubuntu LTS image, then apply updates before installing the AI stack. Create a named administrative account, use SSH keys, and disable password login after confirming that key-based access works. On a new server, a typical first pass is sudo apt update && sudo apt full-upgrade -y followed by sudo apt install -y curl ca-certificates. Reboot if the kernel or a driver was updated, and record the operating-system, GPU-driver, Docker, and Ollama versions so later changes are easy to trace.
The current Ollama Linux installation method is the official installer. Run curl -fsSL https://ollama.com/install.sh | sh, then verify the binary with ollama --version. The installer is convenient because it follows Ollama’s current package path rather than requiring an old tutorial to be updated by hand. For controlled environments, review the installer and pin a specific Ollama release when your validation process requires it. GPU hosts should also confirm that the required vendor driver is visible before pulling a large model.
Make Ollama start after a reboot and confirm that its local API is healthy. If the installer created the service, use sudo systemctl enable --now ollama and sudo systemctl status ollama; otherwise start it with ollama serve while testing. A simple check is curl http://127.0.0.1:11434/api/tags. A response showing the model list means the service is reachable on the host. Keep port 11434 private: Open WebUI can reach it over the local host path, so there is usually no reason to expose the Ollama API directly to the public internet.
Pull one small model first, then add a stronger model after the basic path works. Current Ollama tags to evaluate include gemma4:e4b, gemma4:12b, qwen3:8b, qwen3:14b, and qwen3:30b. Run commands such as ollama pull gemma4:e4b and ollama pull qwen3:8b, then use ollama list and ollama show qwen3:8b to confirm what is installed. Model tags, quantizations, and sizes can change, so check the current Ollama library entry before committing storage or RAM. Pulling a model downloads its weights to the server; plan the download time and disk space before starting a production migration.
Tip: Download one primary model first, verify the complete chat path, and add larger models only after checking disk space, RAM, and response latency.
Run Open WebUI with Docker
Docker keeps the browser interface and its data volume separate from the Ollama service. Generate a persistent secret with openssl rand -hex 32, replace your-secret-key in the following current Open WebUI quick-start command, and run it on the XLC server: docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data -e WEBUI_SECRET_KEY=your-secret-key --name open-webui --restart always ghcr.io/open-webui/open-webui:main. Keep the secret safe and reuse the same value after recreating the container; otherwise users can be logged out. The named volume stores chats, users, and settings across image updates.
When the container is running, open http://server-ip:3000 for the first setup. The first account becomes the administrator, so use a strong password and store it in the team’s approved password manager. In Open WebUI, go to the administrator connections area and confirm the Ollama connection. When Ollama is on the same host, the host mapping in the command lets the container reach http://host.docker.internal:11434; when Ollama is on a different private server, configure OLLAMA_BASE_URL for that endpoint instead. Start a new chat, choose the pulled model, and send a short test prompt before adding documents.
Do not treat port 3000 as a finished public deployment. Put Open WebUI behind a reverse proxy with HTTPS, keep 11434 closed to the internet, and allow only the ports and source addresses the application needs. If the reverse proxy is on the same server, bind the container privately by replacing -p 3000:8080 with -p 127.0.0.1:3000:8080. Configure the proxy to pass WebSocket connections, set timeouts appropriate for model responses, and renew certificates automatically. Apply host firewall rules such as allowing SSH only from trusted addresses and exposing 80/443 only when the service really needs public web access.
Open WebUI is the interaction layer, not a model provider. It gives users a browser chat, model selection, file workflows, user administration, and a persistent data volume while Ollama performs the local inference. That separation makes upgrades easier to reason about: you can update the UI container without moving model files, or update a model while keeping the chat database. For a small internal assistant, begin with one or two approved models and a simple system prompt. Add extra models only after measuring their latency, memory use, and answer quality on real tasks.
XLC dedicated servers help you keep model execution, documents, and application services on predictable single-tenant resources
- Gemma 4 E4B keeps a first CPU or small-GPU deployment responsive for multimodal prompts
- Gemma 4 12B is a stronger general assistant when the host has more memory
- Qwen3 8B or 14B is a good fit for multilingual prompts, coding, and reasoning
- Qwen3 30B or Gemma 4 26B/31B should be used only after validating memory, latency, and concurrency
Do not choose the largest model simply because it is available. A smaller quantized model with enough context and fast responses can be more useful than a larger model that swaps to disk or leaves no headroom for retrieval and concurrent users. Benchmark the prompts that matter: document questions, bilingual writing, structured extraction, coding, tool calls, and long-context summarization. Record first-token latency, tokens per second, error rate, context length, and memory use with ollama ps. Then choose the smallest server and model combination that meets the service target, leaving capacity for operating-system updates, indexes, and temporary files.
Tip: Use a small fast model for titles, classification, and routine questions, and reserve a larger reasoning model for tasks that justify the extra latency.
Add private documents with retrieval-augmented generation
Retrieval-augmented generation, or RAG, lets the assistant search relevant passages from your documents and place those passages in the prompt before the model answers. It does not retrain the model and it does not make every document permanently known. That distinction is useful for policies, product manuals, project notes, and support material that changes over time. A local model can then answer from a controlled knowledge base while the source files and vector index remain on the XLC server. RAG is usually the quickest way to make a general assistant useful for a specific team.
Prepare the source material before uploading it. Remove duplicates, old versions, hidden secrets, and pages that users should not see; use clear filenames and preserve headings, tables, and dates. In Open WebUI, start with a small Knowledge collection or attach a file to a test chat, then verify that the retrieved passages actually answer representative questions. If you want the whole workflow to remain local, select a local embedding engine and confirm where its embedding model is stored. The current Open WebUI workflow can fetch embedding assets on first use, so an offline or air-gapped deployment must prepare those assets or configure Ollama as the embedding engine before ingesting documents.
Treat document access as seriously as model access. Keep separate knowledge bases for teams with different permissions, do not upload passwords or API keys, and make sure the Open WebUI account that can search a collection is the account that should see its contents. Back up the Open WebUI data volume and the source documents, but test restoring them on a separate instance. A local server improves control over the data path, yet it does not automatically provide encryption, identity management, retention rules, or protection from an administrator with excessive permissions. Those controls still belong in the deployment design.
Create a small evaluation set before calling the assistant production-ready. Ask questions whose answers are known, include questions that should produce “I do not know,” and test conflicting or outdated documents. Inspect the retrieved passages or citations when available, not just the final prose. If the right passage is missing, improve chunking, metadata, document quality, or the embedding choice before increasing the language model size. If the passage is present but the answer is wrong, adjust the system prompt, context budget, or model settings and rerun the same test set.
Make the assistant useful, not just local
Start with a clear system prompt that defines the assistant’s role, supported languages, response format, source-grounding rules, and escalation behavior. Tell it to distinguish retrieved facts from general knowledge, state when evidence is missing, and ask a focused clarifying question instead of inventing a detail. For bilingual teams, specify whether the answer should follow the user’s language or use a fixed language for technical terms. In the Ollama CLI, Qwen3 can switch its reasoning behavior with /set think and /set nothink; use thinking for complex planning only when the additional latency and token use are worthwhile.
Add tools gradually. Open WebUI supports tool-calling and community functions, but a local assistant should begin with read-only, allow-listed actions such as searching an approved knowledge base or looking up a status page. Use separate service accounts, least-privilege tokens, input validation, timeouts, and logs. Do not give a general chat model unrestricted shell access or permission to send external messages without a human approval step. Test prompt-injection attempts in documents and tool results, because retrieved text can contain instructions that are not trustworthy.
Operational discipline keeps a local assistant predictable. Monitor CPU, GPU memory, system RAM, NVMe capacity, request latency, container health, and error logs. Use docker logs -f open-webui for the UI and journalctl -u ollama for the service when diagnosing a failed request. Keep a tested backup of the Open WebUI volume, model inventory, knowledge-base source files, reverse-proxy configuration, and system prompt. The quick-start :main image is a rolling channel; for production, pin a tested Open WebUI release tag such as vX.Y.Z, test upgrades with a separate volume, and record every model tag and configuration change. Update Ollama through its current installer or a pinned release process after validation.
Conclusion
A free local AI assistant is a software-and-infrastructure pattern, not a single download. Ollama provides a practical local model runtime, Open WebUI provides the browser and user layer, and current Gemma 4 and Qwen3 tags give you several model sizes to evaluate. An XLC dedicated server adds a single-tenant foundation where model files, document indexes, application services, and monitoring can be planned together. Start with a right-sized model, keep the Ollama API private, secure Open WebUI behind HTTPS, validate RAG with real questions, and pin production updates after testing.
For teams that want a private local AI assistant, XLC dedicated servers provide a practical single-tenant foundation for Ollama, Open WebUI, model files, private documents, and the APIs around them. Start with a right-sized model, keep the first deployment secure, and scale only after measuring real conversations.
