Ollama is a convenient way to run open models locally for development, private experiments and small internal tools. It is not a complete production platform: you still need access control, monitoring, capacity planning and a fallback.
Key takeaways
- Ollama runs a local server and exposes an HTTP API; the default port is 11434.
- Model size and quantisation determine memory use and speed.
- Local models are useful for privacy-sensitive experiments and offline development.
- Measure quality on a fixed evaluation set before you optimise the prompt.
What matters in practice
- Production needs authentication, rate limits, logging and a capacity plan.
- Quality varies by model and task; evaluate before replacing a hosted API.
How to implement it
- Install Ollama. Run the installer or container and confirm the API is reachable.
- Pull a model. Choose the smallest model that passes your task evaluation.
- Test the API. Call
/api/generateor/api/chatfrom a server route. - Add a timeout and limit. Local inference can be slow on CPU; cap output and concurrent requests.
- Protect the endpoint. Do not expose port 11434 to the public internet.
- Plan production. Use a GPU host, a queue, monitoring and a hosted fallback.
A working example
const response = await fetch("http://127.0.0.1:11434/api/chat", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model: "llama3.1:8b",
messages: [{ role: "user", content: "Summarise this ticket." }],
stream: false,
}),
});
Cost, latency and reliability
| Concern | What drives it | What to do |
|---|---|---|
| Speed | CPU vs GPU, model size | Use a smaller model or a GPU; stream on CPU |
| Memory | Parameters and context | Quantise and cap context |
| Quality | Model choice | Evaluate on your task; keep a hosted fallback |
| Security | Local API has no auth by default | Bind to localhost and put auth in front |
| Operations | Manual updates and monitoring | Containerise, log and version the model |
Security and privacy
- Keep provider keys on the server and scope them to the minimum permissions.
- Treat retrieved documents and user input as untrusted; separate instructions from data.
- Redact sensitive fields before sending them to a third-party model.
- Log prompts and outputs only as long as you need them, with access controls.
- Provide a human escalation path for high-impact or low-confidence answers.
Common mistakes
- Exposing the Ollama port to the internet.
- Using a 70B model on a CPU and expecting interactive latency.
- Assuming a local model is automatically private if the host is shared.
- Replacing a hosted API without comparing quality on real tasks.
How to evaluate it
- Build a fixed set. Real questions, ambiguous cases, no-answer cases and adversarial prompts.
- Score the output. Correctness, citation quality, refusal behaviour, latency and cost.
- Review failures. Group them by retrieval, prompt, model or product gap.
- Version changes. Record the prompt, model and retrieval settings with each release.
Evaluation rubric
| Dimension | Pass condition | How to test |
|---|---|---|
| Correctness | Matches the source or is clearly marked as uncertain | Human review against a reference answer |
| Grounding | Every factual claim has a source | Citation coverage and spot checks |
| Refusal | Declines when the context is insufficient | No-answer and adversarial cases |
| Safety | Does not follow injected instructions | Prompt-injection test set |
| Latency | Within the product's interaction budget | p95 on a production-like request |
| Cost | Within the per-task budget | Tokens, retrieval calls and retries |
Rollout plan
- Internal only. Test with the team and a fixed evaluation set; log every failure.
- Small pilot. Expose it to a narrow user group with a visible feedback control.
- Human in the loop. Require review for high-impact answers or actions.
- Measure. Compare task completion, correction rate, latency and cost with the baseline.
- Expand or stop. Expand only if the rubric passes; keep a kill switch and a fallback.
Data governance
| Question | What to decide | Why |
|---|---|---|
| What data is sent? | Minimum fields, redaction and retention | Third-party processing and privacy |
| Where is it stored? | Logs, embeddings and evaluation sets | Access and deletion |
| Who can see it? | Roles, support and vendor access | Least privilege |
| How long is it kept? | Prompts, outputs and feedback | Retention policy |
| How is it deleted? | User, tenant and backup paths | Privacy requests |
| What is the fallback? | Search, form or human handoff | The model will fail sometimes |
Frequently asked questions
Yes, for small internal tools or with a proper GPU host and operations. For public, high-traffic products, a managed inference service is usually simpler.
It provides an OpenAI-compatible endpoint for some use cases. Check the current docs and test the features you need, especially tools and structured output.
It depends on the model size and quantisation. A rough rule is model parameters × bytes per parameter plus context overhead. Measure the actual model on your hardware.
Ollama is primarily for running models, not a full training platform. Use a training workflow and then serve the resulting model if needed.
Sources and further reading
- Ollama documentation (docs.ollama.com)
- OWASP Top 10 for LLM Applications (genai.owasp.org)
Related reading
Planning an AI feature?
Describe the user, the task and the data. I will tell you whether a prompt, RAG or a deterministic workflow is the right starting point.
How to cite this guide: Nur Hossain Repon, “Ollama local LLM in Next.js 2026”, Nhrepon Portfolio, reviewed October 2026, https://nhrepon.com/posts/ollama-local-llm-in-next-js-2026.





Comments
No comments yet.