A useful AI chatbot is a narrow product feature, not a chat box bolted onto a website. Start with one job, a known knowledge source and a human escalation path; then add streaming, retrieval and evaluation.
Key takeaways
- Keep the OpenAI key on the server; the browser calls your route, not the model provider.
- Stream tokens so the interface feels responsive, but keep the final answer and citations in your own database.
- Retrieve from your own content before generating; a general model does not know your prices or policies.
- Measure quality on a fixed evaluation set before you optimise the prompt.
What matters in practice
- Add rate limits, input limits and a content policy before exposing the endpoint.
- Log prompts, retrieved passages, latency and user feedback so you can evaluate quality.
How to implement it
- Define the job. One user, one task and one success measure. “Answer pre-sales questions from the service pages” is a job; “be an AI assistant” is not.
- Build the retrieval layer. Chunk your content, create embeddings and store them with source metadata.
- Create a server route. Validate the request, retrieve context, call the model and stream the answer.
- Add guardrails. Limit length, block prompt injection patterns and require a citation for factual claims.
- Design escalation. Offer a contact form or human handoff when confidence is low.
- Evaluate. Use a fixed question set, score correctness and citation quality, and review failures weekly.
A working example
import { streamText } from "ai";
import { openai } from "@ai-sdk/openai";
export async function POST(request: Request) {
const { messages } = await request.json();
const context = await retrieve(messages.at(-1)?.content ?? "");
const result = streamText({
model: openai("gpt-4.1-mini"),
system: `Answer only from the context below. If the answer is not present, say so and offer a human handoff.\n\n${context}`,
messages,
maxOutputTokens: 600,
});
return result.toTextStreamResponse();
}
Cost, latency and reliability
| Concern | What drives it | What to do |
|---|---|---|
| Cost | Tokens in + tokens out + embeddings | Cache retrieved context, cap output and route simple queries to a smaller model |
| Latency | Retrieval + model + streaming | Stream, reduce context and run retrieval and model warm-up in parallel |
| Quality | Grounded answers with citations | Evaluate against a fixed set, not vibes |
| Safety | Prompt injection, PII and abuse | Input limits, allowlists, redaction and rate limiting |
| Reliability | Provider outages and rate limits | Timeouts, fallback model and a clear error state |
Security and privacy
- Keep provider keys on the server and scope them to the minimum permissions.
- Treat retrieved documents and user input as untrusted; separate instructions from data.
- Redact sensitive fields before sending them to a third-party model.
- Log prompts and outputs only as long as you need them, with access controls.
- Provide a human escalation path for high-impact or low-confidence answers.
Common mistakes
- Sending the whole website into every prompt.
- Letting the model answer questions about prices or policies without a source.
- Exposing the API key in client-side code.
- Launching without a human escalation path or a feedback loop.
How to evaluate it
- Build a fixed set. Real questions, ambiguous cases, no-answer cases and adversarial prompts.
- Score the output. Correctness, citation quality, refusal behaviour, latency and cost.
- Review failures. Group them by retrieval, prompt, model or product gap.
- Version changes. Record the prompt, model and retrieval settings with each release.
Evaluation rubric
| Dimension | Pass condition | How to test |
|---|---|---|
| Correctness | Matches the source or is clearly marked as uncertain | Human review against a reference answer |
| Grounding | Every factual claim has a source | Citation coverage and spot checks |
| Refusal | Declines when the context is insufficient | No-answer and adversarial cases |
| Safety | Does not follow injected instructions | Prompt-injection test set |
| Latency | Within the product's interaction budget | p95 on a production-like request |
| Cost | Within the per-task budget | Tokens, retrieval calls and retries |
Rollout plan
- Internal only. Test with the team and a fixed evaluation set; log every failure.
- Small pilot. Expose it to a narrow user group with a visible feedback control.
- Human in the loop. Require review for high-impact answers or actions.
- Measure. Compare task completion, correction rate, latency and cost with the baseline.
- Expand or stop. Expand only if the rubric passes; keep a kill switch and a fallback.
Data governance
| Question | What to decide | Why |
|---|---|---|
| What data is sent? | Minimum fields, redaction and retention | Third-party processing and privacy |
| Where is it stored? | Logs, embeddings and evaluation sets | Access and deletion |
| Who can see it? | Roles, support and vendor access | Least privilege |
| How long is it kept? | Prompts, outputs and feedback | Retention policy |
| How is it deleted? | User, tenant and backup paths | Privacy requests |
| What is the fallback? | Search, form or human handoff | The model will fail sometimes |
Frequently asked questions
It depends on traffic, context size and model choice. Estimate tokens per conversation, multiply by expected conversations and add embeddings and infrastructure. A small support bot can be inexpensive; a high-traffic agent with long context is not.
You can redact or minimise data, choose a provider with the right terms and avoid sending sensitive fields. Review the provider's data-processing terms and your own privacy notice.
Ground the answer in retrieved sources, require citations, add a refusal path and evaluate the questions where the model is most likely to guess. No prompt fully eliminates hallucinations.
Not for a small knowledge base. Postgres with pgvector is often enough. Add a dedicated vector service when scale, filtering or latency requires it.
Sources and further reading
- OpenAI API documentation (platform.openai.com)
- Vercel AI SDK documentation (ai-sdk.dev)
- Supabase pgvector guide (supabase.com)
- OWASP Top 10 for LLM Applications (genai.owasp.org)
Related reading
Planning an AI feature?
Describe the user, the task and the data. I will tell you whether a prompt, RAG or a deterministic workflow is the right starting point.
How to cite this guide: Nur Hossain Repon, “How to ship an AI chatbot with Next.js OpenAI”, Nhrepon Portfolio, reviewed October 2026, https://nhrepon.com/posts/how-to-ship-an-ai-chatbot-with-next-js-openai.





Comments
No comments yet.