AI Engineer · Project 5

Hardened LLM Gateway

Cloudflare for LLM APIs. One endpoint, all providers, automatic caching, instant rollback.

Duration: ~12 daysDeliverable: A production-hardened service repo. Live URL with auth. Chaos test report. Production architecture write-up. 4 checkpoint PRs reviewed.

Build the production-grade LLM gateway that indie AI builders integrate as their LLM proxy. Flagship pitch: a multi-provider gateway with built-in semantic caching (50%+ cost reduction), prompt injection defense, gradual prompt rollout, and circuit-breaker fallback when a provider goes down. Real market: Helicone, Portkey, Vellum are growing fast in this exact space. The portfolio outcome: 5+ real indie AI projects integrating with your gateway as their LLM proxy. Or take the variation route — harden your Project 2 or Project 4 as the user-facing product itself, with all of the same hardening built in.

The product in detail

The product is the production layer that sits between an AI app and its model providers. The user changes one URL in their codebase and their LLM calls now route through your gateway: multi-provider failover (Anthropic → OpenAI → Cohere if one goes down), semantic caching (50%+ cost reduction on semantically similar queries), per-user rate limits, prompt injection defense, content moderation, audit logs, gradual prompt rollout (10% → 50% → 100%), and instant rollback if a metric drops. One endpoint, one bill, one observability pane.

The target user is an indie AI builder or small-startup engineer who ships AI features fast and doesn't have time to build production infrastructure from scratch. They've been burned at least once — a hallucination they couldn't trace, an API outage that killed a live demo, or a $4,000 bill from a prompt injection that ran the model in a loop. They want one endpoint that handles all of this. They'd pay $50/month flat to stop worrying about production AI plumbing.

Real market: Helicone (YC, $4M raised, 10,000+ developers), Portkey ($3M raised, 4,000+ developers), Vellum ($5M Series A, enterprise-focused), LiteLLM (OSS, 8,000+ GitHub stars). The LLM gateway space is growing fast but indie builders are still mostly rolling their own with ad-hoc scripts. Your gateway is positioned as the "one-weekend setup, scales when you need it" version — the Cloudflare analogy for LLM APIs.

Technically you build: input guardrails (prompt injection detection, PII redaction, token budget cutoffs), output guardrails (schema validation via Pydantic, content moderation, refusal detection), a semantic cache layer (embed queries, hit on cosine similarity > 0.95 with cost / hit-rate dashboarding), Anthropic prompt caching for system prompts, model routing (simple queries → cheap model, complex → strong model, verified by evals), structured logging with trace_id / user_id / model / tokens / cost / latency per request, circuit breakers with cached-response fallback, and A/B test infrastructure with auto-rollback on metric regression.

What you ship: a hardened service with a live URL, public auth, 5+ real projects integrating it as their proxy. A chaos-test report proving the service survives provider outages (LLM API forced unavailable → fallback works). A senior-engineer-grade architecture write-up documenting every tradeoff (cost of caching vs latency of cache miss, quality drop from model routing vs cost savings). When the interview is for a senior or staff AI Engineer role, this is the project that signals production seniority — you've shipped infrastructure, not just features.

Pick what you'll build

This is one project with one skill graph — but you pick the product framing that fits your interests or career goals. Each option below is a viable portfolio piece. We'll narrow it together in the kickoff call.

  • LLM gateway for indie AI builders — one endpoint, all providers, caching + observability built in (audience: 10,000+ indie AI builders)
  • Harden Project 2's agent for paying users (auth, rate limits, full guardrails, audit logs)
  • Harden Project 4's research tool for an enterprise pilot (PII redaction, SOC2-ready logging, audit trail)
  • Production-ize an open-source AI demo someone else built — turn a viral GitHub demo into a real service
  • Internal AI service for a real team (Slack bot, internal Q&A) that an org actually depends on
Why this project
  • Picks up the under-covered Theme 5 (Production AI Engineering — only 20% of posts but heavy interview emphasis on cost optimization, guardrails, semantic caching).
  • After Project 3 instrumented things, this project hardens them.
  • Bucket B (non-AI-native companies) cite 'production AI' most heavily — this project mirrors enterprise engineering reality.

This project leans toward enterprise AI platform engineering — roles like Capital One's 'Senior Lead AI Engineer (Gen AI Platform Services)' or Notion's productionization track.

Research backing this project

Every project on projeckt.ai is calibrated against real job posts and real interview questions, not opinion. Here's the data behind this one.

9
Job posts citing concepts in this project
22
Interview questions this project prepares you for
4
Tacit must-knows drilled — concepts interviewers test but JDs rarely list
Tacit must-knows you'll drill
  • Cost Optimization (Δ+8)
  • Guardrails (Δ+6)
  • Semantic caching (interview-heavy)
  • Model routing

Themes covered

Each chip shows how many of the 46 analysed job posts cite this theme.

LLM & Prompt Foundations · 43 postsEvals, Observability & Safety · 24 postsProduction AI Engineering · 9 postsGuardrails (tacit must-know) · 2 posts

Justified by real job posts

Verbatim quotes from the postings that drove the design of this project.

Deploy LLMs and other Machine Learning models into production, ensuring high availability and performance

MoneyLion — LLM Engineer

Develop caching layers to optimize GenAI application performance

MoneyLion — LLM Engineer

Productionize and scale asynchronous workflows across Notion's infrastructure

Notion — Software Engineer, AI Workflows

Own scoped productionization projects: integrate new models/techniques, add monitoring and guardrails, or improve latency/cost/reliability

Notion — Software Engineer, New Grad (AI)

What you'll be able to do

  • 1Implement input guardrails: PII redaction, prompt injection detection, token-budget cutoffs
  • 2Implement output guardrails: schema validation, content moderation, refusal-pattern detection
  • 3Build a semantic cache (embed-and-store responses, hit on cosine similarity above threshold) with hit-rate metrics
  • 4Use Anthropic prompt caching or OpenAI response cache to reduce repeated-context cost
  • 5Implement model routing: cheap model for simple queries, strong model for complex — verified by evals
  • 6Add rate limiting per user / per IP with Redis or similar
  • 7Structured logging — every request has trace_id, user_id, model used, tokens, cost, latency, output status
  • 8Circuit breakers around external dependencies; fallback behaviors when LLM API is down
  • 9A/B test prompts in production with gradual rollout (10% / 50% / 100%) and automatic rollback on metric regression
  • 10Articulate the latency / cost / quality / reliability tradeoff for at least 5 architectural decisions

Tools you'll learn

Hands-on with these tools across the paid and open-source stacks. The Paid / OSS / Both label shows which stack each tool belongs to.

Anthropic prompt caching
Anthropic prompt caching
Paid stack
HE
Helicone (LLM proxy)
Paid stack
PO
Portkey (LLM gateway)
Paid stack
SE
Semantic Router
OSS stack
NE
NeMo Guardrails
OSS stack
Llama Guard 3
Llama Guard 3
OSS stack
OpenAI Moderation API
OpenAI Moderation API
Paid stack
LangSmith (live tracing)
LangSmith (live tracing)
Paid stack
LA
LangFuse (live tracing)
OSS stack

Week-by-week checkpoints

Each checkpoint is a Pull Request to your public GitHub repo. Rohan reviews on the PR; merge = checkpoint complete.

CP1: Guardrails pipeline
Days 1–3
What ships

Input pipeline: prompt injection detection (NeMo or LLM-based classifier), PII redaction. Output pipeline: schema validation, content moderation, max-token enforcement. 5 adversarial test cases pass through unscathed.

Acceptance criteria

Adversarial suite (10+ injection attempts, 5 toxic outputs) — guardrails block all; legitimate traffic unaffected.

New skills
  • Prompt injection defense
  • PII redaction patterns
  • Moderation pipelines
  • Refusal-pattern detection
CP2: Caching layer
Days 4–5
What ships

Semantic cache (embed query, hit if cosine sim > 0.95). Anthropic prompt caching enabled for system prompts. Dashboard shows cache hit rate + cost savings.

Acceptance criteria

Cache hit rate ≥30% on benchmark traffic; cost dashboard shows measurable reduction vs baseline.

New skills
  • Semantic caching
  • Prompt caching
  • Cost attribution per request type
CP3: Model routing + A/B infrastructure
Days 6–9
What ships

Classifier (regex + LLM) routes queries to cheap vs strong model. Project 3's eval harness verifies quality didn't drop. A/B test infra deployed — 10% traffic to new prompt, auto-rollback if Faithfulness drops.

Acceptance criteria

Routing in production; cost ↓ measurably without quality drop; A/B harness runs at least one ramp + auto-rollback in dry-run.

New skills
  • Model routing classifiers
  • Gradual rollout patterns
  • Auto-rollback on metric regression
CP4: Reliability + production write-up
Days 10–12
What ships

Circuit breakers around LLM API + vector DB. Fallback behaviors (cached response if LLM down). Structured logs with trace_id. 4-min Loom walks through architecture, guardrails, caching, routing. Production-grade README.

Acceptance criteria

Service survives chaos test (LLM API forced unavailable → fallback works); demo shows full request lifecycle.

New skills
  • Circuit breakers
  • Graceful degradation patterns
  • Production architecture documentation
  • Senior-interview demo craft

Pick your stack

Same skills, different credentials. Start free with the OSS stack or use the paid stack to learn directly on production tools.

ToolPurposeLicenseHardwareMarket equivalent
Ollama + Qwen 2.5 + DeepSeek R1 (model router)LLM mix for routingVarious OSSMac M-series 16GB+ or Linux GPUClaude + GPT-4o router
NeMo Guardrails + Llama Guard 3Guardrails pipelineApache-2.0 / Llama CommunityRuns locallyGuardrails AI + OpenAI Mod
Redis self-hostedCaching + rate limitBSDDockerUpstash
Glitchtip (or self-hosted Sentry)Error monitoringMIT (Glitchtip)DockerSentry hosted
Coolify on VPSDeployApache-2.0$5/mo VPSVercel / Railway

Estimated cost: $5/month (VPS only).

Interview questions you'll be ready for

Real questions from 13 sources, mapped to the skills this project builds.

What are some challenges associated with deploying LLMs in production?
Source: DataCamp — Top 36 LLM Interview Questions (2026)
Prepares you for: You've lived all of them — latency, cost, hallucination, injection, observability, versioning.
How do you handle prompt injection in production AI systems?
Source: LockedIn AI — AI Engineer Interview Prep Top 40 (2026)
Prepares you for: CP1 is the implementation.
What is semantic caching and how can it reduce cost and latency?
Source: Medium (Adil Shamim) — 100+ Real Interviews
Prepares you for: CP2 is the implementation.
Your LLM keeps ignoring your instructions. How do you make it follow structured output formats?
Source: GitHub — amitshekhariitbhu/ai-engineering-interview-questions
Prepares you for: Schema validation in CP1 is the answer.
How would you design the UX for an AI assistant that is often slow?
Source: Medium (Adil Shamim) — 100+ Real Interviews
Prepares you for: CP4's graceful degradation patterns address this.

Ready to start building?

Book a free 30-minute project planning session. We'll tailor this project to your stack, team, and goals.