Hardened LLM Gateway
Cloudflare for LLM APIs. One endpoint, all providers, automatic caching, instant rollback.
Build the production-grade LLM gateway that indie AI builders integrate as their LLM proxy. Flagship pitch: a multi-provider gateway with built-in semantic caching (50%+ cost reduction), prompt injection defense, gradual prompt rollout, and circuit-breaker fallback when a provider goes down. Real market: Helicone, Portkey, Vellum are growing fast in this exact space. The portfolio outcome: 5+ real indie AI projects integrating with your gateway as their LLM proxy. Or take the variation route — harden your Project 2 or Project 4 as the user-facing product itself, with all of the same hardening built in.
The product in detail
The product is the production layer that sits between an AI app and its model providers. The user changes one URL in their codebase and their LLM calls now route through your gateway: multi-provider failover (Anthropic → OpenAI → Cohere if one goes down), semantic caching (50%+ cost reduction on semantically similar queries), per-user rate limits, prompt injection defense, content moderation, audit logs, gradual prompt rollout (10% → 50% → 100%), and instant rollback if a metric drops. One endpoint, one bill, one observability pane.
The target user is an indie AI builder or small-startup engineer who ships AI features fast and doesn't have time to build production infrastructure from scratch. They've been burned at least once — a hallucination they couldn't trace, an API outage that killed a live demo, or a $4,000 bill from a prompt injection that ran the model in a loop. They want one endpoint that handles all of this. They'd pay $50/month flat to stop worrying about production AI plumbing.
Real market: Helicone (YC, $4M raised, 10,000+ developers), Portkey ($3M raised, 4,000+ developers), Vellum ($5M Series A, enterprise-focused), LiteLLM (OSS, 8,000+ GitHub stars). The LLM gateway space is growing fast but indie builders are still mostly rolling their own with ad-hoc scripts. Your gateway is positioned as the "one-weekend setup, scales when you need it" version — the Cloudflare analogy for LLM APIs.
Technically you build: input guardrails (prompt injection detection, PII redaction, token budget cutoffs), output guardrails (schema validation via Pydantic, content moderation, refusal detection), a semantic cache layer (embed queries, hit on cosine similarity > 0.95 with cost / hit-rate dashboarding), Anthropic prompt caching for system prompts, model routing (simple queries → cheap model, complex → strong model, verified by evals), structured logging with trace_id / user_id / model / tokens / cost / latency per request, circuit breakers with cached-response fallback, and A/B test infrastructure with auto-rollback on metric regression.
What you ship: a hardened service with a live URL, public auth, 5+ real projects integrating it as their proxy. A chaos-test report proving the service survives provider outages (LLM API forced unavailable → fallback works). A senior-engineer-grade architecture write-up documenting every tradeoff (cost of caching vs latency of cache miss, quality drop from model routing vs cost savings). When the interview is for a senior or staff AI Engineer role, this is the project that signals production seniority — you've shipped infrastructure, not just features.
Pick what you'll build
This is one project with one skill graph — but you pick the product framing that fits your interests or career goals. Each option below is a viable portfolio piece. We'll narrow it together in the kickoff call.
- LLM gateway for indie AI builders — one endpoint, all providers, caching + observability built in (audience: 10,000+ indie AI builders)
- Harden Project 2's agent for paying users (auth, rate limits, full guardrails, audit logs)
- Harden Project 4's research tool for an enterprise pilot (PII redaction, SOC2-ready logging, audit trail)
- Production-ize an open-source AI demo someone else built — turn a viral GitHub demo into a real service
- Internal AI service for a real team (Slack bot, internal Q&A) that an org actually depends on
- Picks up the under-covered Theme 5 (Production AI Engineering — only 20% of posts but heavy interview emphasis on cost optimization, guardrails, semantic caching).
- After Project 3 instrumented things, this project hardens them.
- Bucket B (non-AI-native companies) cite 'production AI' most heavily — this project mirrors enterprise engineering reality.
This project leans toward enterprise AI platform engineering — roles like Capital One's 'Senior Lead AI Engineer (Gen AI Platform Services)' or Notion's productionization track.
Research backing this project
Every project on projeckt.ai is calibrated against real job posts and real interview questions, not opinion. Here's the data behind this one.
- •Cost Optimization (Δ+8)
- •Guardrails (Δ+6)
- •Semantic caching (interview-heavy)
- •Model routing
Themes covered
Each chip shows how many of the 46 analysed job posts cite this theme.
Justified by real job posts
Verbatim quotes from the postings that drove the design of this project.
“Deploy LLMs and other Machine Learning models into production, ensuring high availability and performance”
“Develop caching layers to optimize GenAI application performance”
“Productionize and scale asynchronous workflows across Notion's infrastructure”
“Own scoped productionization projects: integrate new models/techniques, add monitoring and guardrails, or improve latency/cost/reliability”
What you'll be able to do
- 1Implement input guardrails: PII redaction, prompt injection detection, token-budget cutoffs
- 2Implement output guardrails: schema validation, content moderation, refusal-pattern detection
- 3Build a semantic cache (embed-and-store responses, hit on cosine similarity above threshold) with hit-rate metrics
- 4Use Anthropic prompt caching or OpenAI response cache to reduce repeated-context cost
- 5Implement model routing: cheap model for simple queries, strong model for complex — verified by evals
- 6Add rate limiting per user / per IP with Redis or similar
- 7Structured logging — every request has trace_id, user_id, model used, tokens, cost, latency, output status
- 8Circuit breakers around external dependencies; fallback behaviors when LLM API is down
- 9A/B test prompts in production with gradual rollout (10% / 50% / 100%) and automatic rollback on metric regression
- 10Articulate the latency / cost / quality / reliability tradeoff for at least 5 architectural decisions
Tools you'll learn
Hands-on with these tools across the paid and open-source stacks. The Paid / OSS / Both label shows which stack each tool belongs to.
Week-by-week checkpoints
Each checkpoint is a Pull Request to your public GitHub repo. Rohan reviews on the PR; merge = checkpoint complete.
Input pipeline: prompt injection detection (NeMo or LLM-based classifier), PII redaction. Output pipeline: schema validation, content moderation, max-token enforcement. 5 adversarial test cases pass through unscathed.
Adversarial suite (10+ injection attempts, 5 toxic outputs) — guardrails block all; legitimate traffic unaffected.
- •Prompt injection defense
- •PII redaction patterns
- •Moderation pipelines
- •Refusal-pattern detection
Semantic cache (embed query, hit if cosine sim > 0.95). Anthropic prompt caching enabled for system prompts. Dashboard shows cache hit rate + cost savings.
Cache hit rate ≥30% on benchmark traffic; cost dashboard shows measurable reduction vs baseline.
- •Semantic caching
- •Prompt caching
- •Cost attribution per request type
Classifier (regex + LLM) routes queries to cheap vs strong model. Project 3's eval harness verifies quality didn't drop. A/B test infra deployed — 10% traffic to new prompt, auto-rollback if Faithfulness drops.
Routing in production; cost ↓ measurably without quality drop; A/B harness runs at least one ramp + auto-rollback in dry-run.
- •Model routing classifiers
- •Gradual rollout patterns
- •Auto-rollback on metric regression
Circuit breakers around LLM API + vector DB. Fallback behaviors (cached response if LLM down). Structured logs with trace_id. 4-min Loom walks through architecture, guardrails, caching, routing. Production-grade README.
Service survives chaos test (LLM API forced unavailable → fallback works); demo shows full request lifecycle.
- •Circuit breakers
- •Graceful degradation patterns
- •Production architecture documentation
- •Senior-interview demo craft
Pick your stack
Same skills, different credentials. Start free with the OSS stack or use the paid stack to learn directly on production tools.
| Tool | Purpose | License | Hardware | Market equivalent |
|---|---|---|---|---|
| Ollama + Qwen 2.5 + DeepSeek R1 (model router) | LLM mix for routing | Various OSS | Mac M-series 16GB+ or Linux GPU | Claude + GPT-4o router |
| NeMo Guardrails + Llama Guard 3 | Guardrails pipeline | Apache-2.0 / Llama Community | Runs locally | Guardrails AI + OpenAI Mod |
| Redis self-hosted | Caching + rate limit | BSD | Docker | Upstash |
| Glitchtip (or self-hosted Sentry) | Error monitoring | MIT (Glitchtip) | Docker | Sentry hosted |
| Coolify on VPS | Deploy | Apache-2.0 | $5/mo VPS | Vercel / Railway |
Estimated cost: $5/month (VPS only).
Interview questions you'll be ready for
Real questions from 13 sources, mapped to the skills this project builds.
Ready to start building?
Book a free 30-minute project planning session. We'll tailor this project to your stack, team, and goals.