AI engineering
Building with LLMs, not just coding with them: RAG, evals, agents, inference, and the SDKs, skills, and subagents that get AI features to production.
Start here
Read these in order — each one assumes the last.
- 01GuideThe AI Engineer Roadmap for 2026A staged path from API calls to production agents — the skills that matter in 2026, what to skip, and the guides and tools for each stage, in order.
- 02GuideGetting Started with Claude Code AgentsWhat Claude Code subagents are, why they help, and how to add your first one — with a working example agent you can copy into .claude/agents today.
- 03GuideHow RAG Actually Works: Ingestion, Chunking, Retrieval & RerankingA clear, practical walkthrough of the retrieval-augmented generation pipeline — what each stage does, where it fails, and how the pieces fit together.
- 04GuideWrite Evals for an LLM App: From Zero to a CI GateHow to evaluate an LLM feature — build a dataset, choose metrics, set a baseline, score offline, add an LLM judge, and gate CI so quality changes are measured.
- 05GuideDeploying LLMs to Production: A Reliability & Cost ChecklistTake an LLM feature from prototype to production: API vs self-host, provider fallback, retries, caching, observability, eval gates, and safe rollout.
Guides73
Tutorials and deep-dives
Codex MCP Setup: Connect Tools and Live Context
Connect OpenAI Codex to MCP servers through the CLI, desktop app, IDE, or config.toml — with practical scoping, OAuth, and tool-approval guidance.
Codex Subagents: Parallel Work Without Losing Control
Use Codex subagents for parallel exploration, review, tests, and implementation — including decomposition, custom agents, permissions, and merge discipline.
Idempotency Keys: Design APIs That Are Safe to Retry
Design idempotent API mutations with client keys, atomic claims, response replay, payload fingerprints, in-flight handling, TTLs, and downstream safety.
LLM Observability in Production: Traces, Evals, Cost, and Feedback
Instrument production LLM applications with end-to-end traces, online evals, cost and latency metrics, privacy controls, alerts, and a failure-feedback loop.
LLM-as-Judge: Build Evaluators You Can Actually Trust
Design and calibrate LLM-as-judge evaluators — rubrics, pointwise and pairwise scoring, bias controls, human agreement, reliability tests, and CI gates.
Production Model Routing: Cut Cost Without Hiding Regressions
Design an LLM model router with capability gates, difficulty signals, cascades, fallbacks, per-route evals, shadow traffic, budgets, and safe rollout.
AI Coding Statistics 2026: The Numbers That Are Actually Sourced
How much code AI writes, who uses the tools, and what it does to quality — every statistic dated and traced to its primary source, updated on a cadence.
LLM API Pricing in 2026: Every Major Model Compared
Per-million-token prices for Claude, GPT, Gemini, DeepSeek, Mistral, and Grok — plus caching and batch discounts — verified against vendor pricing pages.
The Best MCP Servers in 2026
The MCP servers actually worth connecting in 2026 — Context7, GitHub, Chrome DevTools, Playwright, Serena, Exa, Firecrawl, and official vendor servers.
Getting Web Data into AI Agents: Search & Scraping APIs Compared
The agent web-data layer — Exa for semantic search, Firecrawl for extraction at scale, Tavily for all-in-one, Jina Reader for zero-setup — and how they compose.
Adding MCP Servers to Claude Code: Local, Remote, and Project-Scoped
The complete claude mcp add reference — stdio vs HTTP transports, local/project/user scopes, .mcp.json with env expansion, OAuth via /mcp, and the gotchas.
Building Agents with the Claude Agent SDK
A working tutorial for the Claude Agent SDK in TypeScript and Python — query(), tool permissions, custom in-process MCP tools, subagents, hooks, and auth.
Claude vs GPT vs Gemini for Coding in 2026
The three frontier model families compared for real coding work — agentic depth, ecosystem fit, context, and cost shape — plus how to actually choose.
GraphRAG Explained: When Knowledge Graphs Beat Vector Search
What GraphRAG is, how graph-based retrieval differs from vector RAG, the query shapes where it wins, and the honest costs before you build one.
Sandboxing AI-Generated Code: E2B vs Modal vs Daytona vs Vercel Sandbox
Where should agent-written code run? E2B vs Modal vs Daytona vs Vercel Sandbox compared on isolation, persistence, and cost, plus rules for safe execution.
Vibe Coding in 2026: What It Is, When It Works, When It Bites
An honest guide to vibe coding — where prompt-and-accept development genuinely pays, where it accumulates risk, and the guardrails that make it professional.
Deploying a Remote MCP Server: Stateless, Streamable HTTP, and Horizontal Scaling
Take an MCP server from local stdio to a remote, multi-user HTTP service — Streamable HTTP, stateless vs. stateful sessions, OAuth, and horizontal scaling.
Fine-Tune vs RAG vs Prompt vs Distill: The 2026 Decision Tree
When to reach for prompt engineering, RAG, fine-tuning, or distillation — what each actually changes, where each fails, and how to combine them.
Best LLM & RAG Evaluation Tools in 2026: DeepEval vs RAGAS vs LangSmith vs Phoenix vs promptfoo
A decision guide to the LLM eval landscape — code-first frameworks vs. eval-and-observability platforms, open-source vs. hosted, and which fits your stack.
Cursor vs Claude Code vs GitHub Copilot vs Windsurf in 2026
A practical, opinionated comparison of the four mainstream AI coding tools — form factor, agentic depth, model choice, and who each one is for.
Structured Output vs JSON Mode vs Function Calling: Which to Use in 2026
The reliable ways to get typed data out of an LLM — what JSON mode, function calling, and native structured outputs each guarantee, and when to use which.
Which Agent Framework in 2026? LangGraph vs CrewAI vs AutoGen vs OpenAI Agents SDK vs Claude Agent SDK
A decision guide to the major AI agent frameworks — control vs. abstraction, multi-agent models, state and durability, and which fits your project.
Prompt Patterns for Coding Agents
Practical prompting patterns: chaining, few-shot, context management, tool use, and output structuring.
Adjudicate AI-Assisted Text Labels with an Evidence Ledger
Resolve disputed text labels by preserving each response, source excerpt and task-rule version rather than accepting the model or majority automatically.
Link Records Across Datasets with AI-Assisted Review
Build a cross-source entity crosswalk that preserves both record keys, unresolved candidates and cardinality decisions without silently joining or merging data.
10 Best Claude Skills for RAG
Compare Claude skills and commands for RAG scaffolding, chunking, grounding checks, embeddings, reranking, pgvector, index tuning, and GraphRAG.
Best Web Search APIs for AI Agents in 2026: Tavily vs Exa vs Brave vs Parallel
Tavily, Exa, the Brave Search API, Parallel, Jina Reader and Firecrawl compared on search, extraction, answers, index ownership, and official MCP servers.
Jina Reader API: r.jina.ai and s.jina.ai Headers, Examples, and Limits
How to use the Jina Reader API: r.jina.ai and s.jina.ai quickstart, every documented request header, curl recipes, rate limits, MCP setup, and self-hosting.
Mastra vs LangChain (2026): TypeScript Agent Frameworks Compared
Mastra vs LangChain.js and LangGraph.js in 2026: license, workflows, memory, evals, observability and deployment compared, plus where the Vercel AI SDK fits.
The Best Document Parsers for RAG in 2026: LlamaParse vs Docling vs Unstructured vs Reducto
LlamaParse, Docling, Unstructured, Reducto, Marker and Mistral OCR compared on deployment, license, OCR, tables and output, with a verdict for each.
Tools99
The AI tooling directory
Argilla
Evaluate Argilla for a supervised text task with separately preserved suggestions, human responses and decision evidence.
Tamr
Evaluate Tamr with a bounded mastering pilot that preserves source keys, exposes uncertain candidates and names the owner of each decision.
Brave Search API
Brave's web search API on its own independent index: ranked results, LLM Context grounding chunks, and cited answers from an OpenAI-compatible endpoint.
Cerebras
Cerebras Inference serves open-weight LLMs on wafer-scale chips for very fast output, through an OpenAI-compatible API with a free trial credit.
Jules
Google's asynchronous coding agent: it clones your GitHub repo into a cloud VM, proposes a plan built with Gemini, and opens a pull request once you approve.
Mistral OCR
Mistral AI's hosted OCR and document-understanding API: PDFs, Office files and images in, per-page Markdown and structured JSON out, billed per page.
Parallel Web Systems
Parallel Web Systems' web APIs for AI agents on its own index: objective-based Search, Extract to markdown, async Task research, and a free keyless MCP.
Label Studio
Label Studio is an open-source annotation interface that can present imported or model-generated predictions for people to review and correct.
Dedupe
Evaluate Dedupe through an owner-reviewed matching pilot with known pairs, preserved source keys, candidate coverage checks and unresolved cases.
Agno
A Python framework for building multi-agent systems with memory, knowledge, and tools — formerly Phidata, now with the AgentOS runtime.
Browserbase
Managed headless-browser infrastructure for AI agents and web automation — serverless cloud browsers with stealth, proxies, live view, and Playwright/Stagehand.
Factory
Factory is an agent-native software development platform whose Droids plan, write, test, and ship code from the terminal, IDE, and web with org context.
fal
fal is a generative-media inference cloud for running image, video, and audio diffusion models fast — 1,000+ models, a simple API, and pay-per-use pricing.
Flowise
Open-source, low-code visual builder for LLM apps and AI agents — drag-and-drop to assemble chains, agents, and RAG; self-host or use Flowise Cloud.
Google ADK
Google's open-source, code-first framework to build, evaluate, and deploy AI agents. Python and Java, model-agnostic, deploys to Vertex AI Agent Engine.
Mastra
An open-source TypeScript framework for building AI agents, workflows, RAG, and tool-calling, with memory, model routing, and built-in observability.
turbopuffer
A serverless vector and full-text search database built on object storage (S3/GCS/Azure) — usage-based pricing, hybrid search, and low cost per GB at scale.
Baseten
Production inference platform for ML and LLM models — autoscaling GPU deployments, scale-to-zero, and packaging via the open-source Truss framework.
Docling
Open-source Python library that parses PDFs, DOCX, PPTX, HTML, and images into structured Markdown and JSON with layout, tables, and reading order for RAG.
Fireworks AI
Fast production inference for open models — serverless and dedicated GPU deployments, fine-tuning, and an OpenAI-compatible API on a proprietary serving stack.
Groq
GroqCloud runs open-weight LLMs on custom LPU hardware for very fast, low-latency inference through an OpenAI-compatible API.
LlamaParse
Hosted document-parsing API from LlamaIndex that turns complex PDFs — tables, charts, figures, handwriting — into clean, LLM-ready Markdown for RAG.
Marker
Open-source pipeline that converts PDFs, images, and Office docs into clean Markdown, JSON, or HTML fast, with optional LLM assist for tables and equations.
Reducto
High-accuracy document ingestion API — parsing, agentic OCR, table and figure extraction, and splitting that turns messy PDFs into LLM-ready data for RAG.
Replicate
Run and deploy any open ML model — LLMs, image, video, audio — through one API with pay-per-second billing, and package your own with open-source Cog.
Together AI
A cloud for running, fine-tuning, and deploying open-source models (Llama, DeepSeek, Qwen) via an OpenAI-compatible API plus dedicated GPU endpoints.
Unstructured
Open-source library plus hosted Unstructured Pipelines and API that turn messy documents — PDF, HTML, docx, images, email — into clean, chunked JSON for RAG.
OpenHands
Open-source autonomous AI software-development agent (formerly OpenDevin) — writes code, runs commands, and browses the web in a sandbox.
Swe Agent
Open-source autonomous coding agent from Princeton/Stanford that turns an LLM into a software engineer to fix real GitHub issues.
Assemblyai
Speech AI platform: promptable Universal-3 Pro STT, a flat-rate Voice Agent API, and speech understanding — summarization, sentiment, PII redaction.
Glossary60
AI terms, defined precisely
Record Linkage
Understand the reviewable crosswalk needed before joining sources with unrelated keys, ambiguous candidates and explicit cardinality policies.
Entity Resolution
Understand the identity question behind similar records, including false merges, missed matches and the source owner’s unresolved decisions.
Blocking in Record Matching
Understand how a candidate-selection rule can miss known pairs and why missing fields need coverage checks before identity review.
Attention Mechanism
Attention lets a model weigh how relevant every other token is to each token, building a context-aware representation as a weighted blend of their values.
Context Engineering
Context engineering is the discipline of curating exactly what enters an LLM's context window so it has the right information and nothing else.
Extended Thinking
Extended thinking is the reasoning tokens a model generates before its final answer, trading latency and cost for higher accuracy on hard problems.
Flash Attention
FlashAttention is an IO-aware, exact attention algorithm that runs standard attention far faster and with less memory by tiling on-chip.
LLMOps
LLMOps is the practices and tooling for running LLM apps in production: prompt versioning, evals, tracing, cost and latency monitoring, and guardrails.
Model Routing
Model routing sends each request to the cheapest model that can handle it, escalating only hard cases to a stronger model — cutting cost and latency.
Needle in a Haystack
Needle in a haystack is a long-context eval that hides a fact in filler text and tests whether the model can retrieve it at varying depths and lengths.
Perplexity
Perplexity measures how well a language model predicts a text sample — the exponential of its average per-token negative log-likelihood. Lower is better.
ReAct (Reasoning + Acting)
ReAct is an agent loop that interleaves reasoning with tool actions — Thought, Action, Observation, repeat — so the model plans, calls a tool, and revises.
Semantic Caching
Semantic caching reuses LLM responses keyed by meaning rather than exact text, matching queries by embedding similarity to cut cost and latency.
Test-Time Compute
Test-time compute is spending more computation at inference — longer reasoning, sampling, or search — to improve answers without retraining the model.
Tokenization
Tokenization splits text into tokens — the sub-word units a model reads and writes — and maps each to an integer ID the model processes.
Top-k Sampling
Top-k sampling restricts an LLM's next-token choice to the k most probable tokens before sampling; lower k is more deterministic, higher k more diverse.
Transformer
The neural-network architecture (Vaswani et al., 2017) that uses self-attention to process sequences in parallel — the basis of nearly all modern LLMs.
Agent Harness
An agent harness is the system around the model that makes it an agent — the loop, tools, context management, permissions, and recovery machinery.
Batch Inference
Batch inference processes many LLM requests asynchronously instead of one-at-a-time interactively — typically at ~50% discount via provider batch APIs.
Chunking
Chunking splits documents into retrievable pieces before embedding — the RAG design decision that quietly determines retrieval quality.
Cosine Similarity
Cosine similarity measures how alike two embeddings are by the angle between them — the standard relevance score behind semantic search and RAG retrieval.
DPO (Direct Preference Optimization)
DPO aligns a model to preferences directly from chosen-vs-rejected pairs — no reward model, no RL loop — simpler and more stable than classic RLHF.
Embedding Dimension
Embedding dimension is the length of an embedding vector — how many numbers represent each text — trading capacity against storage and search cost.
Eval Dataset
An eval dataset is the curated set of test cases — inputs with expected outcomes — that an LLM application's quality is measured against.
Grounding
Grounding ties a model's output to verifiable sources — retrieved documents, tool results, citations — instead of training-data memory.
Hybrid Search
Hybrid search runs keyword (BM25) and semantic (vector) retrieval together and merges the results — catching both exact terms and paraphrases.
Open Weights
An open-weights model publishes its parameters for anyone to download and run — unlike API-only models — with licenses from permissive to restricted.
Prompt Template
A prompt template is a parameterized prompt — fixed instructions with variable slots — turning prompts from strings into versioned, testable components.
RLHF (Reinforcement Learning from Human Feedback)
RLHF trains a model against human preferences: people rank outputs, a reward model learns the ranking, and the LLM is optimized to produce preferred responses.
SLM (Small Language Model)
A small language model is a compact LLM — roughly 1–15B parameters — that runs cheaply or locally, trading peak capability for speed and deployability.
Agents19
Specialized subagents for focused work
ML Engineer
Use this agent for production ML — pipelines, training, serving, evaluation, and MLOps. Examples — building a training pipeline, deploying a model, setting up evaluation.
Entity Match Reviewer
Review an entity-match candidate ledger against supplied records and owner criteria; flag unsupported match decisions, conflicting fields and transitive contradictions without merging or identifying people.
Annotation Evidence Reviewer
Review a supervised text-label adjudication packet for traceable excerpts, preserved independent responses and rule-supported resolutions; flag missing evidence without choosing final labels.
Browser Agent Engineer
Use this agent to build, harden, or debug browser-automation agents — web tasks via Browser Use, Stagehand, Skyvern, or Playwright-based stacks. Examples: automate a portal workflow, make a flaky browser agent reliable, add verification and guardrails to web automation, choose between vision and DOM grounding.
Finetuning Engineer
Use this agent to fine-tune an open-weight model end to end — confirming fine-tuning is the right tool, preparing the dataset, choosing the method (LoRA/QLoRA vs. full), running training, and proving the result beats the prompted baseline on a held-out eval set. Examples — "fine-tune a small model to match our support tone and answer format", "we have 800 labeled examples — LoRA-tune and show it beats prompting", "our fine-tune overfits and forgot general ability — fix the data and run".
LLM Cost Optimizer
Use this agent to cut the cost and latency of an application's LLM API usage without losing quality — audit where the tokens and dollars go, then apply caching, model right-sizing, prompt trimming, batching, and budgets, proven against an eval bar. Examples — "our OpenAI bill tripled, find where the spend is and cut it", "this endpoint's p95 is 8s, bring it down", "right-size models per task and add prompt caching to our chat feature".
LLM Inference Engineer
Use this agent to serve and optimize self-hosted LLM inference — sizing GPUs, configuring a serving engine like vLLM (continuous batching, PagedAttention, tensor parallelism), applying quantization, and tuning throughput and tail latency against a cost and p95 budget. Examples — "serve Llama-3-70B at p95 under 2s on our GPUs", "our self-hosted model is slow and the GPUs sit half-idle — raise throughput", "quantize this model to fit one GPU without wrecking quality".
Postgres Migration Engineer
Use this agent to plan and execute a zero-downtime Postgres schema migration — decomposing a breaking change into expand-contract steps, writing batched backfills, building indexes CONCURRENTLY, validating constraints online, and keeping every step reversible with the project's migration tooling. Examples — "add a NOT NULL column to a 200M-row table without downtime", "rename a column safely across a rolling deploy", "split this risky migration into reversible expand/contract steps".
Vector Search Engineer
Use this agent to design, build, and tune the vector-database layer of a search or RAG system — schema and index design (HNSW/IVF + quantization), metadata/payload filtering, hybrid (dense + sparse) search, and ingestion/upsert pipelines — sized to a real latency, recall, and cost budget. Examples — "set up pgvector for our docs with HNSW and filtered search", "our Qdrant queries are slow and recall dropped after quantization", "add metadata filtering so search only returns the current tenant's documents".
Voice Agent Engineer
Use this agent to build or fix a real-time voice agent — the streaming STT → LLM → TTS pipeline, conversational (mouth-to-ear) latency, turn-taking, barge-in/interruptions, and per-stage provider selection. Examples — "our voice bot feels laggy and talks over people, fix the turn-taking and latency", "build a phone agent that transcribes, answers with our LLM, and speaks back", "get our voice agent's response time under a second".
Agent Tool Integration Engineer
Use this agent to wire tools and function-calling into an agent loop reliably — clean tool schemas, errors fed back as observations, retries with limits, idempotency, and parallel calls. Examples — "connect our APIs as agent tools", "our agent calls tools wrong / ignores tool errors", "add function-calling with proper error recovery to our agent".
LLM Evaluation Engineer
Use this agent to make an LLM feature's quality measurable — building the dataset, choosing metrics, setting a baseline, and turning evals into a CI gate so prompt and model changes are scored, not guessed. Examples — "we changed the prompt and don't know if it's better, set up evals", "add a regression gate for our extraction feature", "our RAG quality is drifting, build an eval suite".
LLM Integration Engineer
Use this agent to add an LLM feature to an application and make it production-grade — typed/structured output, streaming, provider fallback and retries, caching, and cost/latency controls. Examples — "add an AI summary endpoint to our app", "our LLM calls return unparseable JSON and break, make them reliable", "add streaming and a fallback provider to our chat feature".
LLM Observability Engineer
Use this agent to make a production LLM app observable — tracing every step, scoring live traffic with online evals, and monitoring quality, cost, and latency — so you can debug agent runs and catch regressions in production. Examples — "add tracing to our RAG/agent so we can debug bad answers", "set up online evals and cost/latency dashboards", "production quality is slipping and we're flying blind".
Prompt Engineer
Use this agent to design and iterate the prompts behind an LLM-powered product feature — instructions, few-shot examples, tool schemas, and the evals that prove a change actually helped. Examples — "this classification prompt is flaky, make it reliable", "design the system prompt and function schema for our support agent", "our extraction prompt regressed after I tweaked it, set up evals so this stops happening".
Rag Pipeline Engineer
Use this agent to design, build, and harden a production retrieval-augmented generation (RAG) pipeline end to end — ingestion, chunking, embeddings, indexing, retrieval, reranking, and grounded generation — with evals that prove each stage works. Examples — "stand up RAG over our docs", "our RAG hallucinates and misses obvious answers, fix the pipeline", "take our prototype RAG to production with evals and citations".
Retrieval Engineer
Use this agent to raise the retrieval quality of a search or RAG system — recall and precision, hybrid (dense + sparse) search, reranking, query transformation, and metadata filtering — measured against a labeled eval set. Examples — "our RAG retrieves irrelevant chunks, fix recall", "add hybrid search and reranking and prove it helps", "queries with acronyms/IDs return nothing, fix it".
Data Engineer
Use this agent to build and maintain data pipelines — ingestion, ELT/ETL, warehouse modeling, orchestration, and data-quality tests. Examples — building an idempotent ingestion job, modeling a fact/dimension table in dbt, writing a safe backfill for a changed schema.
Data Scientist
Use this agent for data analysis — exploration, statistics, SQL, and clear findings. Examples — analyzing a dataset, writing an analytical SQL query, summarizing experiment results.
Skills23
Reusable capabilities Claude loads on demand
Annotation Task Spec Builder
Specify the record and annotation-interface contract for an owner-defined supervised text-labeling task: approved label IDs, input text, model suggestions, human response fields and abstain handling.
Entity Match Candidate Ledger
Build a field-evidence ledger for proposed duplicate or cross-source record pairs, preserving source keys, original values and match/different/unresolved owner decisions without merging records.
Annotation Disagreement Ledger
Build an adjudication ledger from conflicting supervised text-label responses, preserving original reviewer labels, exact excerpts, task-rule versions and unresolved owner decisions.
Agent Memory Designer
Design a project's CLAUDE.md and memory hierarchy by exploring the repo to learn its real build/test/lint commands, architecture, and non-obvious gotchas, then writing a concise, skimmable memory that keeps only what belongs in context. Use when onboarding a repo to Claude Code with no CLAUDE.md, or when an existing one is bloated, stale, or being ignored.
Agent Trajectory Evaluator
Evaluate a multi-step AI agent's whole run — tool calls, intermediate steps, and final result — not just final-answer correctness, so you can pinpoint WHERE it went wrong. Use when building or debugging a tool-using or multi-step agent, when final-answer-only evals can't explain failures, or when a prompt/model change quietly makes the agent less efficient or more error-prone even though the answer still looks right.
Hallucination Evaluator
Detect and measure ungroundedness in LLM and RAG outputs — claims the source doesn't support — by decomposing answers into atomic claims and checking each for entailment, so you can quantify faithfulness and gate on it instead of eyeballing it. Use when a RAG/LLM feature makes confident wrong claims, before shipping anything that must be factual, or to add a groundedness gate to evals/CI.
Model Router Designer
Design a model router that sends each LLM request to the cheapest model that can handle it and escalates only the hard cases to the strongest — cutting cost and latency without tanking quality, gated by an eval set so the savings don't come from silently worse answers. Use when one expensive model serves all traffic (most of it easy), when LLM cost or latency is too high, or when balancing quality against spend across a range of request difficulty.
Prompt Regression Tester
Build a regression test harness for an LLM prompt so a prompt edit or model upgrade can't silently degrade quality — a fixed eval set, checkable assertions, and a diff against a committed baseline. Use when changing a production prompt, migrating model versions, or any time 'I tweaked the prompt' needs to be backed by evidence instead of eyeballing two outputs.
Semantic Cache Designer
Design a semantic cache for LLM responses — serve a cached answer when a new query is similar enough to a past one — to cut cost and latency on repetitive traffic, with the similarity threshold calibrated on real query pairs and a cache key that prevents cross-user/model leaks. Use when an LLM app sees many near-duplicate prompts (FAQs, support, search), when token spend on repetitive queries is high, or when latency on common questions matters.
Token Usage Profiler
Measure and attribute LLM token usage and cost across an app — input vs output tokens by feature, route, model, and tenant — then rank the waste and the specific lever to cut it. Use when LLM spend is high or climbing with no clear cause, before scaling a feature that calls a model, or when you need per-feature or per-tenant cost attribution for billing or budgets.
Graphrag Scaffolder
Stand up a GraphRAG experiment the disciplined way: audit whether your failed queries are actually connection-shaped, scope a minimal entity/relationship ontology, build extraction → graph → community-summary indexing on a corpus slice, and measure against vector-RAG baselines before committing. Use when multi-hop or whole-corpus questions keep failing plain RAG.
Web Research Pipeline
Run a structured web-research pass on a question: plan the searches, find sources via search APIs, fetch and read the best ones, cross-check claims, and synthesize a cited answer — with source quality and disagreements surfaced honestly. Use for 'research X and tell me what's actually true' tasks that need more than one search and less than a day.
Finetune Dataset Builder
Turn raw examples into a training-ready fine-tuning dataset — normalize to the trainer's chat/instruction format, deduplicate (including near-duplicates), strip PII, balance, validate the schema and token lengths, and carve a leak-free eval split. Use when you have raw examples and need a clean, formatted, split dataset before training.
Multimodal Document Extractor
Extract structured data from documents and images with a vision-language model — define the target schema, prompt the VLM to fill it from the page (invoices, forms, receipts, statements, IDs), and verify critical fields against the source. Use when you need reliable structured output from messy, varied, or scanned documents that defeat template-based OCR.
Prompt Cache Optimizer
Restructure an LLM call to maximize prompt-cache hit rate and add response/semantic caching — move the stable prefix (system prompt, instructions, few-shot, context) to the front and variable input to the end, set cache breakpoints, and measure the hit rate and savings. Use when repeated calls share large common context and token cost or latency is too high.
Prompt Optimizer
Diagnose why a prompt underperforms and rewrite it with the technique that fixes it — clearer structure, few-shot examples, an explicit output contract, or reasoning scaffolding — returning an optimized prompt, the rationale for every change, and what to measure to confirm the lift. Use when a prompt is flaky, verbose, drifting in format, or just not good enough.
Qlora Finetune Runner
Run a QLoRA (4-bit LoRA) fine-tune of an open-weight model from a prepared dataset — set up the config, train memory-efficiently (e.g. with Unsloth/PEFT), watch for overfitting, save the adapter, and run a quick eval against the prepared split. Use when you have a clean dataset and want to execute a parameter-efficient fine-tune on a single GPU.
Chunking Strategy Optimizer
Find the chunking strategy and size that maximizes retrieval quality for a specific corpus, by sweeping configurations against a fixed eval set instead of guessing. Use when RAG answers miss obvious content, when standing up a new corpus, or when picking chunk size/overlap.
Embedding Set Inspector
Diagnose the health of an embedding set before blaming the retriever — checking normalization, dimensionality, near-duplicates, degenerate vectors, and corpus/query distribution mismatch. Use when retrieval quality is poor, after a re-embed, or before shipping a new index.
Human In The Loop Gate
Add a human approval checkpoint to an agent so it pauses before a risky or irreversible action (spending money, deleting data, sending messages, merging code) and resumes only after a human approves. Use when an agent acts autonomously on consequential operations.
LLM As Judge Scorer
Design a reliable LLM-as-judge metric — a calibrated rubric, a clear scoring scale, and bias controls — and validate it against human labels before trusting it. Use when grading open-ended LLM output (summaries, answers, tone) that exact-match can't score.
LLM Eval Suite Scaffolder
Stand up an evaluation suite for an LLM feature from scratch — a representative dataset, the right metrics, a baseline score, and a CI gate — using DeepEval, promptfoo, or RAGAS. Use when a feature has no evals, before tuning a prompt, or when adding an LLM feature to CI.
SQL Optimizer
Diagnose a slow SQL query from its execution plan and propose a verified optimization — finding the real bottleneck (sequential scan, missing or unused index, bad join order, app-side N+1) and measuring the fix before and after. Use when a query is slow and you need a fix backed by EXPLAIN ANALYZE, not a guess.
Commands7
Slash commands for Claude Code
Check Annotation Batch
Check annotation-batch.json for approved labels, distinct response reviewers, exact text evidence and unresolved disagreements while preserving the input.
Scaffold RAG Pipeline
Scaffold a Retrieval-Augmented Generation pipeline — ingestion (load, chunk, embed, upsert) and retrieval (search, rerank, grounded prompt with citations) — fitted to the project's stack.
Red Team LLM
Red-team an LLM app or agent for prompt injection, jailbreaks, and data leakage — probe the real attack surface (input, RAG, tools, system prompt) with adversarial inputs and report what got through and how to fix it.
Scaffold a vLLM Serving Config
Scaffold a vLLM serving config for a model on a target GPU — pick precision/quantization and parallelism to fit, set batching and context length, and expose an OpenAI-compatible server.
Add a Streaming LLM Endpoint
Scaffold a token-streaming LLM endpoint — server-side streaming plus the client handler — so responses render incrementally instead of after a long wait.
Add Human Approval Step
Scaffold a human-in-the-loop approval gate into an agent so it pauses before a consequential action and resumes after approval.
Benchmark Rerankers
Measure whether adding a reranker actually improves retrieval, by scoring reranked vs. un-reranked results on a labeled query set.