Groq
GroqCloud runs open-weight LLMs on custom LPU hardware for very fast, low-latency inference through an OpenAI-compatible API.
Groq is a hosted inference service that serves open-weight models — Llama, Qwen, Mistral, OpenAI open-weights, Whisper — on its custom LPU chips for unusually high tokens-per-second and low latency. The GroqCloud API is OpenAI-compatible, so most SDKs work by changing the base URL, and pricing is per-token with a free tier.
Groq is an AI inference company built around custom silicon. Where most providers serve models on GPUs, Groq designed the LPU (Language Processing Unit) specifically for inference, and the result is unusually high tokens-per-second and low time-to-first-token. Its hosted platform, GroqCloud, exposes that hardware through a simple API so you don't manage any of it yourself.
The catalog is open-weight models — Llama, Qwen, Mistral, OpenAI open-weights (GPT-OSS), and Whisper for speech-to-text — rather than closed frontier models. The draw is speed and predictable per-token pricing on those open models, not exclusive access to a proprietary model family.
The API is OpenAI-compatible: point an OpenAI client at https://api.groq.com/openai/v1 with a Groq key and most existing code, SDKs, and tooling work with a one-line change. There is a usable free tier for prototyping, then per-token Developer pricing and Enterprise plans.
NOTE
Groq (the LPU inference company) is unrelated to Grok, the xAI chatbot — they're easy to confuse but different products.
Groq sits alongside other hosted open-model inference services like Together AI, Fireworks AI, and Baseten; its differentiator is latency and throughput from the LPU. If you're weighing a hosted API against running models yourself, see self-host vs API and LLM cost and latency engineering.
Frequently asked questions
- What is Groq?
- Groq is an AI inference company that runs open-weight large language models on its own custom silicon, the Language Processing Unit (LPU). Through GroqCloud, it serves models like Llama, Qwen, Mistral, OpenAI open-weights, and Whisper at very high throughput and low latency over an OpenAI-compatible API. Note: this is Groq the hardware/inference company, not Grok the chatbot.
- Is Groq free?
- Groq is freemium. GroqCloud has a genuinely usable free tier (rate-limited, no card required) for prototyping, then paid usage billed per token on a Developer tier, plus Enterprise plans. The official Groq SDKs (Python, TypeScript) are open source under Apache-2.0, but the hosted LPU service itself is a commercial product.
- How do I use Groq?
- Sign up at console.groq.com, create an API key, and call the OpenAI-compatible endpoint at https://api.groq.com/openai/v1. Because it mirrors the OpenAI API, most existing OpenAI SDKs and tools work by swapping the base URL and key; Groq also ships official Python and TypeScript libraries.
Related
- Deploying LLMs to Production: A Reliability & Cost ChecklistTake an LLM feature from prototype to production: API vs self-host, provider fallback, retries, caching, observability, eval gates, and safe rollout.
- Self-Host vs API: When Does Running Your Own LLM Actually Pay Off?The real economics of self-hosting an LLM vs. calling a hosted API — GPU utilization, privacy, latency, and the hidden ops costs that decide the crossover.
- LLM Cost and Latency Engineering: Caching, Right-Sizing, and p95 BudgetsA practical playbook for cutting LLM cost and tail latency — caching, model right-sizing, prompt trimming, and enforced p95 budgets — without losing quality.
- Choosing the Right Model: Haiku vs Sonnet vs OpusHow to pick the right Claude model tier — Haiku, Sonnet, or Opus — for any Claude Code agent or task, with a clear decision rubric and per-agent examples.
- Calling Any Model: Unified LLM Gateways & SDKs in 2026Why teams put a unified layer in front of LLM providers — and how LiteLLM, OpenRouter, and the Vercel AI SDK compare for fallback and cost control.