Groq
GroqCloud runs open-weight LLMs on custom LPU hardware for very fast, low-latency inference through an OpenAI-compatible API.
Groq is a hosted inference service that serves open-weight models — OpenAI GPT-OSS, Whisper, enterprise-only Llama, and Qwen in preview — on its custom LPU chips for unusually high tokens-per-second and low latency. The GroqCloud API is OpenAI-compatible, so most SDKs work by changing the base URL, and pricing is per-token with a free tier.
Groq is an AI inference company built around custom silicon. Where most providers serve models on GPUs, Groq designed the LPU (Language Processing Unit) specifically for inference, and the result is unusually high tokens-per-second and low time-to-first-token. Its hosted platform, GroqCloud, exposes that hardware through a simple API so you don't manage any of it yourself.
In December 2025 Groq entered a non-exclusive licensing agreement with NVIDIA for its inference technology. It was not an acquisition: founder Jonathan Ross and other team members joined NVIDIA, while Groq continues as an independent company and GroqCloud kept running without interruption. In August 2026 Groq became an NVIDIA Cloud Partner and said it will also deploy NVIDIA hardware, including Groq 3 LPX, in its cloud.
The catalog is open-weight models rather than closed frontier models. As of September 2026 its production models are OpenAI's GPT-OSS 120B and 20B, Whisper for speech-to-text, and Llama 3.1 8B and Llama 3.3 70B, which are now enterprise-only with contact-sales pricing; Qwen models are in preview. The draw is speed and predictable per-token pricing on those open models, not exclusive access to a proprietary model family.
The API is OpenAI-compatible: point an OpenAI client at https://api.groq.com/openai/v1 with a Groq key and most existing code, SDKs, and tooling work with a one-line change. There is a usable free tier for prototyping, then per-token Developer pricing and Enterprise plans.
NOTE
Groq (the LPU inference company) is unrelated to Grok, the xAI chatbot — they're easy to confuse but different products.
Groq sits alongside other hosted open-model inference services like Together AI, Fireworks AI, and Baseten; its differentiator is latency and throughput from the LPU. If you're weighing a hosted API against running models yourself, see self-host vs API and LLM cost and latency engineering.
Frequently asked questions
- What is Groq?
- Groq is an AI inference company that runs open-weight large language models on its own custom silicon, the Language Processing Unit (LPU). Through GroqCloud, it serves models like OpenAI's open-weight GPT-OSS and Whisper at very high throughput and low latency over an OpenAI-compatible API; its Llama models are now enterprise-only, and Qwen models are in preview. Note: this is Groq the hardware/inference company, not Grok the chatbot.
- Is Groq free?
- Groq is freemium. GroqCloud has a genuinely usable free tier (rate-limited) for prototyping, then paid usage billed per token on a Developer tier, plus Enterprise plans. The official Groq SDKs (Python, TypeScript) are open source under Apache-2.0, but the hosted LPU service itself is a commercial product.
- How do I use Groq?
- Sign up at console.groq.com, create an API key, and call the OpenAI-compatible endpoint at https://api.groq.com/openai/v1. Because it mirrors the OpenAI API, most existing OpenAI SDKs and tools work by swapping the base URL and key; Groq also ships official Python and TypeScript libraries.
Filed under
inference · llm-api · low-latency · open-models
Related
- Self-Host vs API: When Does Running Your Own LLM Actually Pay Off?The real economics of self-hosting an LLM vs. calling a hosted API — GPU utilization, privacy, latency, and the hidden ops costs that decide the crossover.
- Calling Any Model: Unified LLM Gateways & SDKs in 2026Why teams put a unified layer in front of LLM providers — and how LiteLLM, OpenRouter, and the Vercel AI SDK compare for fallback and cost control.
- Deploying LLMs to Production: A Reliability & Cost ChecklistTake an LLM feature from prototype to production: API vs self-host, provider fallback, retries, caching, observability, eval gates, and safe rollout.
- Choosing the Right Model: Haiku vs Sonnet vs OpusHow to pick the right Claude model tier — Haiku, Sonnet, or Opus — for any Claude Code agent or task, with a clear decision rubric and per-agent examples.
- LLM Cost and Latency Engineering: Caching, Right-Sizing, and p95 BudgetsA practical playbook for cutting LLM cost and tail latency — caching, model right-sizing, prompt trimming, and enforced p95 budgets — without losing quality.
- CerebrasCerebras Inference serves open-weight LLMs on wafer-scale chips for very fast output, through an OpenAI-compatible API with a free trial credit.
- The Best LLM Inference Providers in 2026: Groq vs Together vs Fireworks vs Baseten and MoreGroq, Cerebras, Together AI, Fireworks AI, Baseten, Modal, fal and Replicate compared by workload, with 2026 ownership changes and OpenAI compatibility.