llm-d
@llm-d.ai
llm-d is a Kubernetes-native distributed inference serving stack providing well-lit paths for anyone to serve large generative AI models at scale. Learn more at: https://llm-d.ai
AI is entering the era of large-scale, distributed, agentic systems. Applications coordinate multiple models, tools, and services, process millions of requests, and demand enormous amounts of computer capacity. For many organizations, this creates a new kind of dependency. buff.ly/pw23Cuo
Scaling agentic AI: How llm-d enables infrastructure sovereignty
Scale agentic AI with llm-d to achieve full infrastructure sovereignty across diverse hardware.
redhat.com
Sticky Until Saturated: Token-Aware Routing in llm-d The llm-d router's default configuration is built on token-aware routing: keep each request on the cache-warm endpoint until a calibrated token-load limit is exceeded, then route by load alone ... llm-d.ai/blog/sticky-...
Sticky Until Saturated: Token-Aware Routing in llm-d | llm-d
The llm-d router's default configuration is built on token-aware routing: keep each request on the cache-warm endpoint until a calibrated token-load limit is exceeded, then route by load alone,…
llm-d.ai
Great new blog post by @pcheslock.bsky.social ont the Red Hat blog: www.redhat.com/en/blog/llm-... #ai #llm-d #inference
llm-d: Breaking the cost and capacity barriers
Optimize inference for large language models (LLMs) with llm-d, reducing redundant computation and lowering AI workload costs.
redhat.com
Benchmarking disaggregated VLM serving of Kimi-VL-A3B-Instruct with 4 Intel Arc Pro B60 vision encoder and 1 NVIDIA H200 language model worker, delivering 2.4x-2.8x higher throughput and 69%-80% lower TTFT than collocated serving. llm-d.ai/blog/scaling...
Watch the latest vLLM office hours to learn how llm-d leverages Wide EP for scaling large MoE models like GLM-5.2 across nodes. www.youtube.com/watch?v=LXLK...
[vLLM Office Hours #53] - llm-d Project Update and Wide EP for Agentic Workloads - July 9, 2026
YouTube video by Red Hat
youtube.com
Part 2 of our 𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝗱 𝗔𝗜 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 series is now live on Red Hat Developer: 𝘖𝘱𝘵𝘪𝘮𝘪𝘻𝘪𝘯𝘨 𝘋𝘪𝘴𝘵𝘳𝘪𝘣𝘶𝘵𝘦𝘥 𝘈𝘐 𝘐𝘯𝘧𝘦𝘳𝘦𝘯𝘤𝘦: 𝘈𝘥𝘷𝘢𝘯𝘤𝘦𝘥 𝘋𝘦𝘱𝘭𝘰𝘺𝘮𝘦𝘯𝘵 𝘗𝘢𝘵𝘵𝘦𝘳𝘯𝘴. In Part 1, we covered prefill/decode phases and the 5D parallelism framework.
The era of the uniform KV cache is officially over. ❌ Modern hybrid models mix attention types—interleaving full attention with sliding-window or Mamba layers. This means the entire serving stack has to adapt to handle this heterogeneity. 👇
Excited to share Part 1 of our blog series on Red Hat Developer: 𝘋𝘦𝘴𝘪𝘨𝘯𝘪𝘯𝘨 𝘋𝘪𝘴𝘵𝘳𝘪𝘣𝘶𝘵𝘦𝘥 𝘈𝘐 𝘐𝘯𝘧𝘦𝘳𝘦𝘯𝘤𝘦: 𝘊𝘰𝘳𝘦 𝘊𝘰𝘯𝘤𝘦𝘱𝘵𝘴 𝘢𝘯𝘥 𝘚𝘤𝘢𝘭𝘪𝘯𝘨 𝘋𝘪𝘮𝘦𝘯𝘴𝘪𝘰𝘯𝘴. LLM inference is two workloads pretending to be one. The prefill phase is compute-bound, processing entire prompts in parallel to populate the KV cache.
👉 Check out the June newsletter here: https://inferenceops.substack.com/p/state-of-the-model-serving-communities-dee 👉 Subscribe to get future issues in your inbox: https://inferenceops.substack.com/ 🚀 Thanks to everyone who subscribed so far! Kudos to all contributors to this edition!
📢 𝗧𝗵𝗲 𝗦𝘁𝗮𝘁𝗲 𝗼𝗳 𝗠𝗼𝗱𝗲𝗹 𝗦𝗲𝗿𝘃𝗶𝗻𝗴 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝗶𝗲𝘀: 𝗝𝘂𝗻𝗲 𝗘𝗱𝗶𝘁𝗶𝗼𝗻 𝗶𝘀 𝗼𝘂𝘁! We recently launched our newsletter publicly after sharing it internally at Red Hat AI for over a year.
Can a cache-aware, saturation-aware router make a mixed, 3-vendor GPU cluster perform like one unified, high-performance inference service? 🚀 Yes, it can. We benchmarked llm-d v0.7.0 across a real-world heterogeneous cluster with IBM Research, Red Hat, and NxtGen Cloud. Here is what we found 👇
🎉 llm-d v0.7 is officially live! 🚀 If our earlier releases proved what llm-d could do, v0.7 is about making sure you can easily deploy it in production. Backed by a massive 3.5x surge in community PR volume, this release hardens the stack for serious scale. 👇 llm-d.ai/blog/llm-d-v...
llm-d v0.7: From Feature Introduction to Production Hardening | llm-d
llm-d v0.7 shifts focus from proving capabilities to making them deployable, with changes across deployment tooling, hardware support, documentation, and continuous integration.
llm-d.ai
Want to learn more about how @opensource.google is helping the open source community with their involvement in llm-d? Join us in Boston THIS WEEK for the llm-d meetup at their Cambridge, MA office. luma.com/eqbc1gxq Join now before registration closes later today.
Open Source Distributed AI Inference (llm-d/vLLM) Meetup · Luma
Open Source Distributed AI Inference (llm-d/vLLM) Meetup Boston/Cambridge Hosted by Google Cloud, Red Hat AI, and the llm-d Community Date: Thursday, May 28th…
luma.com
Boston AI Devs! 🏙️ Join the llm-d meetup on May 28 during Boston Tech Week. Hear the latest in LLMs from: 🎙️ Tyler Michael Smith (@RedHatAI) 🎙️ Sean Horgan (@Google) 🎙️ Peter Tanski (@CapitalOne) Huge thanks to @Google for the support! 🎟️ Register: luma.com/eqbc1gxq
Open Source Distributed AI Inference (llm-d/vLLM) Meetup · Luma
Open Source Distributed AI Inference (llm-d/vLLM) Meetup Boston/Cambridge Hosted by Google Cloud, Red Hat AI, and the llm-d Community Date: Thursday, May 28th…
luma.com
Moving an LLM from demo to production is where the real work begins. It’s not just about accuracy, it’s about latency, GPU efficiency, and cost-to-serve. We’re thrilled to see @OracleCloud’s deep dive on scaling llm-d on OCI for enterprise workloads! 🧵👇
Standard metrics often miss what happens between instances in P/D (Prefill/Decode) mode. Bridge the gap with: 🟣 llm-d tracing: Full request from Gateway to GPU. 🟣 vLLM NIXL Metrics: Real-time KV cache transfer. 🟣 True E2E Latency: Moving beyond TTFT. www.youtube.com/watch?v=Tz3m...
llm-d Distributed Tracing & vLLM NIXL Metrics: Solving the Observability Gap in P/D Mode
In this video, Sally from SIG Observability demonstrates how to achieve comprehensive observability for Large Language Model (LLM) deployments using llm-d distributed tracing and vLLM NIXL metrics. …
youtube.com
How do you scale LLM inference without the "Day 2" operational headaches? We just shared a new deep dive on the llm-d blog: building a production-grade stack with KServe + vLLM + llm-d. Collaborative work with contributors from Red Hat & Tesla. Read the full breakdown: 🔗 llm-d.ai/blog/product...
Production-Grade LLM Inference at Scale with KServe, llm-d, and vLLM | llm-d
How migrating from a simple vLLM deployment to a robust MLOps platform utilizing KServe, llm-d's intelligent routing, and vLLM solved significant scaling and operational challenges in LLM deployment…
llm-d.ai
Our answer: KServe + @llm-d.ai + vLLM with prefix-cache aware routing. The results: 3x more output tokens/sec and 2x faster time to first token. Check out the full writeup here: llm-d.ai/blog/product...
Production-Grade LLM Inference at Scale with KServe, llm-d, and vLLM | llm-d
How migrating from a simple vLLM deployment to a robust MLOps platform utilizing KServe, llm-d's intelligent routing, and vLLM solved significant scaling and operational challenges in LLM deployment t...
llm-d.ai
Excited to share our latest blog post on how we're solving real-world LLM inference challenges at production scale, a collaboration between Red Hat AI and Tesla engineering teams.
What do you do when your local GPUs are tapped out? Check out this KubeCon talk breaking down Federated llm-d. Global pooling treats multi-cluster GPUs as one resource. Smart sourcing routes to accelerators a "hop away" in other regions. Full demo: www.youtube.com/watch?v=FKcG...
Federated llm-d: Elevating Distributed Inference Beyond Clus... Madhuri Yechuri & Abhishek Malvankar
YouTube video by CNCF [Cloud Native Computing Foundation]
youtube.com
📢 𝗧𝗵𝗲 𝗦𝘁𝗮𝘁𝗲 𝗼𝗳 𝗠𝗼𝗱𝗲𝗹 𝗦𝗲𝗿𝘃𝗶𝗻𝗴 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝗶𝗲𝘀: 𝗔𝗽𝗿𝗶𝗹 𝗘𝗱𝗶𝘁𝗶𝗼𝗻 𝗶𝘀 𝗼𝘂𝘁! Our goal with this newsletter is to give a clear, community-driven view of what’s happening across the model serving ecosystem, including updates from projects like vLLM, KServe, @llm-d.ai, @kubernetes.io, Llama Stack, and more.
Check out the latest newsletter to stay up to speed on the changes happening in the model serving communities!
Yuan Tang@terrytangyuan.xyz · 4mo ago📢 𝗧𝗵𝗲 𝗦𝘁𝗮𝘁𝗲 𝗼𝗳 𝗠𝗼𝗱𝗲𝗹 𝗦𝗲𝗿𝘃𝗶𝗻𝗴 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝗶𝗲𝘀: 𝗔𝗽𝗿𝗶𝗹 𝗘𝗱𝗶𝘁𝗶𝗼𝗻 𝗶𝘀 𝗼𝘂𝘁! Our goal with this newsletter is to give a clear, community-driven view of what’s happening across the model serving ecosystem, including updates from projects like vLLM, KServe, @llm-d.ai, @kubernetes.io, Llama Stack, and more.
ICYMI: llm-d is officially a @CNCF Sandbox project! 🚀 We’re evolving #Kubernetes into SOTA AI infrastructure through a powerhouse coalition including Red Hat , Google Cloud , IBM Research, NVIDIA, Mistral AI, Hugging Face , and many more. www.cncf.io/blog/2026/03...
It’s official: llm-d has joined the cncf.io ! 🚀 Our mission to evolve Kubernetes into SOTA AI infrastructure just got a massive boost. This milestone belongs to our amazing community. Thank you for building this with us. 💜 We’re just getting started! 🔗 www.cncf.io/blog/2026/03...
Welcome llm-d to the CNCF: Evolving Kubernetes into SOTA AI infrastructure
We are thrilled to announce that llm-d has officially been accepted as a Cloud Native Computing Foundation (CNCF) Sandbox project! As generative AI transitions from research labs to production…
cncf.io
It’s official: llm-d has joined the cncf.io ! 🚀 Our mission to evolve Kubernetes into SOTA AI infrastructure just got a massive boost. This milestone belongs to our amazing community. Thank you for building this with us. 💜 We’re just getting started! 🔗 www.cncf.io/blog/2026/03...
Welcome llm-d to the CNCF: Evolving Kubernetes into SOTA AI infrastructure
We are thrilled to announce that llm-d has officially been accepted as a Cloud Native Computing Foundation (CNCF) Sandbox project! As generative AI transitions from research labs to production…
cncf.io
Deploying or scaling LLM inference? This is the room to be in. 📈 The vLLM Inference Meetup hits Boston on March 31! Join us for an evening of deep technical sessions, live demos, and real conversations with the community. 📅 Mar 31, 5PM 📍 314 Main St, Cambridge 🔗 luma.com/4rmkrrb7
vLLM Inference Meetup · Boston · Luma
Deep technical sessions. Live demos. Real conversations. If you're deploying, or scaling LLM inference, this is the room to be in. Join Red Hat AI, IBM,…
luma.com
LLMInferenceService is now fully production-ready and built on the high-performance @llm-d.ai framework. 𝗪𝗵𝗮𝘁’𝘀 𝗶𝗻𝗰𝗹𝘂𝗱𝗲𝗱? - KV-cache aware routing and disaggregated prefill-decode to maximize throughput.
Watch this preview of distributed tracing (llm-d 0.6) and Prefix Cache-Aware Routing. 🔹 State Tracking: llm-d tracks KV cache via ZMQ. 🔹 Smart Scoring: EPP pods tokenize prompts and query to find cached blocks. 🔹 Optimal Routing: Reqs go to the pod for the best cache hit.
The first llm-d NYC Meetup is live! Deep dive into the open-source stack for cloud-native inference with @IBMResearch, @AMD, and @RedHat State-aware scheduling & KV cache reuse P/D disaggregation Scaling MoE models AMD ROCm & llm-d Watch: www.youtube.com/watch?v=_ZBQ...
llm d NYC 2026 Meetup
YouTube video by llm-d Project
youtube.com
📢 𝗧𝗵𝗲 𝗦𝘁𝗮𝘁𝗲 𝗼𝗳 𝗠𝗼𝗱𝗲𝗹 𝗦𝗲𝗿𝘃𝗶𝗻𝗴 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝗶𝗲𝘀: 𝗠𝗮𝗿𝗰𝗵 𝗘𝗱𝗶𝘁𝗶𝗼𝗻 𝗶𝘀 𝗼𝘂𝘁! We launched our newsletter publicly last year to share our contributions to upstream communities from our Red Hat AI teams. We’ve gained over 𝟭𝟯𝟬𝟬 𝘀𝘂𝗯𝘀𝗰𝗿𝗶𝗯𝗲𝗿𝘀!