State Of Vllm 2026 Inferact Ray Summit 2026

Summary

Agentic production traffic brings long multi-turn sessions, heavy prefix reuse, and bursty, heterogeneous requests that challenge ... Large language models demand significant computational power for inference, and TPUs are becoming a first-class target for ... How fast can you actually stand up GPU capacity, and what breaks along the way? At AI Infra Training trillion-parameter MoE models on long-horizon agentic tasks demands that inference and optimization scale ... llm-d brings production-grade orchestration and distributed inference optimizations to Serving LLMs for agentic workloads at Chase scale means balancing latency, cost, and resilience. The team cut latency from ...