State Of Vllm 2026 Inferact Ray Summit 2026

Summary

Agentic production traffic brings long multi-turn sessions, heavy prefix reuse, and bursty, heterogeneous requests that challenge ... Large language models demand significant computational power for inference, and TPUs are becoming a first-class target for ... Training trillion-parameter MoE models on long-horizon agentic tasks demands that inference and optimization scale ... Only about 5% of scientific hypotheses can be verified cheaply. The rest require real experiments, which makes most of science ... How fast can you actually stand up GPU capacity, and what breaks along the way? At AI Infra As LLMs grow in size, context length, and architectural complexity, Serving LLMs for agentic workloads at Chase scale means balancing latency, cost, and resilience. The team cut latency from ...