Scaling LLM-Based Ranking Systems for Latency-Critical Search & Recommendation Workloads

Large Language Models are powerful — but deploying them in latency-critical ranking systems is a fundamentally different problem than building chat applications.

At LinkedIn, we scaled LLM-based ranking systems to power high-throughput, low-latency search and recommendation workloads such as Job Search and People Search. These systems must score hundreds of candidates per query under strict latency budgets while serving millions of users globally.

This talk presents how we designed and scaled a prefill-only LLM ranking architecture using SGLang, optimized for throughput, cost efficiency, and predictable latency. I will cover key production learnings, inference optimizations, infrastructure decisions, and the trade-offs required to make LLM ranking viable at scale.

We will also discuss how we upstreamed our production optimizations to the open-source SGLang ecosystem.