04b Theory and Analysis

For years, the way large language models handle inference has been stuck inside a box — literally. The high-bandwidth RDMA networks that make modern LLM serving work have confined both prefill and decode to the same datacenter, sometimes even the same rack. A team of researchers at Moonshot AI and Tsinghua University is making the case that this constraint is about to break down — and that the right architecture can already exploit that shift.

The research team introduces Prefill-as-a-Service (PrfaaS), a cross-datacenter serving architecture that selectively offloads long-context prefill to standalone, compute-dense prefill clusters and transfers the resulting KVCache over commodity Ethernet to local PD clusters for decode. The result, in a case study using an internal 1T-parameter hybrid model, is 54% higher serving throughput than a homogeneous PD baseline and 32% higher than a naive heterogeneous setup — while consuming only a fraction of available cross-datacenter bandwidth. The research team note that when compared at equal hardware cost, the throughput gain is approximately 15%, reflecting that the full 54% advantage comes partly from pairing higher-compute H200 GPUs for prefill with H20 GPUs for decode.

Training a modern large language model (LLM) is not a single step but a carefully orchestrated pipeline that transforms raw data into a reliable, aligned, and deployable intelligent system. At its core lies pretraining, the foundational phase where models learn general language patterns, reasoning structures, and world knowledge from massive text corpora. This is followed by supervised fine-tuning (SFT), where curated datasets shape the model’s behavior toward specific tasks and instructions. To make adaptation more efficient, techniques like LoRA (Low-Rank Adaptation) and QLoRA (Quantized LoRA) enable parameter-efficient fine-tuning without retraining the entire model.

Alignment layers such as RLHF (Reinforcement Learning from Human Feedback) further refine outputs to match human preferences, safety expectations, and usability standards. More recently, reasoning-focused optimizations like GRPO (Group Relative Policy Optimization) have emerged to enhance structured thinking and multi-step problem solving. Finally, all of this culminates in deployment, where models are optimized, scaled, and integrated into real-world systems. Together, these stages form the modern LLM training pipeline—an evolving, multi-layered process that determines not just what a model knows, but how it thinks, behaves, and delivers value in production environments.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.