Tag: inference
All the articles with the tag "inference".
vLLM Quantization
How vLLM loads quantized checkpoints, quantizes BF16 models at load time, picks kernels, and which format to choose (FP8, INT8, INT4, MXFP4, KV cache).
How vLLM Sleep Mode Works
Free and restore vLLM's GPU memory without restarting the process — Level 0/1/2, how CuMemAllocator does it, usage, and the speedup.