Inference scaling part 1. Starting with a modded text generation function (temperature scaling, top-p filtering, multinomial sampling) to generate diverse outputs for self-consistency and best-of-N (improving answer accuracy by>2x) 00:00 Introduction and recap 00:31…
Inference scaling part 1. Starting with a modded text generation function (temperature scaling, top-p filtering, multinomial sampling) to generate diverse outputs for self-consistency and best-of-N (improving answer accuracy by>2x) 00:00 Introduction and recap 00:31 Training-time and inference-time scaling 07:52 What we'll implement 11:47 Notebook setup and model loading 17:43 Building a flexible text generation function 24:40 Chain-of-thought prompting 28:26 Sampling and output diversity 33:43 Next-token logits and greedy decoding 38:20 Temperature scaling step by step 42:46 Softmax and token probabilities 47:42 Multinomial sampling 54:51 Adding temperature sampling to text generation 59:31 Top-p filtering step by step 1:10:23 Adding top-p filtering to text generation 1:13:43 Sampling and LLM watermarking 1:16:01 Self-consistency and majority voting 1:20:36 Implementing self-consistency 1:29:02 MATH-500 results 1:35:01 Accuracy and compute tradeoffs 1:36:50 Next steps and self-refinement
discussion1 selected reply
The tutorial walks through temperature and top-p sampling, self-consistency, and best-of-N generation, reporting more than a twofold accuracy improvement in its setup and explicitly flagging the associated compute trade-off. The result is tied to the described implementation and MATH-500 exercise.
@jatingargiitk This is only for a fixed model size here (0.6B so it works for most people, since it's a from-scratch approach). But it also works for bigger models. E.g., see DeepSeekMathV2, which is based on DeepSeek V3.2 (671B): https://t.co/GgZMZPNypU