Training

A 70B-parameter model does not fit on an 80GB GPU, and the reason is not the parameters. In mixed-precision training with Adam you carry roughly sixteen bytes p…

Synthetic data can fill gaps, protect privacy, and create rare examples. It can also duplicate a teacher model's blind spots at industrial scale. The question i…

Alignment techniques are easy to flatten into slogans: humans teach the model, or a constitution teaches the model. The real difference is the source and shape …