Pierre Ablin

Scaling Categorical Flow Maps

Oscar Davis, Anastasiia Filippova, Pierre Ablin, et al.

Optimal Splitting of Language Models from Mixtures to Specialized Domains

Skyler Seto, Pierre Ablin, Anastasiia Filippova, et al.

This paper addresses how to optimally train multiple specialized language models for different domains by determining how to split computational resources between general pretraining and domain-specific fine-tuning. The authors develop a method using scaling laws to predict model performance and find the best allocation of computing power, showing improvements in reasoning and knowledge tasks across different model sizes.