Optimal Splitting of Language Models from Mixtures to Specialized Domains
Skyler Seto, Pierre Ablin, Anastasiia Filippova, et al.
This paper addresses how to optimally train multiple specialized language models for different domains by determining how to split computational resources between general pretraining and domain-specific fine-tuning. The authors develop a method using scaling laws to predict model performance and find the best allocation of computing power, showing improvements in reasoning and knowledge tasks across different model sizes.
language modelspretrainingmulti-domain specializationscaling laws