Jiayuan Ye

Generalization in LLM Problem Solving: The Case of the Shortest Path

Yao Tong, Jiayuan Ye, Anastasia Borovykh, et al.

Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts

Jiayuan Ye, Vitaly Feldman, Kunal Talwar

Optimal Splitting of Language Models from Mixtures to Specialized Domains

Skyler Seto, Pierre Ablin, Anastasiia Filippova, et al.

This paper addresses how to optimally train multiple specialized language models for different domains by determining how to split computational resources between general pretraining and domain-specific fine-tuning. The authors develop a method using scaling laws to predict model performance and find the best allocation of computing power, showing improvements in reasoning and knowledge tasks across different model sizes.