Characterization of Gaussian Universality Breakdown in High-Dimensional Empirical Risk Minimization