SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models

This paper introduces SEM (Sparse Embedding Modulation), a method to remove social and spurious biases from vision-language models like CLIP without hurting their performance. Instead of working directly with CLIP's dense embeddings where bias is tangled with useful information, SEM uses a Sparse Autoencoder to break down embeddings into cleaner, separate features, allowing it to precisely remove bias-related features while keeping task-relevant ones intact. The method shows strong improvements in fairness across multiple benchmarks without requiring retraining.