Elisa Ricci

Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearning

Lorenzo Orsingher, Thomas De Min, Massimiliano Mancini, et al.

SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models

Quentin Guimard, Federico Bartsch, Simone Caldarella, et al.

This paper introduces SEM (Sparse Embedding Modulation), a method to remove social and spurious biases from vision-language models like CLIP without hurting their performance. Instead of working directly with CLIP's dense embeddings where bias is tangled with useful information, SEM uses a Sparse Autoencoder to break down embeddings into cleaner, separate features, allowing it to precisely remove bias-related features while keeping task-relevant ones intact. The method shows strong improvements in fairness across multiple benchmarks without requiring retraining.