What's the Catch? Evaluating Temporal Consistency in Vision-Language Models