The Surprising Effectiveness of Approximate Value Iteration in Self-Play