Maximum-Entropy Exploration with Future State-Action Visitation Measures

This paper proposes a new way for reinforcement learning agents to explore by rewarding them for visiting diverse state-action combinations in the future. The key innovation is using a mathematical bound to show that encouraging exploration of future possibilities helps the agent discover more varied behaviors, and the authors develop a practical algorithm that can learn this exploration bonus even from past experiences. Experiments show this method helps agents explore more efficiently within single episodes, though it doesn't significantly improve actual task performance.