问题提出#

在讲解什么是curiosity之前，我们得先认识到强化学习的两大主要问题

稀疏奖励问题#

most rewards do not contain information, and hence are set to zero.

Remember that RL is based on the reward hypothesis, which is the idea that each goal can be described as the maximization of the rewards. Therefore, rewards act as feedback for RL agents; if they don’t receive any, their knowledge of which action is appropriate (or not) cannot change.

For instance, in Vizdoom, a set of environments based on the game Doom “DoomMyWayHome,” your agent is only rewarded if it finds the vest. However, the vest is far away from your starting point, so most of your rewards will be zero. Therefore, if our agent does not receive useful feedback (dense rewards), it will take much longer to learn an optimal policy, and it can spend time turning around without finding the goal. curiosity2

环境奖励函数的定义#

in each environment, a human has to implement a reward function. But how we can scale that in big and complex environments 即在实际的复杂的环境中，人为很难定义合适的奖励函数

解决方案#

A solution to these problems is to develop a reward function intrinsic to the agent

This intrinsic reward mechanism is known as Curiosity because this reward pushes the agent to explore states that are novel/unfamiliar. To achieve that, our agent will receive a high reward when exploring new trajectories. 也就是说,智能体的内在的奖励机制，会去奖励那些具备好奇心的智能体，也就是选择新环境或者复杂环境

常见的intrinsic reward函数 are different ways to calculate this intrinsic reward. The classical approach (Curiosity through next-state prediction) is to calculate Curiosity as the error of our agent in predicting the next state, given the current state and action taken.

也就是预估值与实际值的差异的绝对值

curiosity3

还有一个常见的方法是random network distillation.

If the agent spends a lot of time on these states, it will be good at predicting the next state (low Curiosity). On the other hand, if it’s in a new, unexplored state, it will be hard to predict the following state (high Curiosity).