Value#
Value function in RL#
Value function in RL refers to how much reward an agent will get until it arrives at the end state. For example, suppose you get one point each time you celebrate your birthday. And you have a 50 % chance of living to 100 years old, and 50 % of living to 110 years old. Then the value function would be
Warning
Incoming math!
Formally, value function can be determined by weighted sum of all rewards, and calculated by a recursive function:
where
Since the value function is defined as the rewards an agent receives along its life, value function at the end state is
Where
Relation between value function and rewards.#
Rewards are observed through actions with the environment, and value functions are defined as the expected summation of rewards until termination state is encountered.
Why value functions are important?#
Value functions provide an easy way to rank a state. A state with a higher value is expected to receive more rewards in the future, so it’s better. Intuitively, arriving at a state with a higher value means that there would be more rewards in the future, and that’s the foundation of RL: RL aims to maximize the rewards an agent would get, in other words, tries to modify an agent’s behavior so that it can be in a state where values are higher.