Visualizing how long-term rewards propagate through an environment.
This grid demonstrates Value Iteration, a core dynamic programming algorithm. The goal is to calculate the "Value" of every square (state). The value represents "how much future reward can I expect if I start here?"
Initially, the agent only knows that the Diamond gives +10 points and the Fire gives -10. All other white squares have a value of 0. When you click "Step", the values "leak" into neighboring cells. This follows the Bellman Equation: a cell's value is updated based on the maximum value of its neighbors.
Why does the green fade?
Notice that the green color gets paler the further you are from the diamond. This is due to the Discount Factor (Gamma = 0.9). A reward of 10 today is worth more than a reward of 10 tomorrow. If you are 5 steps away from the diamond, the value is roughly $10 \times 0.9^5$, which is why the color fades.