Why "Optimal" isn't always "Safe": Q-Learning vs. SARSA.
This classic problem highlights the difference between two famous algorithms. The agent (blue circle) must get from Start to Goal. The bottom row is a "Cliff" (huge penalty). The agent has a 10% chance of slipping (noise), meaning it might move down even if it tries to move right.
Q-Learning (Off-Policy):
This algorithm assumes it will act optimally in the future. It finds the mathematically shortest path: walking right along the edge of the cliff. However, because of the 10% slip chance, it frequently falls off. It is "optimistic" but risky.
SARSA (On-Policy):
This algorithm learns from the policy it is actually using, including the exploration slips. It realizes that walking near the edge is dangerous because a slip leads to death. Consequently, it learns a "safe" path through the middle of the room. It takes longer to reach the goal, but it rarely crashes.