Q-learning

How Q-learning Works

The Q-table starts at zero (or small random values). The agent then acts in the environment, balancing exploration – trying actions whose value it does not yet know – against exploitation, repeating actions that have paid off.

After every action, one cell of the table is updated:

\[ Q(s, a) = Q(s, a) + \alpha \times [r + \gamma \times \max_{a'}Q(s', a') - Q(s, a)] \]

Where:

• \(Q(s, a)\) is the value of taking action \(a\) in state \(s\).
• \(r\) is the reward received, and \(s'\) is the state that followed.
• \(\gamma\) is the discount factor, which sets how much future rewards count.
• \(\alpha\) is the learning rate.

“Model-free” means the agent never needs to know how the environment behaves in advance. It learns action values purely from experience.

Example of Q-learning

Put an agent in a maze with a reward at the exit. Early on the Q-table is all zeros, so the agent wanders at random and hits dead ends. Every step feeds the update rule.

Cells along routes that eventually reach the exit gain value, because the reward propagates backward through the max term, one update at a time. After enough episodes the table encodes the maze’s solution: from any square, the highest-valued action points toward the exit, and the agent walks the shortest route without ever having seen a map.

Related AI terms: Reinforcement Learning · Learning Rate · AI Agent · Machine Learning

Did you like the Q-learning gist?

Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.