Teaching objectives
How can a machine learn to play without anyone teaching it the strategy? This lab for lower and upper secondary (ESO and Bachillerato) answers that question with the most beautiful and concrete idea in artificial intelligence: reinforcement learning. We rebuild, step by step, Donald Michie’s matchbox machine (MENACE) and Martin Gardner’s Hexapawn robot: drawers full of counters that, by rewarding the moves that lead to a win and penalising those that lead to a loss, end up discovering the perfect move on their own. At the end we take a leap: a game where there is no fixed optimal move because the best choice depends on what the other player does.
What you'll learn
The intuition that a machine can learn from experience by adjusting probabilities, without being programmed with the strategy: just repeated reward and punishment.
- Block A — The stones game (rooms 1-9): first students play against a random opponent (winning is easy), then against a perfect-play AI to discover that there is a winning strategy (leave the opponent on a multiple of 4: 4 and 8). Then the mechanism is taken apart step by step: how the AI chooses a drawer (sum of counters → random number → counting drawer by drawer) in one position and in all six; the danger of subtracting 1 (penalising good moves for later mistakes empties drawers), which motivates multiplying by a coefficient; the student plays against the learning AI; and finally the AI plays against itself, with the time animation of convergence, the learning rate and curriculum learning.
- Block B — Hexapawn (rooms 10-14): the same principle on Gardner’s 3×3 game, but adding the other side of the coin: the complete game tree and the minimax (maximin) algorithm. The key conclusion: the boxes learn on their own the same move that minimax computes directly — learning and searching arrive at the same place.
- Block C — Teamwork (rooms 15-18): a cooperative 2-player game with 3 strategies (make an effort, do little, do nothing) where the team passes if the joint effort reaches a threshold N. Here there is no single optimal move: two AIs learning at the same time can converge to fair cooperation, to an unfair split of the work (free-riding) or to collapse, depending on the threshold. It is game theory in action: from the volunteer’s dilemma (N=2) to the stag hunt (N=4).
Key mathematical ideas
- Reinforcement learning: reward/punishment on actions to maximise expected reward, with no prior model of the game.
- Probabilistic selection (roulette wheel): the probability of each move is proportional to the number of counters in its drawer; exploration versus exploitation.
- Multiplicative reinforcement and learning rate: multiplying the contents of the drawer by a coefficient (>1 rewards, <1 punishes) never empties it completely; the coefficient is the learning rate, and setting it badly breaks convergence.
- The stones game and the strategy of multiples of 4: the losing positions of the 10-stone game (take 1-3, whoever takes the last one wins) are the multiples of 4.
- Game tree and minimax: each player maximises their result assuming the opponent minimises theirs; the value is propagated from the leaves to find the outcome under perfect play.
- Cooperative games with a threshold: the volunteer’s dilemma and the stag hunt, multiple equilibria (fair vs. unfair, cooperation vs. collapse) and the free-rider problem.
- Multi-agent self-play: several learners trained against each other; the behaviour (pure or mixed) emerges and can be visualised as the evolution of probabilities over time.
Room-by-room contents
Room 1 · The 10-stone game
The rules are introduced: there are 10 stones and, taking turns, each player takes 1, 2 or 3; whoever takes the last one wins. The student plays against an opponent that decides at random and finds that winning is very easy.
Student tasks
- Play several games against the random opponent.
- Notice that beating a random player is almost always easy.
Room 2 · Is there a winning strategy?
Now the opponent is a perfect-play AI; sometimes it starts and sometimes the student does. After 4 games the message “Not so easy to beat the AI now, is it?” appears, and the student is asked to spot how many stones the opponent must be left with in order to win (buttons from 1 to 10).
Student tasks
- Play 4 games against the perfect AI and pay attention to its moves.
- Mark the losing positions: 4 and 8 (leave the opponent on a multiple of 4).
Room 3 · How does the AI pick a drawer?
The step students never get to see: how the AI chooses a drawer. In a single position, with 3 drawers of counters, a slow animation adds up all the counters (+700 ms), draws a random number between 1 and the sum (+700 ms), counts the counters drawer by drawer (200 ms each) and lights up the drawer where the number falls. Then the student solves 3 cases: given an arrangement of counters and a number, choose the right drawer.
Student tasks
- Watch the roulette-wheel selection animation step by step.
- Pick the right drawer in the 3 cases given.
Room 4 · Play the AI, step by step
The student plays with the Take 1/2/3 buttons against the box AI (whole counters, 3 per drawer so that the effect is more marked). The AI’s move is narrated very slowly — sum of counters, random number, counting drawer by drawer and lighting up the drawer — and recorded in a transcript (6 → AI takes 2 → 4 → you take 3 → …). When someone wins, the drawers the AI used are recalculated by ±1 counter, move by move, and stay lit. After the first game the student is asked what has happened to the probability of each strategy. In later games the counting speeds up so that it does not become tedious.
Student tasks
- Play a full game following the AI’s narrated move.
- Watch the ±1 recalculation and answer what happened to the probabilities.
Room 5 · The danger of subtracting 1
A first learning idea with whole counters: after a loss, remove 1 counter from each drawer used. The demo reveals its flaw: a correct move (taking 2 from position 6) is penalised because of a later mistake. Applied 5 times, the drawer of the correct move empties and the AI never plays it again. Hence the idea of multiplying by a coefficient instead of subtracting.
Student tasks
- Watch how the correct drawer in position 6 empties after 5 losses.
- Understand why it is better to multiply than to subtract.
Room 6 · Play the AI that learns
Now the student plays against the box AI, and the AI always starts. After each game, instead of subtracting, the drawers are multiplied by a coefficient: by more than 1 for the moves of the winner and by less than 1 for those of the loser (they never reach 0). The AI also learns from its opponent’s moves: if the student wins, the moves the student made are reinforced. After 6 games, the student predicts what the AI will play at the start (6 stones) and with 3 stones, and compares it with the policy it has actually learnt. It is not possible to continue until at least 7 games have been played.
Student tasks
- Play at least 7 games against the AI (it starts).
- Predict the AI’s most likely move with 6 and with 3 stones and compare with what it has learnt.
Room 7 · The AI plays itself
With the same view of the boxes, there are buttons to “play once”, “play 5 times” (with 1 s between games) and “play 20 times”. A time animation shows how the probability of each strategy evolves in each position over the games, until it converges to the correct move (leave a multiple of 4).
Student tasks
- Run self-play batches of different sizes.
- Follow in the graph how each position converges (or not) to a strategy.
Room 8 · The learning rate
The room explains why it is better to multiply by a coefficient than to subtract counters: that way a drawer never reaches 0 and never becomes “dead”. That coefficient is the learning rate. A slider is used to try extreme values and see their effects: too high and the AI gets stuck on a hasty strategy; too low and it does not converge.
Student tasks
- Try high and low coefficients and watch the convergence.
- Find a reasonable range for the learning rate.
Room 9 · Curriculum learning
Two AIs are trained by self-play from 30 stones, given the SAME total number of moves (this is the fair comparison, because curriculum games, with few stones, are shorter). One always trains from 30; the other follows a curriculum, from few stones up to 30. A strip of 30 squares per AI shows which positions it has learnt (green) and which it has not (grey). With the same effort the curriculum learns quite a few more positions (~90% against ~75%), because always playing from 30 leaves many positions undertrained. Curricula are key in huge games (chess, Go).
Student tasks
- Train both AIs and compare their strips of learnt positions.
- Reason out why, with the same number of moves, the curriculum learns more.
Room 10 · Gardner’s Hexapawn
Hexapawn is introduced: a 3×3 board with 3 pawns per side that move as in chess (they advance straight ahead and capture diagonally). A player wins by promoting a pawn on the opposite row or by leaving the opponent with no moves. The student plays against a random opponent.
Student tasks
- Learn how the pawns move and capture.
- Play several games against chance.
Room 11 · The game tree
The room explains in detail what the game tree is: all possible games drawn together, where each arrow is a move and each branch, from top to bottom, a complete game. The initial board blinks and the student expands the nodes by tapping them. Each leaf says who wins (“Red has no moves: White wins”). Task: find the shortest and the longest game (3 and 7 moves).
Student tasks
- Expand the tree and follow complete branches.
- Find the shortest game (3) and the longest (7) and write down how many moves each has.
Room 12 · Minimax
The minimax strategy is explained in simple words: each player chooses what is best for them, assuming the opponent also plays perfectly. A branch of perfect play is shown (boards in a row, coloured green = the player to move wins / red = loses), with a star on the key move that Red must find. Two multiple-choice tasks: who makes the key move and who wins with perfect play (Red, the 2nd player).
Student tasks
- Identify in the branch who makes the key move marked with a star.
- Work out who wins Hexapawn with perfect play by reading the colour of the initial board.
Room 13 · The boxes learn Hexapawn
The box AI plays Red (2nd player) against a random White and learns with multiplicative reinforcement (×1.3 for a win, ×0.75 for a loss). It is trained in batches; ALL of Red’s possible moves in the key position are DRAWN with their live WEIGHT (they change during training), and a graph shows the win rate going up. A “For the curious” spoiler holds the full mathematics of the reinforcement (E[Δln w]: the weight grows if the move wins more than ~52% of the time).
Student tasks
- Train in batches and watch how the weights of each move in the key position change.
- Check that the optimal move ends up dominating and the win rate goes up.
Room 14 · Can you beat the trained AI?
The student plays White against the AI that has learnt to play (it moves Red with perfect play, minimax). Since Red (2nd player) wins with perfect play, there is no way to beat it: the room lets the student try and, after the loss, explains why. A new-game button allows another attempt.
Student tasks
- Play against the perfect AI and try to win with White.
- Understand that with perfect play the 2nd player (Red) always wins.
Room 15 · Learning is searching (end of the block)
The close of the Hexapawn block. First a test with 3 tricky questions (what changed INSIDE the AI when it learnt, why the trained AI and minimax play the same, what happens to a good move that lost that time). Once they are solved, “What we have learnt” is revealed, along with the key link: ALL neural networks learn by adjusting weights, with a link to the Neural Networks lab.
Student tasks
- Answer the 3 questions of the initial test.
- Read the summary and, if wished, go on to the Neural Networks lab.
Room 16 · The teamwork game
Two players choose their effort at the same time: make an effort (2), do a little (1) or do nothing (0). They pass if the sum of their efforts reaches the threshold N. Each one gets 3 for passing (or loses 1 for failing) minus their own effort. With N=2 the 3×3 payoff matrix is explored and the temptation to free-ride appears.
Student tasks
- Explore the payoff matrix and find the combinations that pass.
- See why each player would rather the other one did the work.
Room 17 · Two AIs learning (N=2)
Each player has its own drawer with the 3 strategies and chooses by probability; the two AIs play against each other and reinforce by multiplying. A stacked strip chart (one pixel per round) shows how each player’s probabilities evolve. Buttons to play once or continuously (very fast). After enough rounds a conclusion appears that diagnoses the outcome: fair cooperation, an unfair split (free-riding) or collapse.
Student tasks
- Start self-play (once or continuously) and watch the strips.
- Read the conclusion: what equilibrium does the team reach?
Room 18 · Raise the bar (N=3, N=4)
Training is repeated with higher thresholds, and it is not possible to continue until both N=3 AND N=4 have been tried. With N=3 the split usually becomes unfair: one AI sees that the other always makes an effort and takes advantage of it (free-riding). With N=4 (a teacher who demands perfection) the team only passes if both play (2,2); since it is so hard to coincide, the AIs give up because their effort is wasted too often, and the team collapses. The emerging behaviour is compared across thresholds.
Student tasks
- Train the AIs with N=3 and with N=4 (both are required) and read the conclusion for each case.
- Compare whether cooperation, an unfair split (free-riding) or giving up/collapse emerges depending on N.
Room 19 · Mixed strategies (theatre or football?)
A coordination game (battle of the sexes) in the first person. The student chooses their favourite outing (the AI prefers the other one) and with what probability they will go to theirs. 1) “Play against the AI”: the AI COPIES the student’s level of insistence, so the split is always fair (being stubborn does not win; the extremes give 0). 2) Animated self-play: two AIs adjust their strategy so as not to be exploitable and converge stably to ≈2/3 (Nash equilibrium). 3) The student guesses the optimal mixed strategy with Nash’s hint (the partner must be indifferent to switching) and earns points: 15 for 66-67, 10 for 60-72, 5 for 50-80.
Student tasks
- Choose a favourite and a probability and play against the AI (check that the split is fair).
- Watch the self-play and guess the optimal probability (≈67%).
Room 20 · Learning is adjusting the weights
The close of the lab: learning on one’s own is not magic, it is adjusting the weights of the strategies with rewards and punishments. It shows only the conclusions (a single strategy: one weight dominates, as in the stones game or Hexapawn; a mixed strategy: the weights are spread; or it depends on the other player), a final sentence that connects with neural networks and a link to the Neural Networks lab. No question.
Student tasks
- Read the conclusions of the lab.
- If wished, go on to the Neural Networks lab.
Rooms to project
The most striking ones to show and discuss in class.