Kevin HY Hau - Hello World 👋

Let the model learn, then download everything

The learner reads your games file and the games you played, copies what winners did, learns who tends to win, and plays some new games against itself. Then it measures itself against fixed opponents before and after, so you can see if it really improved. It runs on this page with the same Python code as in rlgames.zip.

Report

No learning run yet.

Reinforcement learning, and how this Arena's learner is different

Written by Claude, an AI assistant, at Kevin's request. Where it says "Claude", that is the AI that wrote the code and ran the tests.

Short version: reinforcement learning (RL) is learning by trying things and being told how it went, not by being told the right answer. This tab explains it from scratch, shows the main families, and then says honestly where the learner in this Arena is the same as the famous ones and where it is not.

1. The idea in one picture

• Agent = the player (here: the model). World = the game.
• The agent picks a move. The world answers with the new position and a reward (win +1, draw 0, loss −1).
• Repeat thousands of times. The agent slowly learns which moves lead to rewards. Nobody ever tells it "this move was good".

2. How it differs from other machine learning

3. The three hard parts of RL

• Delayed reward (credit assignment). In a 40-move game you only learn at the end that you won. Which of the 40 moves deserves the credit? Every RL method is a different answer to this.
• Explore or exploit. Keep playing the move that has worked (exploit), or try a new one that might be better (explore)? Only exploiting gets stuck; only exploring never improves.
• Making its own data. If the agent gets worse, the games it plays get worse, and it learns from those. Mistakes can feed on themselves.

4. The family tree of RL methods

Dates are approximate (first well-known paper or result). Claude has not run these other methods in this project; this table is background, not measurements.

5. Where the Arena sits

• It is in the "planning with search" family: a small copy of AlphaZero. The game rules are known, so the search can imagine moves ahead exactly (tab "How the code works", part 3).
• One small network with two heads: a policy head (a probability for each move) and a value head (who is winning).
• It learns by self-play: the model plays itself with search, then trains so that its quick guess matches what the search found and who really won.
• That answers the three hard parts: the search gives a better move than the raw guess (so credit is assigned move by move), random noise at the root of the search and a little variety in the moves it plays keep it exploring, and fixed opponents (random, greedy, plain search) check it is really improving.

6. How Claude's learner is different (the honest part)

■ Green = same as AlphaZero. ■ Orange = Claude's own addition, not a published algorithm.

• Why not just copy every game? Copying moves from a file is imitation learning (a kind of supervised learning). It can only get as good as the people or programs in the file. Self-play search can go beyond them.
• Why mix? AlphaGo (2016) used human games first and self-play after. Claude mixes them in the same round, so you can teach a model from your own games and let it check them by playing itself.
• Why 40%? In Claude's tests 30% to 50% worked as well as or better than 80%: copying winners too hard made models worse. This number is a hand-tuned setting, not a law.

7. What Claude's runs showed

• Self-play with search did learn in the full runs (score against random and greedy players near 1.0).
• The "learn from my games" step is much weaker evidence: one tiny Connect-4-mini test did not improve the model. Claude fixed what made it worse (copying weak players’ moves, and copying winners too hard at 80%), but cannot claim it makes a model stronger in general.
• Claude did not compare against other RL algorithms (Q-learning, PPO and so on), so there is no claim that this one is better or worse than them.
• Each row rests on only 8 to 30 games, so read trends, not single numbers.

8. Best of five: merging several runs into one model

• Idea: train five variants of the learner (full recipe, no root noise, no temperature, few searches, no symmetry views), score them all the same way, keep the best, and then merge all five into one student model.
• How the merge works (distillation): the five models answer together as a committee, and the better a model scored, the more its answer counts. A student network, which starts as a copy of the best model, is trained to give the committee's answer by itself.
• Not new science: ensembles (many models voting, like a random forest), distillation and "keep the winner" are old ideas. Claude only wrote a small readable version. Averaging the numbers of separately trained networks does not work, which is why distillation is used.
• Run it: python run.py best_of tictactoe from the downloaded package. It writes model.npz, the student and the best single model, a pooled games file for the Learn tab, and a report.
• Honest rule: the student is only chosen if it scores at least as well as the best single model. In a quick test on tic-tac-toe (20 iterations per variant, 20 games per check) it did not beat the best single model: they were level within noise. Tic-tac-toe is probably too small for this to help; a bigger game such as Connect 4 is where it could show.

How the code works

The package is called rlgames. It is a small copy of AlphaZero's idea, written in plain Python and NumPy so every line can be read. Download it here (it also runs this page).

1. The big picture: the learning loop

• Rules are the only game-specific part. Everything else works for any game that follows the same five-function shape.
• Self-play makes training data without any human: the model plays itself and keeps what happened.
• Fixed opponents never change, so a rising score means real progress, not just "better than yesterday's me".

2. A game is five things

A move is one number. Because moves and positions are plain numbers, the search can try a move, look at the result, and try another, without any picture or screen.

3. Choosing one move: tree search (MCTS)

• The network gives a quick guess: which moves look good (the "prior" P) and who is winning (the "value" v).
• The search then checks the guess by imagining moves ahead. It picks the child with the best Q + c·P·√N/(1+n): Q is how well that move has gone so far, P the network's hunch, and the last part nudges it to try moves it has not visited much.
• When the imagined line ends, the value is added back up the line, and its sign flips at every step, because my gain is your loss.
• After N simulations the move that was visited most is played. More simulations means stronger but slower.

4. The network and how it learns

• It is a small "multi-layer perceptron": a few layers of numbers multiplied together, with negative values cut to zero (ReLU).
• Two heads share it: the policy head (a probability for every move) and the value head (one number from −1 to +1: who is winning).
• The targets come from self-play: the policy target is what the search chose; the value target is who actually won.
• Backpropagation is written out by hand in rlg/net.py (about 70 lines). The optimiser is Adam. A learning step looks at a small random batch of positions, each shown through a random rotation or mirror (symmetry), so what it learns in one corner it knows in all.

5. What this page does with your files

• The page loads Pyodide: Python and NumPy compiled to run inside your browser, in a background worker so the screen never freezes.
• It then loads rlgames.zip (the same file you can download) and calls rlg/web_api.py.
• Files are checked before use: the model must be plain float numbers whose sizes fit one of the games (no pickled code); every move of every game is replayed through the real rules, and games with an illegal move are dropped.

6. How "learn from my games" works exactly

• For every position in a game, the value target is who won from that player's point of view.
• The move target: if the player who moved went on to win, copy their move (40% of the weight on it, the rest on what the network already thought). If the game was a draw, half as strongly. If they lost, leave the network's own opinion alone.
• Games you played count 3× by default, then self-play games are added and the network trains on the mix.
• Moves made by the weak fixed opponents (random, greedy) in a games file are never copied: a lucky win teaches nothing.
• Before and after, the model plays fixed opponents (random, greedy, and perfect for tic-tac-toe) and also the model you gave it, so you can see if it got better.
• What Claude measured: a Connect 4 model learning from its own run's later games beat the original in head-to-head matches (about 0.7 to 0.9 over 36 games, 4 tries) while self-play alone gave about 0.55. A tiny Connect 4 mini model from a very short run did not improve. So learning from a file helps when the file is large and from a reasonably trained model; it is not magic.

7. Where everything lives

8. A few lines you can read

The whole forward pass of the network:

def forward(self, X):
    hs = [X]; a = X
    for W, b in zip(self.W, self.b):
        a = np.maximum(0, a @ W + b); hs.append(a)        # layer: multiply, add, cut negatives to 0
    return hs, a @ self.Wp + self.bp, np.tanh(a @ self.Wv + self.bv)[:, 0]   # policy scores, value (-1..+1)

The choice inside the search (who to explore next):

q = np.divide(node.W, node.N, out=np.zeros_like(node.W), where=node.N > 0)    # how well has each move gone so far
i = int(np.argmax(q + c_puct * node.P * math.sqrt(tot + 1) / (1 + node.N)))   # plus the network's hunch, plus curiosity

One self-play game, and what it keeps:

while game.result(s) is None:
    legal, visits = search(game, s, ev, sims, rng, noise=sw['dirichlet'])   # think ahead
    a = pick(legal, visits, T, rng)                                         # choose (random early on, best later)
    hist.append((game.encode(s), pi, s.turn)); s = game.step(s, a)          # remember the position and the search's opinion
# at the end every position is labelled with who won: +1 / 0 / -1

9. Add your own game

• Write a class with the five functions in rlg/user_games.py (about 60 lines). python run.py lab yourgame then measures it and train can teach a network (add a settings line in rlg/train.py).
• This page plays only the games listed on it, because it needs a board picture for each.

Results from Claude's own runs

Score = wins plus half the draws, out of 1. Single rows rest on 8 to 30 games: read trends, not single numbers. A GPU does not help here: about 80% of the time is search and game rules in Python; the network is under 20%.

Why AI invented games

I asked AI to code chess, shatranj, shogi, xiangqi, draughts, dama, Go, Othello, backgammon, mancala, Connect 4 and more. Each one taught AI a trick. Then I asked a different question: what happens if AI summarised all these experience and start innovate new kind of game? Chapters 59 and 60 are the answer.

Chapter 59: Flock

A game with one kind of stone and three rules, where a stone's power depends on its neighbours.

Self-played summary

• Result: a depth-3 computer beat a depth-1 computer 12–0 on 6×6 and 14–0 on 5×5. In 600 random games per board, Light won 248 and Dark 237 (5×5), and Light 220 and Dark 242 (6×6): side advantage looks small.
• How the computer thinks: alpha-beta search, captures tried first, a score of 100 per stone plus a little for range and mobility. Easy = depth 1 plus noise; Normal 3; Hard 5; Master up to 9 within 2.4 s.
• Honest limit: there are no human experts for a new game, so the levels are only as good as the self-play that tuned them.

Chapter 60: Runes

Claude's second invented game, with dice. The number you roll chooses which old game's magic you may cast.

What self-play changed

• Honest limits: the computer mostly steps and Leaps; Charm and Summon are rarely used because they need a 5 or 4 and a matching board. 40 games cannot prove whether Light has a first-move edge (depth-2 against depth-2 split 24–16). Runes has dice, so the learner on this page cannot play it yet.

Seven lessons that held for every game

1. Keep the rules in one place that never draws. Easy to test with a bot, easy to put a new screen on.
2. A move is one small value (an integer or a short path) you can list, compare, undo and send to a search.
3. make and unmake beat copying in a search, if a test proves undo restores the exact position.
4. Tune by self-play, not by feeling. Claude's first Flock was drawn to death; one rule fixed it, and Claude only knew because it counted the games.
5. Repetition is the enemy of every perfect-information game. Decide how yours ends before you build it.
6. Show the rule on the board (the range number on a Flock stone) so players learn without reading.
7. Say honestly what is untested. A new game has no experts; the levels are only as good as the tuning.

Full write-ups are in the wiki pages for Chapters 59 and 60. You can load a Flock model and play it on this page.