·Python · Gymnasium · NumPy

Machine Learning

Reinforcement-learning experiments on classic control.

github.com/Sebastoshi/machine-learning · Playing with ML ↗

The questions

How far can simple, tabular reinforcement learning get on classic control before you need function approximation?

◇

How it builds up

Each block answers a piece of the question and feeds the next — foundations at the top, the answer at the bottom.

flowchart TD
  Q(["How far does tabular RL get<br/>on classic control?"]):::q
  A["random agent<br/>baseline"]
  B["tabular Q-learning<br/>discretized states"]
  C["Q-learning swing-up<br/>the hard cart-pole task"]
  Q --> A --> B --> C --> ANS(["where tabular methods<br/>start to break down"]):::a
  classDef q fill:#1a1206,stroke:#f0a83e,color:#f3c98a;
  classDef a fill:#f0a83e,stroke:#f0a83e,color:#10141a;

◇

Inside the build

Each stage of the calculation. Click one to expand its full write-up and validation figure.

01random agent — calibration

A do-nothing baseline that sets the floor — what reward you get with no learning at all.

02tabular Q-learning — the workhorse

Classic value iteration on a discretized state space — learns a control policy from reward alone on the standard control environments.

03Q-learning swing-up — the stress test

The cart-pole swing-up task, where the pole starts hanging down — a harder, sparser problem that pushes tabular discretization to its limits.

◇

What it shows

Tabular Q-learning solves the easy control tasks cleanly, but the swing-up problem exposes where discretized value tables strain — the natural motivation for function approximation and deep RL.