DA6400 · Introduction to Reinforcement Learning

Interactive RL Lab

Explore the GridWorld policy and browse the DQN and actor-critic experiments from Assignments 2 and 3.

Water empty · policy and state values

Lake / refillFire / goalSmokeBoulderOrdinary cell

Arrows show the optimal action; the small number is the estimated state value. The outlined cell is the selected start.

How to use the policy explorer

1 · Choose the task phase

Use Water status to switch between empty-handed and carrying water. The displayed policy is recomputed for that phase.

2 · Set the discount factor

Move the γ slider, then select Recompute policy. Higher γ values give more weight to future rewards.

3 · Read the policy

Each arrow is the optimal action for that cell. The small number is its estimated state value under the current phase and γ. Lake, fire, smoke, and boulder cells are marked.

4 · Choose a starting cell

Click a grid cell. The gold outline marks the selected start; the inner highlight shows the agent’s current position.

5 · Try the rollout

Select Step once to advance one action, or Run to animate. Moves are stochastic, so repeated rollouts may take different paths.

6 · Start over

Select Reset to return to your chosen cell. Change γ or water status to compare policies.

Assignment 2 · MountainCar-v0

Deep Q-Network experiments

Compare DQN training choices and replay strategies. These runs use Python and are not executed in the browser.

Vanilla DQN baseline

Train a DQN agent with replay memory, a target network, and ε-greedy exploration. In MountainCar, the agent must build momentum by moving back and forth before reaching the hilltop.

python Vanilla_DQN.py --auto

Run from Q3_2_Vanilla_DQN_Implementation. The auto pipeline trains, then generates plots and videos.

Episode truncation

Compare episode limits of 200, 1,000, and 2,000 steps and inspect how truncation changes learning and completion rates.

python main.py --mode truncation

Run from the Q3_3Vanilla_DQN_Truncate folder.

Replay-factor and sensitivity studies

Run the Assignment 2 suite for replay-factor comparisons and hyperparameter sensitivity. The suite also includes prioritized experience replay (PER).

cd assignment2/Codes
python run_all.py --num_seeds 2

Use 2 seeds for a quick check; omit the flag for the full 15-seed runs.

Run Assignment 2 locally

Clone the repository, open a terminal in assignment2/Codes, create a Python environment, install requirements.txt, then run the command above. The scripts generate logs and plots locally; training can take time.

Assignment 2 code Setup and experiment README

This tab is an experiment guide, not a hosted DQN runtime. The browser demo currently runs only the GridWorld policy explorer.

Assignment 3 · Deep RL and preference learning

SAC, DQN, and PEBBLE experiments

Browse the experiment families and their quick-run commands. Model training requires the Python environment described in the assignment README.

LunarLander · SAC vs DQN

Compare discrete SAC and DQN, plus hover-reward variants. Evaluate learning behavior and the effect of the reward design.

python run_q2_2.py --experiments sac_disc dqn --num-seeds 2 --parallel 2 --total-steps-disc 5000 --video-every 0 --checkpoint-every 0

Reacher · reward variants

Compare SAC with reward formulations Ra, Rb, and Rc, then inspect self-evaluation and cross-evaluation results.

python run_q2_3.py --num-seeds 2 --parallel 2 --total-steps 5000

PEBBLE · preference-based RL

Compare a SAC ground-truth baseline with PEBBLE under different preference-query budgets on Pendulum, and with reward teachers on Reacher.

python run_q3.py --experiments sac_gt pebble_1000 --num-seeds 2 --parallel 2 --pendulum-steps 5000 --reacher-steps 5000

Run Assignment 3 locally

Use Python 3.12 and install Implementation/shared/requirements.txt. Run the commands from the working directory specified in the assignment README; the README also documents full-run settings and expected runtimes.

Assignment 3 code Setup and experiment README

The hosted page summarizes these Python experiments; it does not run the SAC, DQN, or PEBBLE training jobs in your browser.