In development · 2026
DQN Pac-Man Agent
Deep Q-Network learning Pac-Man from raw state, with a PPO comparison planned.
The problem
Pac-Man punishes greedy policies: reward is sparse, ghosts make the environment non-stationary, and naive Q-learning oscillates.
What I built
A Deep Q-Network with replay buffer, target network, and epsilon-greedy scheduling learning from raw state, set up so a PPO baseline can be compared under identical conditions.
Stability before score
The first runs diverged, as DQN usually does without help. The replay buffer and a periodically-synced target network are the two changes that turned a noisy value function into one that trends.
Why add PPO
A value-based and a policy-gradient method failing differently on the same environment is more informative than one method scoring well. The comparison runs under identical reward shaping and wall-clock budget.
Outcome
- Replay buffer + target network for stability
- Epsilon-greedy schedule tuned for sparse reward
- PPO comparison planned against the DQN baseline
Stack
Data flow
DQN training loop
- 1
Environment
Pac-Man, raw state observations
- 2
Replay buffer
Decorrelates sequential experience
- 3
Q-network
PyTorch value estimator
- 4
Target network
Periodically synced, stabilizes bootstrapping
- 5
Epsilon-greedy
Scheduled exploration decay
- 6
PPO baseline
Planned side-by-side comparison
This diagram is generated from portfolio.ts. Edit the `architecture` field to change it.
Keep reading
Peptide AI RAG and Agent Platform
Hybrid retrieval, agent trajectories, and a privacy-safe semantic cache in production.
Case studySynthDrive
You describe a driving scenario in plain English, and it generates a 3D world, runs object detection on it, and scores how hard that scene is to perceive.
Case studyPanNote + PipelineEvolve
A Chrome extension that turns Panopto lectures into structured notes, plus a research harness that tries to improve the note pipeline automatically.
Case studyWant the deeper version of this?
I am happy to walk through the tradeoffs, the failure modes, and what I would do differently.