Denoised RL Starter Pack: a Curated Shortlist of Reinforcement Learning Resources
Selected guides from industry veterans. You can’t learn AI by reading AI-generated material.
57% of content on the internet right now is AI generated. The prediction is that by 2026, that number will be 90%.
This article won’t be another “Awesome RL” list where you save and forget. Instead, I aim to provide a short (ideally less than 5) list of really great resources to guide anyone from begging to capable.
Criteria of Selection
- Author’s approach should be friendly to reader, meaning they convey core ideas in a gradual way.
- RL is math heavy, but author should show his insights above classic 101 stuff or “rigorous maths”.
- Author should have theoretical as well as piratical background. RL is used to solve real problems.
- Selected materials should also cover theoretical and practical (like environment setup and demo codes) aspects.
The List
- OpenAI’s SpinningUp RL Education Document: online document form, 2019.
It’s launched as a serious educational resource around 2018, when RL went kind of viral with success of AlphaGo, AlphaZero and Dota2. All documents are carefully written by Joshua Achiam, a research scientist from OAI and previously PhD from UC Berkeley.
There’re three main contents in it, each has its own merit:
1) A succict introduction to RL with good balance of concepts and maths;
2) a curated list of important papers organized by topic;
3) an essay about how to grow into an RL research role.
The introduction also comes with a library of core RL algorithms (with intentionally less implementation tricks).
2. UCL Course on RL by David Silver: Lecture videos form (with slides), 2015.
David Silver is both a principal research scientist at DeepMind and a professor at UCL, project lead of AlphaGo, AlphaZero, AlphaStar, etc. This course is a collaboration between two institutions in 2015.
It’s a rare opportunity to attend lectures given by real academical AND industrial leader, at the peak performance stage of his career.
3. <Reinforcement Learning: An Introduction> by Sutton: textbook form, 2018
Sutton is considered one of the founders of modern computational reinforcement learning. His book is considered “bible” of the field but still very friendly. The book is also the main textbook of the above UCL course.
Feel free to take a look of Chapter one to get a feeling of Sutton’s way of deliver insights from introductory materials.
4. HuggingFace’s Deep-RL-Course: Online interactive form, 2022
The main selling point IMO is it comes with lots of example codes, directly runnable on colab. It also cuts each chapter into short sections, conveys information in a visually rich way. This is quite useful to readers with ADHD.
5. Andrej Karpathy’s blog post to introduce RL: short essay form, 2016
This is more of a “fun read” instead of comprehensive resources. Andrej wrote this after he got into RL for a year after his PhD from Standford, before going to Tesla.
In the blog post, he show you deadly-simple RL on ATARI game with only 130-line numpy-only Python codes. You could appreicate the most bare metal form of RL. He also shared his experience in learning RL and lots of hands-on advices to new comers.
That’s it, exactly 5 (strictly 4) resources, in different form and different angles, but all from GOAT authors.
Honorable Mentions
- Library for realiable, off-the-shelf RL algorithms: https://github.com/DLR-RM/stable-baselines3
- UC Berkeley RL Course with demo codes and homeworks: https://rail.eecs.berkeley.edu/deeprlcourse/
- Get prepared for frustrations in actually training RL: http://amid.fish/reproducing-deep-rl
