What to study - a fourth option
This is a short post I thought I'd share, inspired by a conversation I had with someone recently about choosing courses to study during university. The three main reasons I usually saw people (including myself) choosing courses were (in some order):
- How much they enjoyed the course
- How good they were at the course
- How directly useful the course would be (for future courses / university paths, or for careers)
When I looked back at which courses I found had been most helpful for me, I didn't see much correlation with any of those three courses (except for possibly the first one). The common thread among courses that I remember today is that the overall flavour of the course (i.e. the kinds of problems it offered, and the kinds of reasoning it made me practice) was much more predictive of how useful I'd eventually find it.
I currently work in mechanistic interpretability, and a lot of the kinds of reasoning I find myself practicing on a daily basis came from some of these courses. To give a few examples:
- Automata and Formal Languages
This was a great course, because it presented objects and systems governed by a set of rules, and required you to think about what these objects were doing intuitively. Grokking this was usually most of the difficulty, and was the key to solving most problems. - Numerical Analysis
This was also very useful, because it presented me with a series of tools and explanations of how and why they work, and required that I use the right tools in different circumstances. This required some degree of creativity and resourcefulness. It also helped that most of the situations could be visualised in terms of fundamnetal physical concepts, like the wave or head equations. - Complex Methods
This is a course I got virtually no use out of. Unlike numerical methods, I felt like there was nothing tying the problems and tools we were given to any easily visualisable or intuitive concepts. Figuring out which tool to use for the situation at hand felt like pattern-matching, not puzzle-solving.
There are two obvious difficulties here - that of knowing which courses are relevant for which kinds of thinking, and knowing which kinds of thinking are most relevant for you. The first difficulty is more easily overcome - just speak to people who have already studied the course (ideally people who have also tried a variety of other courses, so they have a point of comparison). You can probably also get pretty far from skimming over example exercises, and seeing the kinds of steps that usually lead to a solution. The second difficulty is harder to overcome - I didn't know I'd be working in mechanistic interpretability research during my undergrad, and I wasn't even sure I'd be working in AI safety. That said, I think I could still have done better backchaining from the ways I wanted to self-improve, to pick the courses that would have been more optimal for my overall development.
Anyway, the main point I'm trying to make here is - don't study complex methods. It sucks.