j4orz

Course Philosophy

I was working on a bunch of lectures just now. I was reminded back to my days of 231n just how much work it is to create some of these materials and make them good. The amount of iteration and thought and you go down blind alleys and just how much you change it. So creating something good in terms of educational value is really hard and it's not fun. — Karpathy

When people think about education, they think more about what I would say is a softer component of diffusing knowledge. I have something very hard and technical in mind. In my mind, education is the very difficult technical process of building ramps to knowledge... I want to figure out how to build these ramps very efficiently so that people are never stuck and everything is always not too hard or not too trivial, and you have just the right material to progress...I feel like education is..a tangle of understanding and you're trying to lay it out in a way that creates a ramp where everything only depends on the thing before it. — Karpathy

Curriculum Design as Engineering. 90%..accessible to 8billion people. https://www.youtube.com/watch?v=g0qrakAEGcQ

SITP "LLM 201n": The Hacker's Guide to the Internals of Machine Learning Systems

As a compiler writer for domain specific cloud languages, I became frustrated with the non-constructiveness and disjointedness of my learning experience in the discipline of machine learning systems, particularly with domain specific tensor languages. These course notes for The Structure and Interpretation of Tensor Programs is my personal answer to these frustrations. It's

1. the spiritual successor to LLM101n. You will develop your own deep learning framework teenygrad (sharing 90% of its abstractions with tinygrad) in order to run training and inference for the exact same nanogpt and nanochat models developed in LLM101n.
2. inspired by the whirlwind tour form of SICP/HTDP/PAPL¹ applied to substance of training neural networks, hacking deep learning frameworks, and programming massively parallel processors.
3. opensource book creating an omakase learning experience with the teenygrad codebase and the best high level intuitive visualizers from across the world

Contents

• in part 1 you implement a multidimensional Tensor and accelerated BLAS kernels.
• in part 2 you implement .backward() and accelerated cuBLAS kernels for the age of research.
• in part 3 you implement a fusion compiler with OpNode graph IR for the age of scaling.

If you empathize with some of my frustrations, you may benefit from the course too.
I've intentionally designed it so that the only pre-requisite required is the ability to program with highschool-level math.

Good luck on your journey.
Are you ready to begin?