In defense of blub studies

Sometimes people ask me what they should learn to become a better programmer. I feel like the default recommendation here is usually an obscure programming language or a textbook on some high-powered machinery like ML. So I always feel a little bit embarrassed and boring when I instead suggest going really deep on what you already know: your main programming language, web framework, object-relational mapper, UI library, version control system, database, Unix tools, etc. It’s not shiny or esoteric, but for me, building a detailed mental model of those (and how they compare to alternatives) might be the learning that’s contributed most to my effectiveness as an engineer.

A coworker coined the phrase “blub studies” to refer to this sort of mundane, ultra-specific-seeming knowledge. “Blub” comes from a Paul Graham essay, Beating the Averages, in which Blub is a hypothetical middlebrow language whose programmers get defensive when Graham asserts that Lisp is superior. Blub studies is the study of what goes on in the guts of these boring, everyday systems—not the kind you get tenure for inventing, but the kind people actually use.

Blub studies is a never-ending treadmill of engineering know-how. It’s the fiddly technical details of how Git stores data, or how Postgres locking semantics caused your migration to bring down prod, or why pip install failed this time. It’s what goes on inside the boiler rooms of your computer. There’s a seemingly infinite amount of it, full of bespoke details for you to stumble over, and that makes it, often, unbelievably frustrating. Experts in shiny fields like machine learning write shiny-sounding articles like A theory of the learnable; experts in blub studies emit screeds like The Law of Leaky Abstractions and Programming Sucks.

In short, if you’re in search of generalizable knowledge that compounds exponentially over time, then blub studies looks like the crap you have to wade through to get to the good stuff. So it’s easy to see why people give up on understanding all the blub they’re surrounded by, except what they need to get the job done.

But for me, the opposite attitude has been more productive. Computers can be understood—even if it’s hard and takes a while. Blub studies is more generalizable than it seems, and has its own way of compounding over time, too. That makes it a lot more useful than you’d expect.^{✻✻ Of course, there are useless parts of blub studies: if this essay gets you excited to memorize a bunch of command-line flags, consider reversing this advice. But in my experience, it’s more common to neglect the useful parts of blubs, than to over-index on trivia.}

The most straightforward benefit of blub expertise is that it saves you time. “You can’t apply those brilliant insights you learned from SICP if you don’t have the knowledge base and emotional fortitude to fight through pip install first.” If you know how Git’s internal model works, you can get your repository out of its borked state without spending hours on Stack Overflow.

This effect is larger than it might seem. If you’re working with a system you don’t understand, you’re limited to debugging via guess-and-check, which can be arbitrarily slow. A more efficient method would be to get as much information as possible about your program’s execution and then use that information to exclude most of the hypothesis space. But this requires a good understanding of both the system, and the tools available for inspecting it. If you’re tracking down, say, a networking problem, staring at some tcpdump output will often get you most of the way there, but only if you know how to interpret it and what to look for.

If you spend half your programming time debugging, and being a blub expert lets you debug twice as fast, then just the speed gain from blub expertise will let you increase your output by a third.<sup>†† If you think “half of programming time debugging” sounds high, imagine how much faster you’d be if all your code worked the first time.

Doubling debugging speed is probably a conservative estimate—you can save pretty much unlimited time via things like “hmm, 40 milliseconds sounds like the timeout for Nagle’s algorithm, try setting TCP_NODELAY”. I somewhat frequently debug tricky things 5x+ faster than coworkers, just because I’ve been working with our stack for a long time, so I know where to look for problems and how to quickly test hypotheses.</sup> That justifies a lot of time staring at tcpdump output! But there are also more subtle reasons I’ve gotten so much from blub studies. It’s both more general, lasts longer, and has more of a compounding effect, than I expected.

Blub studies are surprisingly broadly applicable because, even if you’re learning about the details of some specific blubby system, that system’s design will contain a juicy non-blubby core of extractible general principles. Unlike many “general principles” people try to teach you, the ones you learn via blub studies are guaranteed to be important to at least one real-world system (the one you’re learning about). And you’ll see them realized in all their messy detail, which academic presentations often leave out.

Suppose your blub of choice is React. You might worry that learning the gory details will be useless if you ever move to a different part of the stack, or even a different web framework. And, yes, some of them will. But the core idea of React—writing pure render functions, using reconciliation to make updates fast—is extremely powerful and general. In fact, it’s now been copied by the next generation of UI frameworks on both iOS (SwiftUI) and Android (Jetpack Compose). Learning the principles behind React makes it easier to learn those other frameworks. In fact, it can even be a useful source of ideas to “import” from one to the other. At Wave, for instance, we’ve gotten a lot of mileage out of importing ideas from Relay into our mobile apps.

This is a good example of an idea that, as far as I know, you can only learn about through blub studies. Academia didn’t give much attention to React-style UI programming. In fact, it doesn’t seem to view user-interface programming paradigms as a particularly interesting object of study at all. People do sometimes publish on it but, for instance, I couldn’t find any courses on it in MIT’s extensive course catalog.^{‡‡ You could argue that this is because UI programming is “too applied” and one shouldn’t expect it to be covered in an academic curriculum. But computer science covers many other equally-“applied” areas, like networking, databases, operating systems, and graphics.}

Blub studies also compound more than you’d naively expect, in two ways. First, knowing about one blub makes it easier to learn about alternative blubs that serve the same purpose—like the React/SwiftUI example above. Second, knowing more about one blub helps you learn blubs in adjacent parts of the stack more quickly.

Once, while pair programming with a more junior coworker, we were writing a complicated SQLAlchemy query. My coworker used user.name (the name field of an object stored in the user variable) instead of User.name (the name field of the class User) and was wondering why her query gave the wrong results. I tried to explain the “magic” by which User.name was an instance of Column while user.name was a simple str. I went around in circles for a little while until I eventually explained Python’s descriptor protocol to her (the language feature SQLAlchemy uses to enable the “declarative” ORM syntax). At that point, everything clicked—and I realized that Python’s __dunder__ methods are the key to decoding quite a lot of “magical” seeming code. If you learn the Python language features well, lots of complicated libraries will become a lot easier to understand.

I had a similar experience myself with Kubernetes. The first time I tried to learn it, it was a bewildering morass of jargon—all those namespaces and containers and Pods and Deployments and Services and Ingresses just to get a simple HTTP server running! Then I read a networking textbook and everything made much more sense. The (arguably) most complicated parts of Kubernetes exist to solve networking-related problems—allowing hundreds of containers to talk to each other independently while hosted on a much smaller set of computers—so the networking textbook gave me a schema onto which I could hang all my Kubernetes factoids. Once I knew how Linux’s IP routing, iptables, and network namespaces worked, it was much easier for me to understand what exactly something like “kube-proxy” was doing.

If you know enough different blubs, you can end up at the point where you don’t even need to look things up to figure out how they’re (probably) implemented. An experienced Python programmer can guess immediately how SQLAlchemy’s “declarative” ORM works under the hood. That’s the point when your blub expertise will really start compounding—almost as soon as you start working with something new, you’ll start figuring out how it works and extracting the kernel of generally-interesting ideas.

Because of this compounding effect, the most important step toward becoming a blub master is to kickstart your “blub flywheel”—the virtuous cycle of blub accumulation—however you can. That means starting with whichever blubs are the easiest or most motivating to learn, and branching out from there. For me, the easiest place to start has been with blubs I’m already using at my day job. I have a couple strategies for getting the most out of those.

First, I’ll try to go deeper than necessary. If I really want to ship something, it’s easy to give into temptation to, say, Google an error message, copy-paste a fix from Stack Overflow, and move on with my day. But it often doesn’t take that much longer to actually read the error message, understand what it means, and try to figure out why that Stack Overflow answer fixed my problem. Similarly, if I’m stuck in a tricky yak shave, I’ll bias against “guess-and-check” style debugging in favor of getting a better understanding of the system I’m trying to debug. It doesn’t always feel worth it to, e.g., dive into the docs of tcpdump and iptables rules to track down my weird one-off networking issue—but over time I’ve run into enough “weird one-off networking issues” that it’s paid off many times over.

The second part of my blub flywheel is to pay attention to magic. Whenever I’m working with something new, I try to continuously update my best-guess mental model of how it’s implemented. “Okay, the docs are telling me to create an Ingress, I guess this is probably the widget that provisions a load balancer to talk to my backend containers?” If I realize I’m wrong, I’ll dig in and update. “Hmm, I can’t ping those pods from outside the cluster, so how could the load balancer be talking to them? Aha—it’s talking to the nodes, and there’s a NodePort Service as a second layer of indirection.” If I have no idea at all how something could work, that usually means it’s time to read a book.

One thing I wish other engineers would do to make blub studies easier is to produce more and better “advanced” documentation of their software. Most mature and widely-used libraries have great tutorials and introductory content, but far fewer make it easy to get past the “copy-pasting examples” stage. I think this is mostly a matter of organization and navigation; if I comb through, say, the SQLAlchemy docs, I can actually find enough information to piece together a good mental model of the system, but it’s scattered among many different pages and it’s not clear what order I should read them in. The React docs are probably the best I know of here, but still leave a lot out.

Over time, by consistently exploring the guts of anything I’m working with that seems magical, I’ve built up a broad base of knowledge about how various technical systems work. This helps me in tons of different ways. It makes it easier to track down tricky bugs across many layers of the stack. I can learn new languages and libraries quickly by pattern-matching them to what I already know. It gives me better ideas for software designs, by imitating other systems I’ve seen, or by reusing ideas or tools I’ve heard of in a different context. Maybe most importantly, it gives me the confidence that, if I run into a tricky problem, I can learn enough to solve it, instead of feeling like I’m at the mercy of a system too complex to hope to understand.

So if you’re looking to learn something that will make you a better, and happier, programmer, ask yourself which parts of your most-used blub seem magical to you, and try to understand how they work. You’ll learn more than you think!

Thanks to David Golden, Awais Hussain, Will Larson, Dan Luu, Kevin Lynagh, Caleb Ontiveros, and Lincoln Quirk for commenting on a draft of this post.