August 2007
A good programmer working intensively on his own code can hold itin his mind the way a mathematician holds a problem he's workingon. Mathematicians don't answer questions by working them out onpaper the way schoolchildren are taught to. They do more in theirheads: they try to understand a problem space well enough that theycan walk around it the way you can walk around the memory of thehouse you grew up in. At its best programming is the same. Youhold the whole program in your head, and you can manipulate it atwill.
That's particularly valuable at the start of a project, becauseinitially the most important thing is to be able to change whatyou're doing. Not just to solve the problem in a different way,but to change the problem you're solving.
Your code is your understanding of the problem you're exploring.So it's only when you have your code in your head that you reallyunderstand the problem.
It's not easy to get a program into your head. If you leave aproject for a few months, it can take days to really understand itagain when you return to it. Even when you're actively working ona program it can take half an hour to load into your head when youstart work each day. And that's in the best case. Ordinaryprogrammers working in typical office conditions never enter thismode. Or to put it more dramatically, ordinary programmers workingin typical office conditions never really understand the problemsthey're solving.
Even the best programmers don't always have the whole program they'reworking on loaded into their heads. But there are things you cando to help:
It's striking how often programmers manage to hit all eight pointsby accident. Someone has an idea for a new project, but becauseit's not officially sanctioned, he has to do it in off hours—whichturn out to be more productive because there are no distractions.Driven by his enthusiasm for the new project he works on it formany hours at a stretch. Because it's initially just anexperiment, instead of a "production" language he uses a mere"scripting" language—which is in fact far more powerful. Hecompletely rewrites the program several times; that wouldn't bejustifiable for an official project, but this is a labor of loveand he wants it to be perfect. And since no one is going to seeit except him, he omits any comments except the note-to-self variety.He works in a small group perforce, because he either hasn't toldanyone else about the idea yet, or it seems so unpromising that noone else is allowed to work on it. Even if there is a group, theycouldn't have multiple people editing the same code, because itchanges too fast for that to be possible. And the project startssmall because the idea issmall at first; he just has some coolhack he wants to try out.
Even more striking are the number of officially sanctioned projectsthat manage to do all eight things wrong. In fact, if you look atthe way software gets written in most organizations, it's almostas if they were deliberately trying to do things wrong. In a sense,they are. One of the defining qualities of organizations sincethere have been such a thing is to treat individuals as interchangeableparts. This works well for more parallelizable tasks, like fightingwars. For most of history a well-drilled army of professionalsoldiers could be counted on to beat an army of individual warriors,no matter how valorous. But having ideas is not very parallelizable.And that's what programs are: ideas.
It's not merely true that organizations dislike the idea of dependingon individual genius, it's a tautology. It's part of the definitionof an organization not to. Of our current concept of an organization,at least.
Maybe we could define a new kind of organization that combined theefforts of individuals without requiring them to be interchangeable.Arguably a market is such a form of organization, though it may bemore accurate to describe a market as a degenerate case—as whatyou get by default when organization isn't possible.
Probably the best we'll do is some kind of hack, like making theprogramming parts of an organization work differently from the rest.Perhaps the optimal solution is for big companies not even to tryto develop ideas in house, but simply tobuy them. But regardlessof what the solution turns out to be, the first step is to realizethere's a problem. There is a contradiction in the very phrase"software company." The two words are pulling in opposite directions.Any good programmer in a large organization is going to be at oddswith it, because organizations are designed to prevent whatprogrammers strive for.
Good programmersmanage to get a lot done anyway. But often itrequires practically an act of rebellion against the organizationsthat employ them. Perhaps it will help if more people understand that the wayprogrammers behave is driven by the demands of the work they do.It's not because they're irresponsible that they work in long bingesduring which they blow off all other obligations, plunge straight intoprogramming instead of writing specs first, and rewrite code thatalready works. It's not because they're unfriendly that they preferto work alone, or growl at people who pop their head in the doorto say hello. This apparently random collection of annoying habitshas a single explanation: the power of holding a program in one'shead.
Whether or not understanding this can help large organizations, itcan certainly help their competitors. The weakest point in bigcompanies is that they don't let individual programmers do greatwork. So if you're a little startup, this is the place to attackthem. Take on the kind of problems that have to be solved in onebig brain.
Thanksto Sam Altman, David Greenspan, Aaron Iba, Jessica Livingston,Robert Morris, Peter Norvig, Lisa Randall, Emmett Shear, Sergei Tsarev,and Stephen Wolfram for reading drafts of this.