March 2024

Despite its title this isn't meant to be the best essay. My goal here is to figure out
what the best essay would be like.

It would be well-written, but you can write well about any topic. What made it
special would be what it was about.

Obviously some topics would be better than others. It probably wouldn't be about
this year's lipstick colors. But it wouldn't be vaporous talk about elevated themes
either. A good essay has to be surprising. It has to tell people something they don't
already know.

The best essay would be on the most important topic you could tell people
something surprising about.

That may sound obvious, but it has some unexpected consequences. One is that
science enters the picture like an elephant stepping into a rowboat. For example,
Darwin first described the idea of natural selection in an essay written in 1844. Talk
about an important topic you could tell people something surprising about. If that's
the test of a great essay, this was surely the best one written in 1844. And indeed,
the best possible essay at any given time would usually be one describing the most
important scientific or technological discovery it was possible to make. [1]

Another unexpected consequence: I imagined when I started writing this that the
best essay would be fairly timeless — that the best essay you could write in 1844
would be much the same as the best one you could write now. But in fact the
opposite seems to be true. It might be true that the best painting would be timeless
in this sense. But it wouldn't be impressive to write an essay introducing natural
selection now. The best essay now would be one describing a great discovery we
didn't yet know about.

If the question of how to write the best possible essay reduces to the question of
how to make great discoveries, then I started with the wrong question. Perhaps
what this exercise shows is that we shouldn't waste our time writing essays but
instead focus on making discoveries in some specific domain. But I'm interested in
essays and what can be done with them, so I want to see if there's some other
question I could have asked.

There is, and on the face of it, it seems almost identical to the one I started with.
Instead of asking what would the best essay be? I should have asked how do you
write essays well? Though these seem only phrasing apart, their answers diverge.
The answer to the first question, as we've seen, isn't really about essay writing. The
second question forces it to be.

Writing essays, at its best, is a way of discovering ideas. How do you do that well?
How do you discover by writing?
An essay should ordinarily start with what I'm going to call a question, though I
mean this in a very general sense: it doesn't have to be a question grammatically,
just something that acts like one in the sense that it spurs some response.

How do you get this initial question? It probably won't work to choose some
important-sounding topic at random and go at it. Professional traders won't even
trade unless they have what they call an edge — a convincing story about why in
some class of trades they'll win more than they lose. Similarly, you shouldn't attack
a topic unless you have a way in — some new insight about it or way of approaching
it.

You don't need to have a complete thesis; you just need some kind of gap you can
explore. In fact, merely having questions about something other people take for
granted can be edge enough.

If you come across a question that's sufficiently puzzling, it could be worth exploring
even if it doesn't seem very momentous. Many an important discovery has been
made by pulling on a thread that seemed insignificant at first. How can they all be
finches? [2]

Once you've got a question, then what? You start thinking out loud about it. Not
literally out loud, but you commit to a specific string of words in response, as you
would if you were talking. This initial response is usually mistaken or incomplete.
Writing converts your ideas from vague to bad. But that's a step forward, because
once you can see the brokenness, you can fix it.

Perhaps beginning writers are alarmed at the thought of starting with something
mistaken or incomplete, but you shouldn't be, because this is why essay writing
works. Forcing yourself to commit to some specific string of words gives you a
starting point, and if it's wrong, you'll see that when you reread it. At least half of
essay writing is rereading what you've written and asking is this correct and
complete? You have to be very strict when rereading, not just because you want to
keep yourself honest, but because a gap between your response and the truth is
often a sign of new ideas to be discovered.

The prize for being strict with what you've written is not just refinement. When you
take a roughly correct answer and try to make it exactly right, sometimes you find
that you can't, and that the reason is that you were depending on a false
assumption. And when you discard it, the answer turns out to be completely
different. [3]

Ideally the response to a question is two things: the first step in a process that
converges on the truth, and a source of additional questions (in my very general
sense of the word). So the process continues recursively, as response spurs
response. [4]
Usually there are several possible responses to a question, which means you're
traversing a tree. But essays are linear, not tree-shaped, which means you have to
choose one branch to follow at each point. How do you choose? Usually you should
follow whichever offers the greatest combination of generality and novelty. I don't
consciously rank branches this way; I just follow whichever seems most exciting;
but generality and novelty are what make a branch exciting. [5]

If you're willing to do a lot of rewriting, you don't have to guess right. You can follow
a branch and see how it turns out, and if it isn't good enough, cut it and backtrack. I
do this all the time. In this essay I've already cut a 17-paragraph subtree, in addition
to countless shorter ones. Maybe I'll reattach it at the end, or boil it down to a
footnote, or spin it off as its own essay; we'll see. [6]

In general you want to be quick to cut. One of the most dangerous temptations in
writing (and in software and painting) is to keep something that isn't right, just
because it contains a few good bits or cost you a lot of effort.

The most surprising new question being thrown off at this point is does it really
matter what the initial question is? If the space of ideas is highly connected, it
shouldn't, because you should be able to get from any question to the most
valuable ones in a few hops. And we see evidence that it's highly connected in the
way, for example, that people who are obsessed with some topic can turn any
conversation toward it. But that only works if you know where you want to go, and
you don't in an essay. That's the whole point. You don't want to be the obsessive
conversationalist, or all your essays will be about the same thing. [7]

The other reason the initial question matters is that you usually feel somewhat
obliged to stick to it. I don't think about this when I decide which branch to follow. I
just follow novelty and generality. Sticking to the question is enforced later, when I
notice I've wandered too far and have to backtrack. But I think this is the optimal
solution. You don't want the hunt for novelty and generality to be constrained in the
moment. Go with it and see what you get. [8]

Since the initial question does constrain you, in the best case it sets an upper bound
on the quality of essay you'll write. If you do as well as you possibly can on the
chain of thoughts that follow from the initial question, the initial question itself is the
only place where there's room for variation.

It would be a mistake to let this make you too conservative though, because you
can't predict where a question will lead. Not if you're doing things right, because
doing things right means making discoveries, and by definition you can't predict
those. So the way to respond to this situation is not to be cautious about which
initial question you choose, but to write a lot of essays. Essays are for taking risks.

Almost any question can get you a good essay. Indeed, it took some effort to think
of a sufficiently unpromising topic in the third paragraph, because any essayist's
first impulse on hearing that the best essay couldn't be about x would be to try to
write it. But if most questions yield good essays, only some yield great ones.

Can we predict which questions will yield great essays? Considering how long I've
been writing essays, it's alarming how novel that question feels.