A Reasonable Approximation

The Elm community is not "very active"

I sometimes see people describe the Elm community as "very active". For example:

• Elmcraft says "The Elm community is very active."
• "Is Elm dead?" says "the community is more active than ever." (I guess that's compatible with "but still not very active", but, well, subtext.)
• Some /r/elm commenter says "The community is very active and productive." (They later clarify: "It's very active compared to similar projects. Tight focus, feature complete. People tend to compare it to dissimilar projects.")

Is this true? Let's get some statistics, including historical ones, and see what comes up.

I think all this data was collected on October 28 2024. I'm splitting things out by year, so we can roughly extrapolate to the end of 2024 by multiplying numbers by 1.2.

Subreddit

I could export 951 posts from reddit. (That's a weird number; I suspect it means there were 49 posts I couldn't see.) The oldest were in December 2019, which means I have about 4⅚ years of data. In a given calendar year, I can easily count: how many posts were there? And how many comments were there, on posts made in that calendar year? (So a comment in January 2022 on a post from December 2021, would be counted for 2021.)

By either measure, 2024 has about a quarter of 2020 activity levels.

Discourse

I got a list of every topic on the discourse, with its creation date, "last updated" date (probably date of last reply in most cases), number of replies and number of views. The first post was in November 2017.

The "C" columns count according to a post's creation date, and the "U" columns count by "last updated" date.

So posts and replies have fallen to about 35% of 2020 levels. Views have fallen to about 15%.

Package releases

For every package listed on the repository, I got the release dates of all its versions.¹ So we can also ask how frequently packages are getting released, and we can break it up into initial/major/minor/patch updates.

My understanding is: if a package has any version compatible with 0.19, then every version of that package is listed, including ones not compatible with 0.19. If not it's not listed at all. So numbers before 2019 are suspect (0.19 was released in August 2018).

Package releases have declined by about half since 2020. Initial (0.48x) and major (0.49x) releases have gone down slightly faster than minor (0.56x) and patch (0.53x) ones, which might mean something or might just be noise.

Where else should we look?

The official community page gives a few places someone might try to get started. The discourse and subreddit are included. Looking at the others:

• The twitter has had two posts in 2024, both in May.
• Elm weekly, from a quick glance, seems to have had more-or-less weekly posts going back many years. Kudos!
• I once joined the slack but I don't know how to access it any more, so I can't see how active it is. Even if I could, I dunno if I could see how active it used to be.
• I don't feel like trying to figure out anything in depth from meetup. I briefly note that I'm given a list of "All Elm Meetup groups", which has six entries, of which exactly one appears to be an Elm meetup group.
• I'm not sure what this ElmBridge thing is.

For things not linked from there:

• I'm aware that Elm Camp has run twice now, in 2023 and 2024, and is planning 2025.
• There's an Elm online meetup which lately seems to meet for approximately four hours a year.
• There are nine other Elm meetups listed on that site. Most have had no meetups ever; none of them have had more than one in 2024.
• The Elm radio podcast hasn't released an episode in 2024. Elm town seems roughly monthly right now.
• hnhiring.com shows generally 0-1 Elm jobs/month since 2023. There was also a dry spell from April 2019 to November 2020, but it's clearly lower right now than in 2018, 2021 and 2022.
• Elmcraft is a website that I think is trying to be a community hub or something? Other than linking to Elm Weekly and the podcasts, it has a "featured article" (from April) and "latest videos" (most recently from 2023).
• How many long-form articles are being written about Elm? Not sure if there's an easy way to get stats on this.
• The elm tag on StackOverflow has had 6 questions asked this year, 22 in 2023. I couldn't be bothered to count further back.
• Github also lets you search for "repositories written partially in language, created between date1 and date2". So e.g. elm repos created in 2023. Quick summary: 2.6k in 2019; 2.3k in 2020; 1.7k in 2021; 1.5k in 2022; 1.2k in 2023; 731 so far in 2024 (extrapolate to 877).²

My take

I do not think the Elm community is "very active" by most reasonable standards. For example:

• I think that if someone asks me for a "very active" programming language community they can join, and I point them at the Elm community, they will be disappointed.
• I think that if I were to look for an Elm job, I would struggle to find people who are hiring.
• I couldn't find a single Elm meetup that currently runs anywhere close to monthly, either online or anywhere in the world.
• It seems like I could read literally every comment and post written on the two main public-facing Elm forums, in about ten minutes a day. If I also wanted to read the long-form articles, when those are linked… my guess is still under twenty minutes?

If you think you have a reasonable standard by which the Elm community counts as "very active", by all means say so.³

I think the idea that the Elm community is "more active than ever" is blatantly false. (That line was added in January 2022, at which point it might have been defensible. But it hasn't been removed in almost three years of shrinking, while the surrounding text has been edited multiple times.)

To be clear, none of this is intended to reflect on Elm as a language, or on the quality of its community, or anything else. I do have other frustrations, which I may or may not air at some point. But here I just intend to address the question: is it reasonable to describe the Elm community as "very active"?

(I do get the vibe that the Elm community could be reasonably described as "passionate", and that that can somewhat make up for a lack of activity. But it's not the same thing.)

Book review: the Iliad

Translated by Emily Wilson

1.

I didn't know what the Iliad was about. I thought it was the story of how Helen of Troy gets kidnapped, triggering the Trojan war, which lasts a long time and eventually gets settled with a wooden horse.

Instead it's just a few days, nine years into that war. The Greeks are camped on the shores near Troy. Agamemnon, King of the Greeks, refuses to return a kidnapped woman to her father for ransom. (Lots of women get kidnapped.) Apollo smites the Greeks with arrows which are plague, and after a while the other Greeks get annoyed enough to tell Agamemnon off. Achilles is most vocal, so Agamemnon returns that woman but takes one of Achilles' kidnapped women instead.

Achilles gets upset and decides to stop fighting for Agamemnon. He prays to his mother, a goddess, wanting the Greeks to suffer ruin without him. She talks to Zeus, who arranges the whole thing, though Hera (his wife and sister) and Athena aren't happy about it because they really want Troy sacked.

So Zeus sends a dream to Agamemnon, telling him to attack Troy. Agamemnon decides to test the Greek fighters, telling them it's time to give up and sail home. So they start running back to the ships, but Athena tells Odysseus to stop them, which he does mostly by telling them to obey orders.

There's a bunch of fighting between Greeks and Trojans, and bickering among the gods, and occasionally mortals even fight gods. In the middle of it there's a one-day truce, which the Greeks use to build a wall, complete with gates. Poseidon says it's such a good wall that people will forget the wall of Troy, which upsets him because they didn't get the gods' blessing to build it.¹ Agamemnon tries to convince Achilles to fight again by offering massive rewards, including the woman he stole earlier, whom he swears he has not had sex with as is the normal way between men and women.

Eventually the Trojans fight past the wall and reach the Greek fleet. At this point Patroclus, Achilles' bff (not explicitly lover, but people have been shipping them for thousands of years), puts on Achilles' armor and goes to fight. He's skilled and scary enough that the Trojans flee. Achilles told him not to chase them, but he does so anyway, and he's killed by Hector. Hector takes the weapons and armor, and there's a big fight for his body, but the Greeks manage to drag it back to their camp.

When Achilles finds out he's pissed. The god Hephaestus makes him a new set of weapons and armor, Agamemnon gives him the massive reward from earlier, and he goes on a killing spree. He briefly gets bored of that and stops to capture twelve Trojan kids, but quickly resumes. Eventually all the Trojans except Hector flee into Troy. He kills Hector and drags the body around the city then back to camp. There's a funeral for Patroclus, which includes killing those twelve kids, but then every morning he just keeps dragging Hector's body around Patroclus' pyre. The gods don't let it get damaged, but this is still extremely poor form.

Eventually the gods get Hector's father Priam to go talk to him. Hermes disguises himself as a super hot Greek soldier to guide Priam to Achilles' tent. Priam begs and also offers a large ransom, and Achilles returns the body and gives them twelve days for a funeral before the fighting restarts. Most of this is gathering firewood.

2.

Some excerpts.

Diomedes meets Glaucus on the battlefield, and asks his backstory to make sure he's not a god. Glaucus tells the story of his grandfather Bellerophon, and Diomedes realizes that his own grandfather had been friends with Bellerophon:

The master of the war cry, Diomedes,
was glad to hear these words. He fixed his spear
firm in the earth that feeds the world, and spoke
in friendship to the shepherd of the people.

"So you must be an old guest-friend of mine,
through our forefathers. Oeneus once hosted
noble Bellerophon inside his house
and kept him as his guest for twenty days.
They gave each other splendid gifts of friendship.
Oeneus gave a shining belt, adorned
with purple, and Bellerophon gave him
a double-handled golden cup. I left it
back home inside my house when I came here.
But I do not remember Tydeus.
He left me when I was a tiny baby,
during the war at Thebes when Greeks were dying.
Now you and I are also loving guest-friends,
and I will visit you one day in Argos,
and you will come to visit me in Lycia,
whenever I arrive back home again.
So let us now avoid each other's spears,
even amid the thickest battle scrum.
Plenty of Trojans and their famous allies
are left for me to slaughter, when a god
or my quick feet enable me to catch them.
And you have many other Greeks to kill
whenever you are able. Let us now
exchange our arms and armor with each other,
so other men will know that we are proud
to be each other's guest-friends through our fathers."

With this, the two jumped off their chariots
and grasped each other's hands and swore the oath.
Then Zeus robbed Glaucus of his wits. He traded
his armor with the son of Tydeus,
and gave up gold for bronze—gold armor, worth
a hundred oxen, for a set worth nine.

At one point Hera dolls herself up to make Zeus want to sleep with her. This is the flirting game of Zeus, greatest of all the gods:

"You can go later on that journey, Hera,
but now let us enjoy some time in bed.
Let us make love. Such strong desire has never
suffused my senses or subdued my heart
for any goddess or for any woman
as I feel now for you. Not even when
I lusted for the wife of Ixion,
and got her pregnant with Pirithous,
a councillor as wise as any god.
Not even when I wanted Danae,
the daughter of Acrisius, a woman
with pretty ankles, and I got her pregnant
with Perseus, the best of warriors.
Not even when I lusted for the famous
Europa, child of Phoenix, and I fathered
Minos on her, and godlike Rhadamanthus.
Not even when I wanted Semele,
or when in Thebes I lusted for Alcmene,
who birthed heroic Heracles, my son—
and Semele gave birth to Dionysus,
the joy of mortals. And not even when
I lusted for the goddess, Queen Demeter,
who has such beautiful, well-braided hair—
not even when I wanted famous Leto,
not even when I wanted you yourself—
I never wanted anyone before
as much as I want you right now. Such sweet
desire for you has taken hold of me."

3.

It feels kinda blasphemous to say, but by modern standards, I don't think the Iliad is very good. Sorry Homer.

Not that there's nothing to like. I enjoyed some turns of phrase, though I don't remember them now. And it was the kind of not-very-good that still left me interested enough to keep listening. I'm sure there's a bunch of other good stuff to say about it too.

But also a lot that I'd criticize, mostly on a technical level. The "what actually happens" of it is fine, but doesn't particularly stand out. But the writing quality is frequently bad.

A lot of the fighting was of the form: Diomedes stabbed (name) in the chest. He was from (town) and had once (random piece of backstory), and he died. (Name) was killed by Diomedes, who stabbed him in the throat; his wife and children would never see him again. Then Diomedes stabbed (name) in the thigh, …

So I got basically no sense of what it was like to be on the battlefield. How large was it? How closely packed are people during fighting? How long does it take to strip someone of their armor and why isn't it a virtual guarantee that someone else will stab you while you do? The logistics of the war are a mystery to me too: how many Greeks are there and where do they get all their food? We're told how many ships each of the commanders brought, but how many soldiers and how many servants to a ship?

I also had very little sense of time, distance, or the motivations or powers of the gods.

There were long lists that bored me, and sometimes Homer seemed to be getting paid by the word—lots of passages were repeated verbatim, and a few repeated again. (Agamemnon dictates the reward he'll offer Achilles to a messenger, then the messenger passes it on to Achilles, and we're told it again after Patroclus' death.)

We're often told that some characters are the best at something or other, but given little reason to believe it. Notably, Hector is supposedly the most fearsome Trojan fighter, but he doesn't live up to his reputation. He almost loses one-on-one against Ajax before the gods intervene to stop the battle; he gets badly injured fighting Ajax again; and even after being healed, he only takes down Patroclus after Patroclus has been badly wounded by someone else. And Achilles is supposed to be the fastest runner, but when he chases after Hector, he doesn't catch up until Hector stops running away. Lots of people are described as "favored by Zeus" but Zeus doesn't seem to do jack for them.

Even when the narrative supports the narrator, it feels cheap. Achilles is supposedly the best Greek fighter, and when he fights, he wins. So that fits. But how did he become so good at it? Did he train harder? Does he have some secret technique? Supernatural strength? We're not told. (His invulnerability-except-heel isn't a part of this story, and half of everyone is the son of some god or goddess so that's no advantage to him. To be fair: a reader points out that Superman stories don't necessarily give his backstory either.)

The poem is in iambic pentameter, with occasional deviations at the beginnings and ends of lines. I guess that's technically impressive, but I mostly didn't notice. I also didn't notice any pattern in where the line breaks occur, so the "pentameter" part of it seems mostly irrelevant. If it had been unmetered, I don't think I would have enjoyed it noticeably less.

Is this just my personal tastes, such that other people would really enjoy the book? I dunno, probably at least a bit, and I've spoken to at least one person who said he did really enjoy it. Still, my honest guess is that if the Iliad was published for the first time today, it wouldn't be especially well received.

If it's not just me, is the problem that writers today are better at writing than writers a long time ago? Or that they're operating under different constraints? (The Iliad was originally memorized, and meter and repetition would have helped with that.) Or do readers today just have different tastes than readers a long time ago? I don't know which of these is most true. I lean towards "writers are better" but I don't really want to try making the argument. I don't think it matters much, but feel free to replace "the Iliad isn't very good by modern standards" with "the Iliad doesn't suit modern tastes" or "isn't well optimized for today's media ecosystem".

And how much is the original versus the translation, or even the narrator of the audiobook? I still don't know, but the translation is highly praised and the narration seemed fine, so I lean towards blaming the original.

4.

What is up with classics?

Like, what makes something a classic? Why do we keep reading them?² Why did it feel vaguely blasphemous for me to say the Iliad isn't very good?

I'm probably not being original here, but….

I think one thing often going on is that classics will be winners of some kind of iterated Keynesian beauty contest. A Keynesian beauty contest is one where judges have incentive to vote, not for the person they think is most beautiful, but the person they think the other judges will vote for. Do a few rounds of those and you'll probably converge on a clear winner; but if you started over from scratch, maybe you'd get a different clear winner.

(I vaguely recall hearing about experiments along the lines of: set up some kind of Spotify-like system where users can see what's popular, and run with a few different isolated sets of users. In each group you get clear favorites, but the favorites aren't the same. If true, this points in the direction of IKBC dynamics being important.)

But what contests are getting run?

A cynical answer is that it's all signaling. We read something because all the educated people read it, so it's what all the educated people read. I'm not cynical enough to think this is the whole story behind classics, or even most of it, but it sure seems like it must be going on at least a little bit.

(Of course we're not going to get a whole crowd of admirers to admit "there's nothing here really, we just pretend to like this to seem sophisticated". So even if this is most of the story, we might not expect to find a clear example.)

To the extent that this is going on, then we'd expect me to be scared to criticize the Iliad because that exposes me as uneducated. Or we might expect me to criticize it extra harshly to show how independent-minded I am. Or both!³ And in fact I don't fully trust myself to have judged it neutrally, without regard to its status.

My guess: this is why some people read the Iliad, but it's not the main thing that makes it a classic.

Less cynically, we might observe that classics are often referenced, and we want to get the references. I don't think this could make something a classic in the first place, but it might help to cement its status.

In fact, it's not a coincidence that I listened to the Iliad soon after reading Terra Ignota. I don't really guess this is a large part of why others read it, but maybe.

(Also not a coincidence that I read A Midsummer Night's Dream soon after watching Get Over It. Reading the source material radically changed my opinion of that film. It went from "this is pretty fun, worth watching" to "this is excellent, one of my favorite movies".⁴ Similarly, See How They Run was improved by having recently seen The Mousetrap.)

A closely related thing here is that because lots of other people have read the classics, they've also written about the classics and about reading the classics. So if you enjoy reading commentary on the thing you've just read, the classics have you covered.

Some things might be classics because they're just plain good. There was a lot of crap published around the same time, and most of it has rightly been forgotten, but some was great even by the standards of today.

Like, maybe if you published Pride and Prejudice today, it would be received as "ah yes, this is an excellent entry in the niche genre of Regency-era romance. The few hundred committed fans of that genre will be very excited, and people who dabble in it will be well-advised to pick this one out".

But as I said above, I don't think the Iliad meets that bar.

I would guess the big thing for the Iliad is that it's a window into the past. Someone might read Frankenstein because they're interested in early science fiction, or because they want to learn something about that period in time. And someone might read the Iliad because they're interested in early literature or ancient Greece.

(And this might explain why classics often seem to be found in the non-fiction section of the library or bookstore. It's not that they're not fiction, but they're not being read as fiction.)

5.

The trouble is, that needs a lot of context. And I don't know enough about ancient Greece to learn much from the Iliad. I did learn a bit—like, apparently they ate the animals they sacrificed! I guessed that that was authentic, and it was news to me.

But when Priam insults his remaining sons, is that a normal thing for him to do or are we meant to learn from this that Priam is unusually deep in grief or something?

Do slaves normally weep for their captors, or does that tell us that Patroclus was unusually kind to them?

When Diomedes gets injured stripping someone of his armor, did that happen often? Is Homer saying "guys this macho bullshit is kinda dumb, kill all the other guys first and then take the armor"?

When Nestor says "no man can change the mind of Zeus", and then shortly afterwards Agamemnon prays to Zeus and Zeus changes his mind, I can't tell if that's deliberate irony or bad writing or what.

I don't even know if the early listeners of the Iliad thought these stories were true, or embellishments on actual events, or completely fabricated.

Consider Boston Legal, which ran from 2004-2008, because that happens to be the show I'm watching currently. I think I understand what the writers were going for.

The show asks the question "what if there were lawyers like Alan Shore and Denny Crane". It doesn't ask the question "what if there were lawyers"—we're expected to take that profession as a background fact, to know that there are real-life lawyers whose antics loosely inspired the show. We're not supposed to wonder "why are people dressed in those silly clothes and why don't they just solve their problems by shooting each other". (Well, Denny does sometimes solve and/or cause his problems by shooting people, but.)

It explores (not very deeply) the conflicts between professional ethics and doing the right thing, and I think that works best if we understand that the professional ethics on the show aren't something the writers just made up. I'm not sure how closely they resemble actual lawyerly professional ethics, but I think they're a good match for the public perception of lawyerly professional ethics.

It shows us a deep friendship between a Democrat and a Republican, assuming we get the context that such things are kinda rare. When Denny insists that he's not gay, that marks him as mildly homophobic but not unusually so. When they have (sympathetic!) episodes on cryonics and polyamory, we're expected to know that those things are not widely practiced in society; the characters' reactions to them are meant to read as normal, not as "these characters are bigoted".

(Boston Legal also has a lot of sexual harassment, and honestly I don't know what's up with that.)

One way to think of fiction is that it's about things that might happen, that might also happen differently. It's not just significant "what happened" but "what didn't happen; what was the space of possibilities that the actual outcome was drawn from". In Boston Legal I know a lot of ways things might turn out, which means when I'm watching I can make guesses about what's going to happen. And when it does happen, I can be like "okay yeah, that tracks"—even if I hadn't thought of it myself, I can see that it fits, and I can understand why the writers might have chosen that one. In the Iliad I don't know the space of possibilities, so when something happens I'm like "okay, I guess?"

Or: fiction is in the things we're shown, and the things we're not shown even though they (fictionally) happened, and the things we're not shown because they didn't happen. Why one thing is shown and another is unshown and another doesn't happen is sometimes a marked choice and sometimes unmarked. (That is, sometimes "the writers did this for a specific reason and we can talk about why" and sometimes "this was just kind of arbitrary or default".) And if you can't read the marks you're going to be very confused.

• Shown, unmarked: the lawyers are wearing suits. Marked: Edwin Poole isn't wearing pants.

• Not-shown, unmarked: people brush their teeth. Marked: an episode ends when the characters are waiting for an extremely time-sensitive call to be answered, and we just never learn what happened.

• Doesn't happen, unmarked: someone turns into a pig in the courtroom. Marked: someone reports a colleague to the bar for violating professional ethics. (At least not as far as I've watched.)

I can't read the marks in the Iliad. I don't know what parts of what it shows are "this is how stuff just is, in Homer's world" and what parts of it are "Homer showing the audience something unusual". That makes it difficult for me to learn about ancient Greece, and also makes it difficult to know what story Homer is telling.

I listened to the audiobook of the Iliad because I happened to know that it was originally listened-to, not read, so it seemed closer to the intended experience. But in hindsight, I can't get the intended experience and I shouldn't have tried.

I think to get the most out of the Iliad, I should have read it along with copious historical notes, and deliberately treated it as "learning about ancient Greece" at least as much as "reading a fun story".

Originally submitted to the ACX book review contest 2024; not a finalist. Thanks to Linda Linsefors and Janice Roeloffs for comments.

Conditional prediction markets are evidential, not causal

Quick note about a thing I didn't properly realize until recently. I don't know how important it is in practice.

tl;dr: Conditional prediction markets tell you "in worlds where thing happens, does other-thing happen?" They don't tell you "if I make thing happen, will other-thing happen?"

Suppose you have a conditional prediction market like: "if Biden passes the DRESS-WELL act, will at least 100,000 Americans buy a pair of Crocs in 2025?" Let's say it's at 10%, and assume it's well calibrated (ignoring problems of liquidity and time value of money and so on).

Let's even say we have a pair of them: "if Biden doesn't pass the DRESS-WELL act, will at least 100,000 Americans buy a pair of Crocs in 2025?" This is at 5%.

This means that worlds where Biden passes the DRESS-WELL act have a 5pp higher probability of the many-Crocs event than worlds where he doesn't. (That's 5 percentage points, which in this case is a 100% higher probability. I wish we had a symbol for percentage points.)

It does not mean that Biden passing the DRESS-WELL act will increase the probability of the many-Crocs event by 5pp.

I think that the usual notation is: prediction markets tell us

\[ P(\text{many-Crocs}\, | \,\text{DRESS-WELL}) = 10\% \]

but they don't tell us

\[ P(\text{many-Crocs}\, | \mathop{do}(\text{DRESS-WELL})) = \, ?\% \]

One possibility is that "Biden passing the DRESS-WELL act" might be correlated with the event, but not causally upstream of it. Maybe the act has no impact at all; but he'll only pass it if we get early signs that Crocs sales are booming. That suggests a causal model

\[   \text{early-sales}
   → \text{many-Crocs}
   → \text{DRESS-WELL}
   ← \text{early-sales}
\]

with

\[ P(\text{many-Crocs}\, | \mathop{do}(\text{DRESS-WELL}))
   = P(\text{many-Crocs})
\]

(I don't know if I'm using causal diagrams right. Also, those two "early-sales"es are meant to be the same thing but I don't know how to draw that.)

But here's the thing that triggered me to write this post. We can still get the same problem if the intervention is upstream of the event. Perhaps Biden will pass the DRESS-WELL act if he thinks it will have a large effect, and not otherwise. Let's say the act has a 50% chance of increasing the probability by 3pp and a 50% chance of increasing it by 5pp. Biden can commission a study to find out which it is, and he'll only pass the act if it's 5pp. Then we have

\[   \text{size-of-impact}
   → \text{many-Crocs}
   ← \text{DRESS-WELL}
   ← \text{size-of-impact} \\

   P(\text{many-Crocs}\, | \mathop{do}(\text{DRESS-WELL})) = \, 9\%
\]

I expect that sometimes you want to know the thing that prediction markets tell you, and sometimes you want to know the other thing. Good to know what they're telling you, whether or not it's what you want to know.

Some other more-or-less fictional examples:

• If Disney sues Apple for copyright infringement, will they win? A high probability might mean that Disney has a strong case, or it might mean that Disney will only sue if they decide they have a strong case.
• If the Federal Reserve raises interest rates, will inflation stay below 4%? A high probability might mean that raising interest rates reliably decreases inflation; or it might mean that the Fed won't raise them except in the unusual case that they'll decrease inflation.
• If I go on a first date with this person, will I go on a second? A high probability might mean we're likely to be compatible; or it might mean she's very selective about who she goes on first dates with.

Cryptocurrency taxation in the UK

Mostly out of curiosity, I've been looking into how cryptocurrency is taxed in the UK. It's not easy to get what I consider to be a full answer, but here's my current understanding, as far as I felt like looking into it. HMRC's internal cryptoassets manual is available but I didn't feel like reading it all, and some of it seems out of date (e.g. page CRYPTO22110 seems to have been written while Ethereum was in the process of transitioning from proof-of-work to proof-of-stake). I also have no particular reason to trust or distrust the non-government sources I use here. I am not any form of accountant and it would be surprising if I don't get anything wrong.

My impression is HMRC tends to be pretty tolerant of people making good faith mistakes? In that if they audit you and you underpaid, they'll make you pay what you owe but you won't get in any other trouble. Maybe they'd consider "I followed the advice of some blogger who explicitly said he wasn't an accountant" to be a good faith mistake? I dunno, but if you follow my advice and get audited, I'd love to hear what the outcome is.

After I published, reddit user ec265 pointed me at another article that seems more thorough than this one. I wouldn't have bothered writing this if I'd found that sooner. I didn't spot anywhere where it disagrees with me, which is good.

Capital gains tax

Very loosely speaking, capital gains is when you buy something, wait a bit, and then sell it for a different price than you bought it for. You have an allowance which in 2023-24 is £6,000, so you only pay on any gains you have above that. The rate is 10% or 20% depending on your income.

But with crypto, you might buy on multiple occasions, then sell only some of what you bought. Which specific coins did you sell? There's no fact of the matter.¹ But the law has an opinion.

Crypto works like stocks here. For stocks HMRC explains how it works in a document titled HS283 Shares and Capital Gains Tax (2023), and there's also manual page CRYPTO22200 which agrees.

The rule is that when you sell coins in a particular currency, you sell them in the following order:

• Any coins you bought that day;
• Any coins you bought in the following 30 days;
• Any coins you bought previously, averaged together as if you'd bought them all for the same price.

The "30 following days" thing is called the "bed and breakfasting" rule, and the point is to avoid wash sales where you try to deliberately pull forward a loss you haven't incurred yet incurred for tax purposes. Wikipedia says "Wash sale rules don't apply when stock is sold at a profit", but that doesn't seem to be true in the UK. The rule applies regardless of if you'd be otherwise selling for profit or loss.

The third bucket is called a "section 104 holding". Every time you buy coins, if they don't offset something in one of the other buckets, they go in a big pool together. You need to track the average purchase price of the coins in that pool, and when you sell, you take the purchase price to be that average. Selling doesn't affect the average purchase price of the bucket.

If there are transaction fees, they count towards the purchase price (i.e. increase the average price in the bucket) and against the sale price (i.e. decrease the profit you made). This detail isn't in HS283, but it's in a separately linked "example 3".

So suppose that at various (sufficiently distant) points in time, I

• buy 0.1 BTC for £100;
• buy 0.1 BTC for £110;
• sell 0.15 BTC for £200;
• buy 0.1 BTC for £300;
• sell 0.15 BTC for £50;

and each of these had £5 in transaction fees.

Then my section 104 holding contains:

• Initially empty.
• Then, 0.1 BTC purchased at a total of £105, average £1050/BTC.
• Then, 0.2 BTC purchased at a total of £220, average £1100/BTC.
• Then, 0.05 BTC purchased at a total of £55, average £1100/BTC.
  • Here I sold 0.15 BTC purchased at a total of £165, and I sold them for £195 after fees, so that's £30 profit.
• Then, 0.15 BTC purchased at a total of £360, average £2400/BTC.
• Then, 0 BTC purchased at a total of £0, average meaningless.
  • Here I sold 0.15 BTC purchased at a total of £360, and I sold them for £45 after fees, so that's £315 loss.

For the same-day bucket, all buys get grouped together and all sells get grouped together. For the 30-day bucket, you match transactions one at a time, the earliest buy against the earliest sell. (Unclear if you get to group them by day; I don't see anything saying you do, but if you don't then interactions with the same-day rule get weird.)

So for example, suppose the middle three events above all happened on the same day. In that case, it would work out as:

• My section 104 holding is initially empty.
• Then, it contains 0.1 BTC purchased at a total of £105, average £1050/BTC.
• Then we have three things happening on the same day.
  • Grouping buys together, I buy 0.2 BTC for £420, average £2100/BTC.
  • I sell 0.15 BTC from that bucket, which I bought for £315.
    • Sale price is £195 so that's a loss of £120.
  • The bucket now contains 0.05 BTC bought for £105, average £2100/BTC.
• That bucket enters my section 104 holding. This now contains 0.15 BTC purchased at a total of £210, average £1400/BTC.
• I sell my remaining BTC for £45, which is a loss of £165.

And if the middle three all happened within 30 days of each other, then:

• My section 104 holding is initially empty.
• Then, it contains 0.1 BTC purchased at a total of £105, average £1050/BTC.
• Then, 0.2 BTC purchased at a total of £220, average £1100/BTC.
• The subsequent buy and sell get matched:
  • I buy 0.1 BTC for £305 and sell it for £130, making a loss of £175.
  • I also sell 0.05 BTC for £65, that I'd bought at £55, making a profit of £10.
  • So in total that sale makes me a loss of £165, and the 30-day bucket contains -0.05 BTC purchased at £55.
• That bucket enters my section 104 holding. This now contains 0.15 BTC purchased at a total of £165, average £1100/BTC.
• I sell my remaining BTC for £45, which is a loss of £120.

In all cases my total loss is £285, which makes sense. But I might get taxed differently, if this happened over multiple tax years.

Some more edge cases:

• I have no idea how these rules would apply if you're playing with options or short selling. I think those are both things you can do with crypto?
• If you receive crypto as a gift, you count it as coming in at market price on the day you recieved it. I'm not sure exactly how that's meant to be calculated (on any given day, lots of buys and sells happened for lots of different prices on various different legible exchanges; and lots also happened outside of legible exchanges) but I assume if you google "historical bitcoin prices" and use a number you find there you're probably good. So it's as if you were gifted cash and used it to buy crypto.
• Similarly, if you give it away as a gift, it's treated as disposing of it at market price on the day, as if you'd sold it for cash and gifted the cash.
• I think in both the above cases, if you buy or sell below market price as a favor (to yourself or the seller respectively) you still have to consider market price.
• If you trade one coin for another, you treat it as disposing of the first for GBP and buying the second for GBP. Mark both the sell and the buy at the market price of the second, so that if you're somehow trading £1000 of one coin for £1200 of another, £200 of profits is taxable now. I assume you also count fees for the sell, reducing your profit now.

Mining and staking

According to this site, mining and staking both count as income. (And so do capital gains, if you look like a professional trader.)

For mining, the market price at the time you recieve the coins counts as miscellaneous income. You can deduct "reasonable expenses" whatever that means. (Price of hardware? Electricity?)

For staking, you can either count it as miscellaneous income or savings income. These two have different tax-free allowances. Unclear if you can count some as miscellaneous and some as savings to use both? Again you can deduct "reasonable expenses" whatever that means.

This reddit thread suggests "savings interest or miscellaneous income?" is just a grey area, in which case I'd expect HMRC to be pretty tolerant of you choosing either but kinda ಠ_ಠ if they notice you trying to use both. It links to manual page CRYPTO21200 which sounds to me like it's just miscellaneous income. ec265 agrees.

I think the normal way staking works is that to get income, you need to lock your coins up for some period of time. New coins you receive are automatically locked, and when you want to do anything with them, you have to unlock them. So do you count as earning the coins when they arrive, or when you first unlock them? (When you initiate the unlocking, or when it completes?) "When they arrive" sounds like a pain in the ass, that can happen every few days with no engagement on your part and a different market price every time. But "when you unlock" has the same problem as CGT: are you unlocking coins you locked, or coins you earned, or what?

I assume it's "when they arrive" and you just gotta deal with that. Coinbase lets you download transaction history, including all staking rewards with market price in GBP at the time of receipt, so that's not so bad. But I've also played around with staking with Trust Wallet and I can't immediately see a way to get staking history from that. Sadly I didn't earn enough to worry about.

For capital gains purposes, it sounds like both mining and staking count the same as if you'd bought the coins for market price at the time you received them. That would mean they can go in the same-day bucket or the B&B bucket, for matching against coins sold.

Are stablecoins an exception?

The point of a stablecoin is to track a currency exactly. If I have 1 USDC, I should always be able to trade that for 1 USD, and vice versa. So should you treat any holdings in USDC the same as you'd treat a bank account denominated in USD?

I think this is relevant for three reasons:

• You don't need to worry about capital gains tax in foreign currency bank accounts.²
• Coinbase pays interest on USDC. This isn't the same as staking, and it's not reported as staking in your transaction history. Interest in a foreign currency bank account counts as savings income, not miscellaneous income (see e.g. this HMRC forum answer).
• I guess it also counts as foreign income? That page isn't very clear, but I think the relevant question isn't "what currency are you getting interest in" but "what country is the bank account in". That probably depends on details of Coinbase's internal structure that I'm not familiar with; but probably they'd need to actively go to effort for UK users' USDC holdings to count as being in the UK, and probably if they did that they'd go out of their way to make sure I knew they do that, and I don't know they do it so probably they don't. If it's foreign income then it looks like that doesn't change how it's taxed, but you might need to report it differently.

I guess this means that if exchange rates don't go your way, you might end up with less money than you started but still have to pay tax, and not be able to offset your losses against capital gains.

…but I don't think that's actually how it works. It looks to me like stablecoins just get treated like any other crypto, based on this site:

Buying crypto with stablecoins is viewed as trading crypto for crypto, so any profits are subject to Capital Gains Tax.

and manual page CRYPTO10100, shortly after talking about stablecoins, saying:

HMRC does not consider cryptoassets to be currency or money.

So I think that no, stablecoins are not an exception. And I weakly guess that coinbase's USDC interest counts as miscellaneous (and non-foreign) income, not personal savings income, unless you decide that staking income is also personal savings income.

What if there's a fork?

Sometimes a cryptocurrency forks, and where you had one type of coin you now have two. How does that work?

Philosophically, I think the answer is: you always had both types of coin, it's just that no one was tracking the distinction between them. So on July 31 2017, I think that I have 0.1 BTC that I paid £100 for; on August 1 2017, I discover that actually I hold 0.1 BTC that I paid ??? for and 0.1 BCH that I paid ??? for, where the two ???s sum to £100.

(And when I sold 0.05 BTC for £30 a week previously, I actually sold 0.05 BTC and 0.05 BCH for amounts summing to £30, and it doesn't matter how they split at the time.)

In every case I know of, one of the split coins is considered the original and one is considered the fork. But I don't think there's a technical distinction there, it's just that there was a social (and sometimes legal) battle to decide who gets to use the original name and one group won that. ("Legal" example: when Ethereum Classic split off from Ethereum, the Ethereum Foundation had a trademark on the name. So whichever copy they endorsed was basically always going to get called "Ethereum", even if it turned out less popular.)

Of course, the outcomes of social-and-sometimes-legal battles can have important legal effects, even if there's no technical meaning to them. So one option would be to say that I paid £100 for 0.1 BTC, and £0 for 0.1 BCH. BTC has just had a drop in price (you can't reliably expect to sell 1 BTC + 1 BCH post-fork, for more than you could sell 1 BTC pre-fork), so your capital gains on BTC have gone down, but you can expect relatively high capital gains on BCH.

Another option would be to take the market price soon after they split. Suppose 1 BTC costs 9x as much as 1 BCH. Then we'd say I paid £90 for my BTC and £10 for my BCH.

This article recommends the second approach:

HMRC does not prescribe any particular apportionment method. It is standard practice (based on the treatment of shares, because cryptoassets use the same rules) that the cost of the original cryptoasset is apportioned between the old and new cryptoasset, pro-rata in line with the respective market values of each cryptoasset the day after the hard fork. …

HMRC has the power to enquire into an apportionment method that it believes is not just and reasonable. Therefore, whichever method an individual chooses to use, they should keep a record of this and be consistent throughout their tax returns.

Airdrops and NFTs

I don't even really know what airdrops are and I don't care how they're taxed, but I suppose some readers might so manual page CRYPTO21250 talks about them.

I don't care about NFTs either and didn't see a manual page on them, so ¯\_(ツ)_/¯.

Ledger

I like to track my finances with ledger, which means I want some way to encode these rules in that.

I think I have something that works decently, which I demonstrate in a sample file that you can see here.

I think it's mostly fairly standard outside of the Holdings top-level account. You can do e.g. ledger bal not Holdings to hide that. It doesn't make use of lot dates or prices to do matching (that's not how the UK needs you to do things). It doesn't use virtual postings.

It doesn't work in hledger because that doesn't support posting cost expressions like 0.01 ETH @ (£300 / 0.01). If you replace those with their calculated value it seems fine.

It should work fairly straightforwardly with stocks as well as crypto, with the caveat that I'm not sure how to encode stock splits and don't know if there are other fiddly details to complicate matters.

The things I'm most unhappy about are that it doesn't balance to 0, and that there's no help with average prices of Section 104 holdings.

;; This ledger demonstrates calculating capital gains on cryptocurrency for UK
;; taxes. For more info see:
;; https://reasonableapproximation.net/2024/03/28/uk-crypto-taxes.html

2020/01/01 Buy
    ; When we buy an asset, we record it in two places. `Assets` holds what we
    ; currently own, grouped in some way that's convenient for general use (by
    ; which account they're in, currency, whatever). `Holdings` holds the same,
    ; but grouped by capital gains buckets.
    ;
    ; Annoyingly, they don't balance, since for capital gains purposes the price
    ; includes transaction fees. So the total ETH balance comes to 0 but the £
    ; balance comes to `Expenses:Fees`.
    ;
    ; The `@` and `@@` ensure the ETH and GBP amounts balance with each other.
    ; But the `Holdings` exchange rate is wrong, so we use `(@@)` to avoid that
    ; getting put in the price database.
    ;
    ; S104 is "Section 104". That's the technical term for that bucket.
    Assets:ETH                                  0.13 ETH @ £765.38
    Assets:GBP                              £-100.00
    Expenses:Fees                              £0.50
    Holdings:S104:ETH                          -0.13 ETH (@@) £100.00
    Holdings:S104:ETH                        £100.00

2020/01/10 Buy
    ; So after this, the "Holdings:S104:ETH" account records that we own 0.21
    ; ETH, that we paid £200.00 for.
    Assets:ETH                                  0.08 ETH @ £1243.75
    Assets:GBP                              £-100.00
    Expenses:Fees                              £0.50
    Holdings:S104:ETH                          -0.08 ETH (@@) £100.00
    Holdings:S104:ETH                        £100.00

2020/01/31 Staking
    ; When we get staking income, we can either record it as Income in ETH or £.
    ; Recording it as ETH seems more powerful, since it lets us answer all of:
    ;
    ; * "how much ETH have I got from staking?" (`ledger bal`)
    ; * "how much £ is that worth now?" (`ledger bal -X £`)
    ; * "how much was it worth when I got it?" (`ledger bal -X £ --historical`)
    ;
    ; Recording in £ would mean `ledger bal` fully balances in ETH (at least all
    ; buys and sells do), and total balance in £ equals `Expenses:Fees`. That
    ; seems like a potentially useful sanity check. We can at least check that
    ; non-staking transactions balance like that with
    ;
    ;     ledger bal not @Staking
    ;
    ; Still, I'm not sure this is better than just recording in £.
    ;
    ; We don't need to add every staking distribution individually. We can group
    ; several together and add them all at once, as long as they don't need to
    ; be distinguished for capital gains or income tax reasons or something. But
    ; then the price isn't accurate, so we probably want to follow it with an
    ; explicit entry for the price on the final day.
    Assets:ETH                                0.0014 ETH
    Income:Staking:ETH                       -0.0014 ETH
    Holdings:S104:ETH                        -0.0014 ETH (@) £942.86
    Holdings:S104:ETH                          £1.32

; This gives the actual price at the time we most recently received staking
; income. Price database entries given by `@` and `@@` are saved at midnight, so
; might as well use that time here too. We could equivalently leave out the
; time, `P 2020/01/31 ETH £981.38`.
P 2020/01/31 00:00:00 ETH £981.38

2020/02/05 Sell
    ; At this point, S104 holds 0.2114 ETH bought for a total of £201.32,
    ; average £952.32. That means 0.0514 ETH was bought for £48.95. I don't know
    ; if there's a way to have ledger help with that calculation or enforce that
    ; we did it right.
    Assets:ETH                               -0.0514 ETH @ £1578.97
    Assets:GBP                                £80.66
    Expenses:Fees                              £0.50
    Income:Capital Gains:ETH                 £-31.71
    Holdings:S104:ETH                         0.0514 ETH (@@) £80.66
    Holdings:S104:ETH                        £-48.95

2020/03/01 Sell
    ; Now a more complicated sell that we'll match with some non-S104 buys.
    ;
    ; When we buy, we know by the end of the day which Holdings bucket(s) it
    ; needs to go in. But when we sell, any buys or other acquisitions in the
    ; next 30 days affect which bucket(s) we're drawing from. So we won't be
    ; able to complete this transaction until April. (The bed-and-breakfasting
    ; bucket for this sell runs March 2-31 inclusive.) Until we do we might
    ; choose to just write the Assets and Expenses postings, leaving the
    ; transaction not to balance in ETH until we come back and fill in the rest.
    ;
    ; This counts as a capital loss (positive income), since after transaction
    ; fees, we buy it back in future for slightly more than we sell it for now.
    ;
    ; The three +ETH and the three -£ in Holdings empty out those buckets, and
    ; in this case there's none left over to take from the S104 bucket. The
    ; `(@)`s ensure that if we get cap gains wrong, the whole thing won't
    ; balance.
    Assets:ETH                                 -0.08 ETH @ £1635.90
    Assets:GBP                               £130.37
    Expenses:Fees                              £0.50
    Income:Capital Gains:ETH                   £1.06
    Holdings:SameDay:20200301:ETH               0.01 ETH (@) (£130.37 / 0.08)
    Holdings:SameDay:20200301:ETH            £-16.71
    Holdings:BnB:20200301:ETH                   0.05 ETH (@) (£130.37 / 0.08)
    Holdings:BnB:20200301:ETH                £-80.45
    Holdings:BnB:20200301:ETH                   0.02 ETH (@) (£130.37 / 0.08)
    Holdings:BnB:20200301:ETH                £-34.27
    ; Suppose that the Mar 31 buy below didn't happen. Then the last 0.02 ETH
    ; here would come from the S104 bucket. At this point the bucket contains
    ; 0.16 ETH bought for £114.72, average £952.31. (It changed slightly in the
    ; last transaction because of rounding errors.) So 0.02 ETH was bought for
    ; £19.05. In that case the Income posting and the last two Holdings postings
    ; would be replaced with:
    ;
    ; Income:Capital Gains:ETH               £-14.16
    ; Holdings:S104:ETH                         0.02 ETH (@) (£130.37 / 0.08)
    ; Holdings:S104:ETH                      £-19.05

2020/03/01 Buy
    ; We buy some back on the very same day. This is within 30 days after the
    ; Feb 5 sell, but the sell from today takes precedence. If we bought more
    ; than 0.08 ETH here, then the remainder would go in a BnB bucket to match
    ; against that. After today, the `SameDay:20200301` account is empty.
    Assets:ETH                                  0.01 ETH @ £1620.81
    Assets:GBP                               £-16.71
    Expenses:Fees                              £0.50
    Holdings:SameDay:20200301:ETH              -0.01 ETH (@@) £16.71
    Holdings:SameDay:20200301:ETH             £16.71

2020/03/07 Buy
    ; We buy some more back within 30 days after selling, so this is also
    ; matched against the Mar 1 buy. It's 31 days after Feb 5, so it doesn't
    ; get matched against that.
    Assets:ETH                                  0.05 ETH @ £1599.01
    Assets:GBP                               £-80.45
    Expenses:Fees                              £0.50
    Holdings:BnB:20200301:ETH                  -0.05 ETH (@@) £80.45
    Holdings:BnB:20200301:ETH                 £80.45

2020/03/31 Buy
    ; And more on the final day in the BnB window. Only 0.02 ETH gets matched
    ; against the previous sale, the rest goes into the S104 bucket. After
    ; today, the `BnB:20200301` account is empty.
    Assets:ETH                                  0.05 ETH @ £1703.67
    Assets:GBP                               £-85.68
    Expenses:Fees                              £0.50
    Holdings:BnB:20200301:ETH                  -0.02 ETH (@) (£85.68 / 0.05)
    Holdings:BnB:20200301:ETH                 £34.27
    Holdings:S104:ETH                          -0.03 ETH (@) (£85.68 / 0.05)
    Holdings:S104:ETH                         £51.41

Headphones hook

Some months ago I got a 3D printer. (An Anycubic Kobra Go, which was pretty high up the list of "best value for money for a starter printer" that I found through /r/3Dprinting at the time.) I haven't used it much, but recently I used it to solve an itch. I wanted my headphones to be in easy reach, so I designed and printed a hook that I could slide over the edge of the table near my desk. This is the first thing I've designed and I'm pretty happy with how it came out!

The curve was pretty awkward. I was working with OpenSCAD (maybe something else would have been easier?) which doesn't have an easy way that I found to draw nice curves. (Lines in general seem annoying, you can extrude a 2d shape to 3d but not a 1d shape to 2d?)

I decided to go with an Archimedean spiral, and did a bunch of math to figure out how the various parameters had to relate to each other.¹ I ended up with equations that I'd have had to solve numerically because they probably had no closed form. Then rather than writing a simple script to give me the answers I just eyeballed it, figuring I could redo it properly if I felt like it. Seems basically fine though.

I ended up printing three copies. The first had a nozzle jam about half way through so it came out shitty. You can see it in the background of the second pic. It actually works, but it doesn't have enough friction on the table so it sags down. I can use it for looping cable over, though a deeper hook would be better at that. The second was mostly fine, but I decided I wanted 100% infill on the curve for strength and for some reason I did that by making the walls thick. In hindsight that's silly, it's pretty flexible and when you bend it a bit it starts to split along the middle. So I did one with normal walls and normal 100% infill and it's absolutely fine. These two are the ones actually holding it up in the second picture, I might print a third to get less pressure on the padding.

I could have made something simpler if I wanted to fasten it to the table, either with screws or duct tape. But I like that it doesn't need that.

The exact design probably won't be any use to anyone who doesn't have the same model of cheap folding table as me. But in case it's interesting or helpful, here's the .scad:

top_depth = 1.5;
bottom_depth = 1.5;
vert_depth = 1.5;
hook_depth = 3;
width = 10;

top_length = 100;
edge_height = 15.54;
b1_length = 45;
support_height = 22.4;
b2_length = 16.3;
b3_length = 20;
hook_drop = 30;

rotate([90, 0, 0]) {

color("red")
  cube([top_length + vert_depth, width, top_depth]);

color("yellow")
  translate([top_length, 0, -edge_height])
  cube([vert_depth, width, edge_height]);

color("red")
  translate([top_length - b1_length, 0, -edge_height - bottom_depth])
  cube([b1_length + vert_depth, width, bottom_depth]);

color("yellow")
  translate([top_length - b1_length, 0, -edge_height - support_height])
  cube([vert_depth, width, support_height - bottom_depth]);

color("red")
  translate([top_length - b1_length - b2_length - vert_depth,
             0,
             -edge_height - support_height - bottom_depth])
  cube([b2_length + 2*vert_depth, width, bottom_depth]);

color("yellow")
  translate([top_length - b1_length - b2_length - vert_depth,
             0,
             -edge_height - support_height])
  cube([vert_depth, width, support_height - bottom_depth]);

color("red")
  translate([top_length - b1_length - b2_length - b3_length,
             0,
             -edge_height - bottom_depth])
  cube([b3_length, width, bottom_depth]);

color("green")
  translate([top_length + vert_depth,
             width,
             -edge_height - bottom_depth])
  rotate([90, 0, 0])
  scale([-1, 1, 1])
  linear_extrude(width)
  archimedean([+10,-58], 43, 0.15, -98, 98, width=3, $fn=180);
}

// https://openhome.cc/eGossip/OpenSCAD/ArchimedeanSpiral.html
module line(point1, point2, width = 1, cap_round_1 = true, cap_round_2 = true) {
    angle = 90 - atan((point2[1] - point1[1]) / (point2[0] - point1[0]));
    offset_x = 0.5 * width * cos(angle);
    offset_y = 0.5 * width * sin(angle);

    offset_1 = [offset_x, -offset_y];
    offset_2 = [-offset_x, offset_y];

    if (cap_round_1)
        translate(point1) circle(d = width, $fn = 24);
    if (cap_round_2)
        translate(point2) circle(d = width, $fn = 24);

    polygon(points=[
        point1 + offset_1, point2 + offset_1,  
        point2 + offset_2, point1 + offset_2
    ]);
}

module polyline(points,
                width = 1,
                cap_round_1 = true,
                cap_round_2 = true)
{
    module polyline_inner(points, index) {
        if(index < len(points)) {
            line(points[index - 1], points[index], width,
            cap_round_1 = index > 1 || cap_round_1,
            cap_round_2 = index < len(points) - 1 || cap_round_2);
            polyline_inner(points, index + 1);
        }
    }

    polyline_inner(points, 1);
}

module archimedean(center, a, b, theta_1, theta_2, width=1, $fn=24) {
    d_theta = (theta_2 - theta_1)/$fn;
    thetas = [ for (i = [0:$fn]) theta_1 + i * d_theta ];
    points = [ for (t = thetas) center + [(a + b * t)*cos(t), (a + b*t)*sin(t)] ];
    polyline(points, width=width);
}

Ruining an expected-log-money maximizer

Suppose you have a game where you can bet any amount of money. You have a 60% chance of doubling your stake and a 40% chance of losing it.

Consider agents Linda and Logan, and assume they both have £1¹. Linda has a utility function that's linear in money (and has no other terms), \( U_\text{Linda}(m) = m \). She'll bet all her money on this game. If she wins, she'll bet it again. And again, until eventually she loses and has no more money.

Logan has a utility function that's logarithmic in money, \( U_\text{Logan}(m) = \ln(m) \). He'll bet 20% of his bankroll every time, and his wealth will grow exponentially.

Some people take this as a reason to be Logan, not Linda. Why have a utility function that causes you to make bets that leave you eventually destitute, instead of a utility function that causes you to make bets that leave you rich?

In defense of Linda

I make three replies to this. Firstly, the utility function is not up for grabs! You should be very suspicious any time someone suggests changing how much you value something.

"Because if Linda had Logan's utility function, she'd be richer. She'd be doing better according to her current utility function." My second reply is that this is confused. Before the game begins, pick a time \(t\). Ask Linda which distribution over wealth-at-time-\(t\) she'd prefer: the one she gets from playing her strategy, or Logan's strategy? She'll answer, hers: it has an expected wealth of \( £1.2^t \). Logan's only has an expected wealth of \( £1.04^t \).

And, at some future time, after she's gone bankrupt, ask Linda if she thinks any of her past decisions were mistakes, given what she knew at the time. She'll say no: she took the bet that maximized her expected wealth at every step, and one of them went against her, but that's life. Just think of how much money she'd have right now if it hadn't! (And nor had the next one, or the one after….) It was worth the risk.

You might ask "but what happens after the game finishes? With probability 1, Linda has no money, and Logan has infinite". But there is no after! Logan's never going to stop. You could consider various limits as \( t→∞ \), but limits aren't always well-behaved². And if you impose some stopping behavior on the game - a fixed or probabilistic round limit - then you'll find that Linda's strategy just uncontroversially gives her better payoffs (according to Linda) after the game than Logan's, when her probability of being bankrupt is only extremely close to 1.

Or, "but at some point Logan is going to be richer than Linda ever was! With probability 1, Logan will surpass Linda according to Linda's values." Yes, but you're comparing Logan's wealth at some point in time to Linda's wealth at some earlier point in time. And when Logan's wealth does surpass the amount she had when she lost it all, she can console herself with the knowledge that if she hadn't lost it all, she'd be raking it in right now. She's okay with that.

I suppose one thing you could do here is pretend you can fit infinite rounds of the game into a finite time. Then Linda has a choice to make: she can either maximize expected wealth at \( t_n \) for all finite \( n \), or she can maximize expected wealth at \( t_ω \), the timestep immediately after all finite timesteps. We can wave our hands a lot and say that making her own bets would do the former and making Logan's bets would do the latter, though I don't endorse the way we're treating infinties here.

Even then, I think what we're saying is that Linda is underspecified. Suppose she's offered a loan, "I'll give you £1 now and you give me £2 in a week". Will she accept? I can imagine a Linda who'd accept and a Linda who'd reject, both of whom would still be expected-money maximizers, just taking the expectation at different times and/or expanding "money" to include debts. So you could imagine a Linda who makes short-term sacrifices in her expected-money in exchange for long-term gains, and (again, waving your hands harder than doctors recommend) you could imagine her taking Logan's bets. But this is more about delayed gratification than about Logan's utility function being better for Linda than her own, or anything like that.

I'm not sure I've ever seen a treatment of utility functions that deals with this problem? (The problem being "what if your utility function is such that maximizing expected utility at time \(t_1\) doesn't maximize expected utility at time \(t_2\)?") It's no more a problem for Linda than for Logan, it's just less obvious for Logan given this setup.

So I don't agree that Linda would prefer to have Logan's utility function.

Counterattack

And my third reply is: if you think this is embarrassing for Linda, watch me make Logan do the same. Maybe not quite the same. I think the overall story matches, but there are notable differences.

I can't offer Logan a bet that he'll stake his entire fortune on. No possible reward can convince him to accept the slightest chance of running out of money. He won't risk his last penny to get \( £(3 ↑↑↑ 3) \), even if his chance of losing is \( 1 / (3 ↑↑↑↑ 3) \)³.

But I can offer him a bet that he'll stake all but a penny on. I can make the odds of that bet 60/40 in his favor, like the bets Linda was taking above, or any other finite probability. Then if he wins, I can offer him another bet at the same odds. And another, until he eventually loses and can't bet any more. And just like Linda, he'll be able to see this coming and he'll endorse his actions every step of the way.

How do I do this? I can't simply increase the payoff-to-stake ratio of the bet. If a bet returns some multiple of your stake, and has a 60% chance of winning, Logan's preferred amount to stake will never be more than 60% of his bankroll.

But who says I need to give him that option? Logan starts with £1, which he values at \( \ln(100) ≈ 4.6052 \).⁴ I can offer him a bet where he wagers £0.99 against £20.55 from me, with 60% chance of winning. He values that bet at

\[ 0.4\ln(100 - 99) + 0.6\ln(100 + 2055) ≈ 4.6053 \]

so he'll accept it. He'd rather wager some fraction of £0.99 against the same fraction of £20.55 (roughly £0.58 against £11.93), but if that's not on the table, he'll take what he can get.

If he wins he has £21.55 to his name, which he values at \( \ln(2155) ≈ 7.6755 \). So I offer him to wager £21.54 against my £3573.85, 60% chance of winning, which he values at… still \( 7.6755 \) but it's higher at the 7th decimal place. And so on, the stakes I offer growing exponentially - Logan is indifferent between a certainty of \( £x \) and a 60% chance of \( £x^{5/3} \) (plus 40% chance of £0.01), so I just have to offer slightly more than that (minus his current bankroll).

Admittedly, I'm not giving Logan much choice here. He can either bet everything or nothing. Can I instead offer him bets where he chooses how much of his money to put in, and he still puts in all but a penny? I'm pretty sure yes: we just need to find a function \( f : ℝ_{>0}^2 → ℝ_{>0} \) such that whenever \( a ∈ (0, x] \),

\[ \ln(x) < 0.4\ln(a) + 0.6\ln(f(x, a)) \\ {d \over d a}(0.4\ln(a) + 0.6\ln(f(x, a))) < 0 \]

Then if Logan's current bankroll is \( x \), I tell him that if he wagers \( w \), I'll wager \( f(x, x-w) - x \) (giving him 60% chance of coming away with \( f(x, x-w) \) and 40% chance of coming away with \( x-w \)). He'll want to bet everything he can on this. I spent some time trying to find an example of such a function but my math isn't what it used to be; I'm just going to hope there are no hidden complications here.

So what are the similarities and differences between Linda and Logan?

Difference: Logan's bets grow a lot faster than Linda's. For some fixed probability of bankrupting them, I need a lot less money for Linda than Logan. Similarity: I need an infinite bankroll to pull this off with probability 1, so who cares how fast the bets grow?

Difference: the structure of bets I'm offering Logan is really weird. Why on Earth would I offer him rewards exponential in his stake? Similarity: why on Earth would I offer any of these bets? They all lose money for me. Am I just a cosmic troll trying to bankrupt some utilitarians or something? (But the bets I'm offering Logan are still definitely weirder.)

Difference: I bring Linda down to £0.00, and then she'd like to bet more but she can't because she's not allowed to take on debt. I bring Logan down to £0.01, and then he'd like to bet more but he can't because he's not allowed to subdivide that penny. Similarity: these both correspond to "if your utility reaches 0 you have to stop playing".

(Admittedly, "not allowed to subdivide the penny" feels arbitrary to me in a way that "not allowed to go negative" doesn't. But note that Linda would totally be able to take on debt if she's seeing 20% return on investment. Honestly I think a lot of what's going on here, is that "not allowed to go negative" is something that's easy to model mathematically, while "not allowed to infinitely subdivide" is something that's hard to model.)

Difference: for Logan, but not for Linda, I need to know how much money he starts with for this to work. Or at least an upper bound.

But all of that feels like small fry compared to this big similarity. I can (given an infinite bankroll, a certain level of trollishness, and knowledge of Logan's financial situation) offer either one of them a series of bets, such that they'll accept every bet in turn and put as much money as they can on it; and then eventually they'll lose and have to stop betting. They'll know this in advance, and they'll play anyway, they'll lose all or almost all of their money, and they won't regret their decisions. If you think this is a problem for Linda's utility function, it's a problem for Logan's too.

What about a Kelly bettor?

I've previously made the case that we should distinguish between "maximizing expected log-money", the thing Logan does; and "betting Kelly", a strategy that merely happens to place the same bets as Logan in certain situations. According to my usage of the term, one bets Kelly when one wants to "rank-optimize" one's wealth, i.e. to become richer with probability 1 than anyone who doesn't bet Kelly, over a long enough time period.

It's well established that when offered the bets that ruin Linda, Kelly bets the same as Logan. But what does Kelly do when offered the bets that ruin Logan?

Well, I now realize that for any two strategies which make the same bet all but finitely many times, neither will be more rank-optimal than the other, according to the definition I gave in that post. That's a little embarrassing, I'm glad I hedged a bit when proposing it.

Still: when offered Logan's all-or-nothing bets, Kelly accepts at most a finite number of them. Any other strategy accepts an infinite number of them and eventually goes bankrupt with probability 1.

What about the bets where Logan got to choose how much he put in? Kelly would prefer to bet nothing (except a finite number of times) than to go all-in infinitely many times. But might she bet smaller amounts, infinitely often?

What are some possible strategies here? One is "bet a fixed amount every time"; this has some probability of eventually going bankrupt (i.e. ending up with less than the fixed amount), but I think the probability is less than 1. I don't think any of these strategies will be more or less rank-optimal than any of the others.

Another is "bet all but a fixed amount every time". This has probability 1 of eventually being down to that amount. Assuming you then stop, this strategy is more rank-optimal the higher that amount is (until it reaches your starting capital, at which point it's equivalent to not betting).

We could also consider "bet some fraction of (your current bankroll minus one penny)". Then you'll always be able to continue betting. My guess is that you'd have probability 1 of unboundedly increasing wealth, so any fraction here would be more rank-optimal than the other strategies, which can't guarantee ever turning a profit. Different fractions would be differently rank-optimal, but I'm not sure which would be the most rank-optimal. It could plausibly be unbounded, just increasing rank-optimality as the fraction increases until there's a discontinuity at 1. Or maybe the fraction shouldn't be fixed, but some function of her current bankroll.

…except that this is cheating a bit because it relies on infinite subdivisibility, and if we have that it's harder to justify the "can't bet below £0.01" thing.

So I think the answer is: Kelly will do the fractional-betting thing if she can, and if not she has no strategy she prefers over "never bet". In general, Kelly will only have a strategy she prefers over that, if she can lose arbitrarily often and still keep playing. (This is necessary but not sufficient.) Otherwise, there's some probability that she just keeps losing bets until she's out of the game; and Kelly really doesn't like that, no matter how small that probability is. Kelly has her own pathologies.

This makes me think that the technical definition of rank-optimality I suggested in the last post is not very useful here. Though nor is the technical definition of growth rate that the actual Kelly originally used.

My own strategy might be something like "bet some fraction, but if that would give me fewer than say 10 bets remaining then bet a fixed amount". That would give me a tiny chance of going bankrupt, but if I don't go bankrupt I'll be growing unboundedly. Also I'm not going to worry about the difference between £0.00 and £0.01.

Thanks to Justis Mills for commentary.

Three configurable prettyprinters

I gave a fifteen minute talk about this at Zurihac 2023. If you read this essay, I don't think there's much point in additionally watching the video.

I've been exploring a new-to-me approach to stringification. Except that right now it's three different approaches that broadly share a common goal. I call it/them pretty-gist.

The lowest-friction way to stringify things in Haskell is usually show. It gives the user close to zero ability to control how the thing is rendered.

The Show1 and Show2 classes give some control, but only in limited ways. Importantly, they only work on parameterized values; you could use Show2 to change how you render the keys of a Map Int String, but not of an IntMap String.

There are also things that give you some control over layout, but not much control over content. For example, aeson-pretty lets you pretty-print json data, and pretty-simple can pretty-print typical output from show. Both let you configure indent width, and pretty-simple additionally lets you choose some different layout styles (where to put newlines and parentheses) and stuff. But they both operate by a model where you produce a relatively simple data structure that they know how to deal with, they give you a few knobs and render it in full. (For aeson-pretty the data structure is JSON, for pretty-simple it's a custom type that they parse from show output and that's pretty good at reproducing it.)

Here are some use cases where I've wanted more control than I can easily get:

• My test suites generate a complicated data structure. Most of the time, most of this data structure is irrelevant to the test failures and I don't want to see it.

• A complicated data structure contains JSON values, and I want them rendered as JSON rather than as a Haskell data type. Or it contains floating-point numbers, and I want them rounded to 3dp with underscore separation (17_923.472). Or strings, which I want printed using C-style escapes rather than Haskell-style, and with unicode rendered.

• A list might be infinite, and I only want to show the first ten elements. Or a tree might actually be cyclic, and I want to only show three levels deep.

pretty-gist aims to enable stuff like this. I call the rendered outputs it produces "gists" following Raku's use of that term, where I think the intention is something like "take a guess about what I'm likely to want to see here and show me that". But if pretty-gist guesses wrong, it lets you correct it.

I've come up with several different approaches to this, which all have different things to recommend and disrecommend them. I'm writing about three of them here.

If you're a Haskell user, I'm interested to hear your thoughts. I have some specific questions at the end.

Design goals

• It should pretty-print, with indentation that adjusts to the available width.

• As a user, you should be able to target configurations specifically or generally. "All lists" or "all lists of Ints" or "only lists found at this specific point in the data structure". "All floating-point numbers" or "Float but not Double".

• It should be low boilerplate, both as a user and an implementer.

• You shouldn't need extra imports to configure the rendering for a type.

• If something almost works, you should be able to make it work. No need to throw everything out and start from scratch.

I don't know how to meet all these design goals at once, but they're things I aim for.

Also: I think there's a sort of hierarchy of demandingness for what sort of situations we expect to use the library in. From most to least demanding:

• We're generating user-facing output, and we want to specify what it looks like down to the glyph.

• We're generating debug output. We probably don't care about the exact layouting, but it would be unfortunate if we accidentally hid some data that we intended to include. We can't simply edit the config and retry.

• We're generating output for a test failure. It's not ideal if it stops doing what we expect without warning, but it's not too bad because we can just change it and rerun the test.

• We're writing debug code that we don't expect to commit.

In more demanding situations, we probably don't mind being more verbose, and updating our rendering code more often when the data structures we're working with change. In less demanding situations, we're more likely to be satisfied with an 80/20 of "not quite what I want but good enough".

I'm not personally working on anything where I care about glyph-level exactness, so that one isn't a situation I feel like optimizing for. The others are all things I'd like pretty-gist to help with, but of course the best design for one might not be the best design for the others.

Example

Here's how pretty-simple would render a particular representation of a chess game. Next to it is how I'd like pretty-gist to be able to, either by default or with a small amount of configuration on the user's part.

GameState
  { turn = White
  , pBlackWin = 0.3463
  , pWhiteWin = 0.3896
  , nMoves = 0
  , board = Board
    [
      [ Just
        ( Piece
          { pieceType = Rook
          , owner = Black
          , lastMoved = Nothing
          }
        )
      , Just
        ( Piece
          { pieceType = Knight
          , owner = Black
          , lastMoved = Nothing
          }
      ...

GameState { turn = White
          , pBlackWin = 35%
          , pWhiteWin = 39%
          , nMoves = 0
          , board = [ [r, n, b, q, k, b, n, r]
                    , [p, p, p, p, p, p, p, p]
                    , [_, _, _, _, _, _, _, _]
                    , [_, _, _, _, _, _, _, _]
                    , [_, _, _, _, _, _, _, _]
                    , [_, _, _, _, _, _, _, _]
                    , [P, P, P, P, P, P, P, P]
                    , [R, N, B, Q, K, B, N, R]
                    ]
          }

So one difference is that I've gone with a hanging bracket style, with no newline directly after GameState or board =. I don't feel strongly about that. It would be nice to let users control this, but I haven't put much thought into it.

I've also rendered the floats as percents. I haven't put much thought into this either, and haven't implemented it. But it seems vaguely useful and easy enough to have as an option, though it usually shouldn't be default.

It's not visible here, but pretty-simple has the ability to colorize its output. That's another thing I haven't thought about, and don't currently expect pretty-gist to support any time soon.

But the most important is rendering each Maybe Piece as a single character. There are three parts to that: a Nothing is rendering as _; a Just is simply rendering the value it contains with no wrapper; and a Piece is rendering as a single character. The combination makes it much easier to see most of the state of the game. You can no longer see when each piece last moved. But if that's not usually useful to you, it's fine not to show it by default.

(At this point, chess pedants may be pointing out that this data type doesn't capture everything you need for chess. You can't reliably tell whether en passant is currently legal. Maybe there are other problems too. Yes, well done chess pedants, you're very clever, now shut up.)

Possible designs

I've come up with several possible designs for this making different tradeoffs. I'm going to talk about three of them that I've actually implemented to some extent.

Classless solution

Perhaps the very simplest solution is just to write a custom renderer every time I need one. I'm not going to do that.

A level up from that, which I've implemented in the module Gist.Classless, is to write renderers for lots of different data types and combine them. We can write

newtype Prec = Prec Int -- precedence level, 0 - 11

data ConfigMaybe = ConfigMaybe { showConstructors :: Bool }
gistMaybe :: ConfigMaybe -> (Prec -> a -> Doc) -> Prec -> Maybe a -> Doc

data ConfigList = ConfigList { showElems :: Maybe Int, ... }
gistList :: ConfigList -> (Prec -> a -> Doc) -> Prec -> [a] -> Doc

data ConfigTuple2 = ConfigTuple2 { ... }
gistTuple2
  :: ConfigTuple2
  -> (Prec -> a -> Doc)
  -> (Prec -> b -> Doc)
  -> Prec
  -> (a, b)
  -> Doc

data ConfigFloat = ConfigFloat { ... }
gistFloat :: ConfigFloat -> Prec -> Float -> Doc

for some data type Doc that supports layout. (I've been using the one from prettprinter, which has a type argument that I'm ignoring here for simplicity.)

The Prec parameters here are needed for the same reason Show has showsPrec. Sometimes we need parentheses, and this lets us choose when we have them. But they clutter things up a lot. We could imagine predence being something that gets put into config types, but then the user needs to specify it everywhere; plus, a renderer might call one of its sub-renderers at two different precedence levels under different circumstances. So that doesn't really work, so we accept the clutter.

But essentially, we've decided that one particular config parameter is important enough that every function accepts it (which means every function expects every other function to accept it). That feels kinda dirty. Is there anything else that ought to be given this treatment?

Anyway, this works, and it has some things to recommend it. It's incredibly simple, a beginner-level Haskell programmer will be able to figure out what's going on. If I, as the library author, make a decision you the library user don't like, you can just write your own function and it plugs in seamlessly. And if you have a type that can't normally be rendered, like a function, you can pick some way to render it anyway.

It also has some things to disrecommend it. Most notably, it's very verbose. You need to specify how to render every type-parameterized node of your data structure. You can have default configs, but there's no "default list renderer" because there's no "default list element renderer". IntMap v can have a default renderer for its keys, but Map Int v can't unless you write a function gistMapInt separate from gistMap.

This also means that changes to your data structure are very often going to need to be reflected in your renderers, which sounds tedious.

Another problem is, I expect consistency to be hard. Whatever design decisions I make, they're not enforced through anything. So someone who disagrees with them, or simply isn't paying attention to them, can easily make different ones, and then users need to remember things. (E.g. I had gistList, gistTuple2 and gistFloat all take precedence parameters, but they'll completely ignore them. So maybe someone in a similar situation decides not to bother with those parameters.)

Those are problems for users. There's also a problem for implementers: roughly speaking, you're going to be allowing the user to pass a renderer for every field of every constructor of your type. For non-parameterized types (like the keys of an IntMap) that can be in the actual config type, and for parameterized types (like the keys of a Map) it comes in separate arguments later, but it's going to be there. That's going to be tedious for you.

data ConfigMaybe = ConfigMaybe { showConstructors :: Bool }
  deriving stock Generic

defaultConfigMaybe :: ConfigMaybe
defaultConfigMaybe = ConfigMaybe { showConstructors = False }

gistMaybe :: ConfigMaybe -> (Prec -> a -> Doc ann) -> Prec -> Maybe a -> Doc ann
gistMaybe (ConfigMaybe {..}) renderElem prec = if showConstructors
  then \case
    Nothing -> "Nothing"
    Just a  -> parensIf (prec > 10) $ "Just" <+> renderElem 11 a
  else \case
    Nothing -> "_"
    Just a  -> renderElem prec a

-- Renders "()".
gistMaybe defaultConfigMaybe (\_ _ -> "()") 0 $ Just ()

-- Renders "Just ()".
gistMaybe (defaultConfigMaybe { showConstructors = True })
          (\_ _ -> "()")
          0
  $ Just ()

import qualified Gist
import           Gist ( Prec )
import qualified Gist as Gist.ConfigMaybe ( ConfigMaybe(..) )

data ConfigPiece = ConfigPiece
  { singleChar      :: Bool
  , renderPieceType :: forall ann . Prec -> PieceType -> Doc ann
  , renderOwner     :: forall ann . Prec -> Player -> Doc ann
  , renderLastMoved :: forall ann . Prec -> Maybe Int -> Doc ann
  }

defaultConfigPiece :: ConfigPiece
defaultConfigPiece = ConfigPiece
  { singleChar      = False
  , renderPieceType = Gist.gistShowily
  , renderOwner     = Gist.gistShowily
  , renderLastMoved = Gist.gistMaybe Gist.defaultConfigMaybe Gist.gistShowily
  }

gistPiece :: ConfigPiece -> Prec -> Piece -> Doc ann
gistPiece (ConfigPiece {..}) prec piece@(Piece {..}) = if singleChar
  then prettyPieceChar piece
  else Gist.record
    prec
    (Just "Piece")
    [ ("pieceType", renderPieceType 0 pieceType)
    , ("owner"    , renderOwner 0 owner)
    , ("lastMoved", renderLastMoved 0 lastMoved)
    ]

gistBoard :: (Prec -> [[a]] -> Doc ann) -> Prec -> Board a -> Doc ann
gistBoard renderer prec (Board a) = renderer prec a

data ConfigGameState = ConfigGameState
  { renderTurn      :: forall ann . Prec -> Player -> Doc ann
  , renderPBlackWin :: forall ann . Prec -> Float -> Doc ann
  , renderPWhiteWin :: forall ann . Prec -> Float -> Doc ann
  , renderNMoves    :: forall ann . Prec -> Int -> Doc ann
  , renderBoard     :: forall ann . Prec -> Board (Maybe Piece) -> Doc ann
  }

defaultConfigGameState :: ConfigGameState
defaultConfigGameState = ConfigGameState
  { renderTurn      = Gist.gistShowily
  , renderPBlackWin = Gist.gistShowily
  , renderPWhiteWin = Gist.gistShowily
  , renderNMoves    = Gist.gistShowily
  , renderBoard     = gistBoard
                      $ Gist.gistList Gist.defaultConfigList
                      $ Gist.gistList Gist.defaultConfigList
                      $ Gist.gistMaybe Gist.defaultConfigMaybe
                      $ gistPiece
                      $ defaultConfigPiece { singleChar = True }
  }

gistGameState :: ConfigGameState -> Prec -> GameState -> Doc ann
gistGameState (ConfigGameState {..}) prec (GameState {..}) = Gist.record
  prec
  (Just "GameState")
  [ ("turn"     , renderTurn 0 turn)
  , ("pBlackWin", renderPBlackWin 0 pBlackWin)
  , ("pWhiteWin", renderPWhiteWin 0 pWhiteWin)
  , ("nMoves"   , renderNMoves 0 nMoves)
  , ("board"    , renderBoard 0 board)
  ]

-- Renders in short form.
gistGameState defaultConfigGameState 0 startPos

-- Renders in long form.
let conf = defaultConfigGameState
      { renderBoard = gistBoard
                      $ Gist.gistList Gist.defaultConfigList
                      $ Gist.gistList Gist.defaultConfigList
                      $ Gist.gistMaybe Gist.defaultConfigMaybe
                      $ gistPiece
                      $ defaultConfigPiece { singleChar = False }
      }
in gistGameState conf 0 startPos

-- Renders in fully explicit form.
let confMaybe =
      Gist.defaultConfigMaybe { Gist.ConfigMaybe.showConstructors = True }
    conf      = CB.defaultConfigGameState
      { CB.renderBoard = CB.gistBoard
                         $ Gist.gistList Gist.defaultConfigList
                         $ Gist.gistList Gist.defaultConfigList
                         $ Gist.gistMaybe confMaybe
                         $ CB.gistPiece
                         $ CB.defaultConfigPiece
                             { CB.singleChar      = False
                             , CB.renderLastMoved = Gist.gistMaybe
                                                      confMaybe
                                                      Gist.gistShowily
                             }
      }
in gistGameState conf 0 startPos

One-class solution

We can maybe-improve on this with typeclasses. We can use type families to let each gistable type have a separate config type.

class Gist a where
  type Config a :: Type
  defaultConfig :: Config a
  gistPrec :: Prec -> Config a -> a -> Doc

data ConfigList a = ConfigList
  { showFirst :: Maybe Int
  , configElem :: Config a
  }

instance Gist a => Gist [a] where
  type Config [a] = ConfigList a
  defaultConfig = ConfigList { showFirst = Nothing, configElem = defaultConfig }
  gistPrec = ...

This is the foundation of the approach I've implemented in the module Gist.OneClass.

There are a few significant complications. One is, this won't handle String well, because that's just [Char]. Other typeclasses (including Show) solve this by having an extra method for "how to handle lists of this type"; then you give that method a default implementation, but override it for Char. This seems fine as a solution, by which I mean "I hate it but I don't have any better ideas". I'm not going to bother showing it here. (Also it's not actually implemented for this approach in the code yet.)

Next: the typechecking here doesn't work very well. If we try

gist (defaultConfig { ... }) [True] -- `gist` is `gistPrec 0`

we probably mean defaultConfig to refer to defaultConfig @[Bool]. But all GHC knows is that we mean defaultConfig { ... } to have type Config [Bool]. That doesn't even fully specify the type of defaultConfig, let alone its value. (We might be using the defaultConfig of some other instance that shares the same type; or that has a different type until we update it. The second possibility means injective type families wouldn't help much.) So we instead need to write

gist ((defaultConfig @[Bool]) { ... }) [True]

which is no fun. That extra unnecessary-looking set of parentheses is an added kick in the teeth. So we add functions gistF and gistPrecF, which replace the Config a argument with a Config a -> Config a argument and apply it to defaultConfig. So now we write

gistF (\c -> c { ... }) [True]

But we do keep the existing functions around for the final complication. Some things can't be rendered automatically (e.g. functions), but we sometimes want to render them anyway, or data structures containing them. Like a function Bool -> Int that's equivalent to a pair (Int, Int). Sometimes we can use newtypes and maybe coerce for this, but not always, and it might be a pain even if we can.

It turns out we can handle this case. Consider the type

data Gister a where
  FnGister :: (Int -> a -> Doc) -> Gister a
  ConfGister :: Gist a => Config a -> Gister a

runGisterPrec :: Int -> Gister a -> a -> Doc
runGisterPrec prec = \case
  FnGister   f -> f prec
  ConfGister c -> gistPrec prec c

A Gister is a renderer for any type. For types implementing Gist, we can create a Gister through Config, but for any other type we can still write our own rendering function.

This lets us have an instance Gist [a] without first requiring Gist a. We can't have a (useful) default config in that case, the only fully general Gister as we could write would ignore the value they're passed. But we can still have a default when we do have Gist a (assuming it has its own default):

class Gist a where
  type Config a :: Type
  type HasDefaultConfig a :: Constraint
  defaultConfig :: HasDefaultConfig a => Config a
  gistPrec :: Int -> Config a -> a -> Doc

data ConfigList a = ConfigList
  { showFirst :: Maybe Int
  , gistElem  :: Gister a
  }

instance Gist [a] where
  type Config [a] = ConfigList a
  type HasDefaultConfig [a] = (Gist a, HasDefaultConfig a)
  defaultConfig =
    ConfigList { showFirst = Nothing, gistElem = defaultConfig }
  gistPrec = ...

So now you can call gist on a [Bool -> Int], and you need to write for yourself how to render one of those functions but you can use gist when you do so. There's no defaultConfig @[Bool -> Int], but you can do a type-changing update of defaultConfig @[Void] or similar. Though this is harder than we might like, because we can't derive a Generic instance for Gister a which means we can't use generic-lens or generic-optics. Fine in this case, annoying for nested structures, might be able to improve.

And that's just about everything for this solution.

So this preserves a lot of the good stuff about the classless solution. It's still reasonably simple as Haskell code, at least according to my judgment. We can still render any type, though it would be even simpler if we removed that option. And the user can still completely override the implementer if they want, though not as seamlessly as before.

And it's usually going to be a lot less verbose, with less need to change when your data structure changes. If the bits you've configured haven't changed, you should be good.

But there's still things not to like. For one, the tedious-for-implementers thing hasn't changed.

For another, if a type shows up at multiple places in the data structure, you probably want to render it the same in all of those places; if you have a [(Float, Float)] you probably want to render the fsts the same as the snds. But to do that you have to remember every place it might show up and configure them all separately; and if it starts showing up in a new place, it's probably easy for you to forget to configure that one. (Unlike with the classless solution, where you'll get type errors if you forget about it.)

You're also going to be dealing with nested record updates, which I find unpleasant and have a bunch of questions about. That's somewhat the case with the classless solution too, but I think less deeply nested due to the structure of arguments and the lack of Gister. And here, you'll sometimes be doing type-changing record updates, and I think the future of those is uncertain (they're not supported by OverloadedRecordUpdate).

data ConfigMaybe a = ConfigMaybe
  { showConstructors :: Bool
  , gistElem :: Gister a
  }
  deriving stock Generic

instance Gist (Maybe a) where
  type Config (Maybe a) = ConfigMaybe a
  type HasDefaultConfig (Maybe a) = (Gist a, HasDefaultConfig a)
  defaultConfig = ConfigMaybe False (ConfGister $ defaultConfig @a)

  gistPrec prec (ConfigMaybe {..}) = if showConstructors
    then \case
      Nothing -> "Nothing"
      Just x  -> parensIf (prec > 10) $ "Just" <+> runGisterPrec 11 gistElem x
    else \case
      Nothing -> "_"
      Just x  -> runGisterPrec prec gistElem x

-- Renders "()". `gist_` is `gistF id`.
gist_ $ Just ()

-- Renders "Just ()".
gistF (\c -> c { showConstructors = True }) $ Just ()

import qualified Gist
import           Gist ( Gist(..) )
import qualified Gist as Gist.ConfigList ( ConfigList(..) )
import qualified Gist as Gist.ConfigMaybe ( ConfigMaybe(..) )

deriving via Gist.Showily Player instance Gist Player
deriving via Gist.Showily PieceType instance Gist PieceType
deriving newtype instance Gist (Board a)

data ConfigPiece = ConfigPiece
  { singleChar    :: Bool
  , gistPieceType :: Gist.Gister PieceType
  , gistOwner     :: Gist.Gister Player
  , gistLastMoved :: Gist.Gister (Maybe Int)
  }
  deriving stock Generic

instance Gist Piece where
  type Config Piece = ConfigPiece
  defaultConfig = ConfigPiece False
                              (Gist.ConfGister $ defaultConfig @PieceType)
                              (Gist.ConfGister $ defaultConfig @Player)
                              (Gist.ConfGister $ defaultConfig @(Maybe Int))

  gistPrec prec (ConfigPiece {..}) piece@(Piece {..}) = if singleChar
    then prettyPieceChar piece
    else Gist.record
      prec
      (Just "Piece")
      [ ("pieceType", Gist.runGister gistPieceType pieceType)
      , ("owner"    , Gist.runGister gistOwner owner)
      , ("lastMoved", Gist.runGister gistLastMoved lastMoved)
      ]

data ConfigGameState = ConfigGameState
  { gistTurn      :: Gist.Gister Player
  , gistPBlackWin :: Gist.Gister Float
  , gistPWhiteWin :: Gist.Gister Float
  , gistNMoves    :: Gist.Gister Int
  , gistBoard     :: Gist.Gister (Board (Maybe Piece))
  }
  deriving stock Generic

instance Gist GameState where
  type Config GameState = ConfigGameState
  defaultConfig = ConfigGameState
    { gistTurn      = Gist.defaultConfGister
    , gistPBlackWin = Gist.defaultConfGister
    , gistPWhiteWin = Gist.defaultConfGister
    , gistNMoves    = Gist.defaultConfGister
    , gistBoard     =
      let
        gPiece  = Gist.defaultConfGisterF $ \c -> c { singleChar = True }
        gMPiece = Gist.defaultConfGisterF
          $ \c -> c { Gist.ConfigMaybe.gistElem = gPiece }
        gLMPiece = Gist.defaultConfGisterF
          $ \c -> c { Gist.ConfigList.gistElem = gMPiece }
        gBoard = Gist.defaultConfGisterF
          $ \c -> c { Gist.ConfigList.gistElem = gLMPiece }
      in
        gBoard
    }

  gistPrec prec (ConfigGameState {..}) (GameState {..}) = Gist.record
    prec
    (Just "GameState")
    [ ("turn"     , Gist.runGister gistTurn turn)
    , ("pBlackWin", Gist.runGister gistPBlackWin pBlackWin)
    , ("pWhiteWin", Gist.runGister gistPWhiteWin pWhiteWin)
    , ("nMoves"   , Gist.runGister gistNMoves nMoves)
    , ("board"    , Gist.runGister gistBoard board)
    ]

-- Renders in short form. `gist_` is `gistF id`.
gist_ startPos

-- Renders in long form. This uses generic-lens to do record updates, but an
-- approach like used in `defaultConfig` would work too:
--     gistF (let ... in \c -> c { gistBoard = gBoard }) startPos
gistF
  ( setField @"gistBoard"
  $ Gist.defaultConfGisterF
  $ setField @"gistElem"
  $ Gist.defaultConfGisterF
  $ setField @"gistElem"
  $ Gist.defaultConfGisterF
  $ setField @"gistElem"
  $ Gist.defaultConfGisterF
  $ setField @"singleChar" False
  )
  startPos

-- Renders in fully explicit form. This could also be done with standard record
-- updates.
gistF
  ( setField @"gistBoard"
  $ Gist.defaultConfGisterF
  $ setField @"gistElem"
  $ Gist.defaultConfGisterF
  $ setField @"gistElem"
  $ Gist.defaultConfGisterF
  $ ( setField @"showConstructors" True
    . ( setField @"gistElem"
      $ Gist.defaultConfGisterF
      $ ( setField @"singleChar" False
        . ( setField @"gistLastMoved"
          $ Gist.defaultConfGisterF
          $ setField @"showConstructors" True
          )
        )
      )
    )
  )
  startPos

Two-class solution

So here's a very different approach.

First, we find some way to store the config for every possible in a single data structure, even though we don't know all the possible configs yet.

Then we make this config store available to renderers. They look up the config that's relevant specifically to them. When rendering their contents, they simply pass down the same config store. A MonadReader helps here.

This makes "update the config of every occurrence of a type" easy. It makes "update the config of just this specific occurrence of a type" impossible. So we also track our location in the data structure, and in the config store we let users say "this option only applies at this location", or "at locations matching …".

(This last bit sounds almost more trouble than it's worth. But without it, it becomes super awkward to handle things like "only show three levels deep of this self-referential data type".)

This is currently implemented in the Gist.TwoClass module. There's also a Gist.Dynamic module which has just the config-data-structure part, and is actually the implementation I've fleshed out the most. But I currently think it's not worth exploring more and not worth discussing in depth by itself.

Somewhat simplified, here's the main stuff going on with this solution:

-- | Opaque storage of config options, implemented with existential types. Not
-- type-safe by construction, but has a type-safe interface.
newtype Config = UnsafeConfig { ... }

-- | Things that can be put into a `Config`.
class (Typeable a, Monoid (ConfigFor a), Typeable (ConfigFor a))
  => Configurable a
 where
  type ConfigFor a :: Type

-- | Tracking and matching locations in the data structure.
newtype GistPath = ...
data PathMatcher = ...

data GistContext = GistContext
  { gcPath :: GistPath
  , gcConf :: Config
  }

-- | Things that can be rendered.
class Configurable a => Gist a where
  renderM :: MonadReader GistContext m => Int -> a -> m Doc

-- | The user-facing interface.
gist :: Gist a => [Config] -> a -> Doc
config :: Configurable a => Maybe PathMatcher -> ConfigFor a -> Config

The separation between Configurable and Gist might seem unnecessary here - why would we configure something we can't render? The answer is that Configurable doesn't specify the kind of its argument. So we have all of

instance Configurable Map
instance Typeable k => Configurable (Map k)
instance (Typeable k, Typeable v) => Configurable (Map k v)

and then users can add configuration for all Maps without needing to specify the exact types. (And they can override that config at specific types, if they want, using the Semigroup instance of the ConfigFor.) We also have instance Configurable Floating, and then the Gist instances for both Float and Double can look that up.

So the flow is that users build up a Config data structure, specifying "for this type, (optionally: at this location in the data structure,) set these options".

Then we walk through the structure. At each point we look up the relevant config values for the type at the current location, and possibly for other types, and combine all these in some way that users will just have to beat into submission if they're doing something complicated. And then we render, passing an updated GistPath and the same Config to any subcomponents.

This isn't very transparent to users. The Float instance looks up config for both Float and Floating, and the Maybe a instance looks it up for both Maybe a and Maybe. But to discover these you have to either try it and see, or look at the source code, or read the instance docs that someone probably forgot to write.

Also, each config option needs to have a Monoid instance that distinguishes between "this value hasn't been set" and "this value has been set to its default". In practice that means Last. But that means users don't know what the default value of any config option is.

So the actual code complexifies the implementation in a few ways, to help users. Instances have a way of specifying "these are the types I look up", as long as those types have the same ConfigFor. Then looking things up from the config happens automatically; and there's a separate function to set default values to what gets looked up, which users can call manually to see what's going on. Once we have the defaults we no longer need Last, so we change to Identity at that point.

We also use a custom monad class instead of MonadReader GistContext. For now it's no more powerful, but it would be easy to add a tracing function. Then if users had trouble figuring out what was going on, they could use that to help figure it out, with no additional work from implementers.

So the actual implementation looks more like

class (Typeable a, Monoid (ConfigFor a Last), Typeable (ConfigFor a Last))
  => Configurable a
 where
  type ConfigFor a (f :: Type -> Type) :: Type

class (Configurable a, CanDoLookups (GistLookups a) (ConfigFor a Last))
  => Gist a
 where
  type GistLookups a
  reifyConfig :: ConfigFor a Last -> ConfigFor a Identity
  renderM :: MonadGist m => Int -> ConfigFor a Identity -> a -> m Doc

config :: Configurable a => Maybe PathMatcher -> ConfigFor a Last -> Config

where CanDoLookups roughly ensures that GistLookups a is a type-level list of things with a given ConfigFor. (But we can't use '[] for these type-level lists, because you can't put types of different kinds inside one of those.)

How does this fare? The big advantage over the previous solutions is the "configure every occurence of a type at once" thing. I anticipate this is usually what users want, so it's good that it's easy. Also, no nested records! And if I decide to add global config options - perhaps indent width - I just need to add them to GistContext.

I think the big downsides are that it's wildly complicated, and we've lost the ability to render anything we can't write a Gist instance for (which also means users can't override implementers' decisions). But also a bunch of other downsides. When you do want to render different occurrences of the same type differently, it's awkward. You won't get errors or warnings if your config gets out of sync with the type you're rendering. Encapsulation is tricky, internals of your types might be exposed in ways you don't want. It's not necessarily clear how you'd want newtypes to be configured, and newtype-deriving only gives you one option which might not be what you want.

data ConfigMaybe f = ConfigMaybe { showConstructors :: f Bool }
  deriving stock Generic
instance Semigroup (ConfigMaybe Last) where
  a <> b = ConfigMaybe (showConstructors a <> showConstructors b)
instance Monoid (ConfigMaybe Last) where
  mempty = ConfigMaybe mempty

instance Configurable Maybe where
  type ConfigFor Maybe f = ConfigMaybe f
instance Typeable a => Configurable (Maybe a) where
  type ConfigFor (Maybe a) f = ConfigFor Maybe f

instance Gist a => Gist (Maybe a) where
  type GistLookups (Maybe a) = CL Maybe
  reifyConfig (ConfigMaybe {..}) =
    ConfigMaybe (Identity $ fromLast False showConstructors)
  renderM prec (ConfigMaybe {..}) = if runIdentity showConstructors
    then \case
      Nothing -> pure "Nothing"
      Just x  -> do
        renderedElem <- subGistPrec 11 Nothing x
        pure $ parensIfPrecGT 10 prec $ "Just" <+> renderedElem
    else \case
      Nothing -> pure "_"
      Just x  -> subGistPrec prec Nothing x

-- Renders "()".
gist [] $ Just ()

-- Renders "Just ()".
gist [configF @Maybe $ \c -> c { showConstructors = pure True }] $ Just ()

import qualified Gist
import           Gist ( Configurable(..), Gist(..), fromLast )
import qualified Gist as Gist.ConfigMaybe ( ConfigMaybe(..) )

deriving via Gist.Showily Player instance Configurable Player
deriving via Gist.Showily Player instance Gist Player

deriving via Gist.Showily PieceType instance Configurable PieceType
deriving via Gist.Showily PieceType instance Gist PieceType

-- We can't derive an instance `Configurable Board`. We have that `Board a` is
-- representation-equivalent to `[[a]]`, but `Board` itself isn't
-- representation-equivalent to anything. Anyway, even if we had an instance,
-- the instance we're deriving for `Gist (Board a)` wouldn't look at it.
deriving newtype instance Typeable a => Configurable (Board a)
deriving newtype instance Gist a => Gist (Board a)

data ConfigPiece f = ConfigPiece { singleChar :: f Bool }
  deriving stock Generic
instance Semigroup (ConfigPiece Last) where
  (ConfigPiece a1) <> (ConfigPiece a2) = ConfigPiece (a1 <> a2)
instance Monoid (ConfigPiece Last) where
  mempty = ConfigPiece mempty

instance Configurable Piece where
  type ConfigFor Piece f = ConfigPiece f

instance Gist Piece where
  type GistLookups Piece = ()
  reifyConfig (ConfigPiece a) = ConfigPiece (Identity $ fromLast False a)
  renderM prec (ConfigPiece {..}) piece@(Piece {..}) =
    if runIdentity singleChar
      then pure $ prettyPieceChar piece
      else Gist.record
        prec
        (Just "Piece")
        [ ("pieceType", Gist.subGist (Just "pieceType") pieceType)
        , ("owner"    , Gist.subGist (Just "owner") owner)
        , ("lastMoved", Gist.subGist (Just "lastMoved") lastMoved)
        ]

instance Configurable GameState where
  type ConfigFor GameState f = Proxy f

instance Gist GameState where
  type GistLookups GameState = ()
  reifyConfig _ = Proxy
  renderM prec _ (GameState {..}) =
    Gist.localPushConf
        (Gist.configF @Piece $ \c -> c { singleChar = pure True })
      $ Gist.record
          prec
          (Just "GameState")
          [ ("turn"     , Gist.subGist (Just "turn") turn)
          , ("pBlackWin", Gist.subGist (Just "pBlackWin") pBlackWin)
          , ("pWhiteWin", Gist.subGist (Just "pWhiteWin") pWhiteWin)
          , ("nMoves"   , Gist.subGist (Just "nMoves") nMoves)
          , ("board"    , Gist.subGist (Just "board") board)
          ]

-- Renders in short form.
gist [] startPos

-- Renders in long form. generic-lens could replace the record update, one of:
--     configF @Piece $ setField @"singleChar" $ pure False
--     configF @Piece $ field @"singleChar" .~ pure False
gist [Gist.configF @Piece $ \c -> c { singleChar = pure False }]
     startPos

-- Renders in fully explicit form. generic-lens would work here too, avoiding
-- the horrible import.
gist
  [ Gist.configF @Piece $ \c -> c { singleChar = pure False }
  , Gist.configF @Maybe
      $ \c -> c { Gist.ConfigMaybe.showConstructors = pure True }
  ]
  startPos

Commentary

I've given three different renders for all of these GameState examples. "Short form" looks like the example I gave above, except I haven't configured the floats to show as percentages. (Like I said, not yet implemented.) "Fully explicit form" is similar to the pretty-simple rendering, except with a different indent style. And "long form" is in between, with abbreviated rendering for Maybe but everything else displayed in full - there's no ambiguity here, so it seems like a good choice even if you don't want any data missing.

In this particular case, rendering GameState for the classless and one-class solutions are about as verbose as each other. I think that's kind of a coincidence based on where the type variables are and what we're doing with them. For example, GameState has no type variables, so its renderers can be passed in ConfigGameState, letting them be defaulted. Board has a type variable, but renderBoard has no other config options; so there are no cases where with one-class we can say "change this option for how we render Board, but leave the sub-renderer alone", but with classless we can only say "change this option for how we render Board, and while we're at it here's the sub-renderer to use". This kind of thing does come up once, when in the fully-explicit form we want to set showConstructors on the lastMoved renderer, and have to also repeat the renderer for the Int. But in that case the renderer is just gistShowily so it doesn't hurt much.

I expect there'd be more difference in other cases. But maybe I'm wrong? Who knows.

A thing I dislike in all of them, is how I've had to do imports to handle record field updates. As of (I think) GHC 9.2, record updates combined with DuplicateRecordFields is more awkward than it used to be, at least if you don't want to get warnings about future compatibility. So when I do the awful

import qualified Gist as Gist.ConfigMaybe ( ConfigMaybe(..) )

that's so I can do val { Gist.ConfigMaybe.showConstructors = ... } to update the field. Another option is to use generic-lens or generic-optics to do updates, which I've also demonstrated in some cases. My guess is that would be fine in a lot of application code, but many libraries won't want to depend on them.

I could also try to choose constructor names not to conflict. But I think that basically means prefixing them all, which would also be super verbose, and it would annoy users of the generic libraries.

(Given what I've implemented so far, the name conflicts only actually exist in the one-class solution, which reuses the name gistElem. But I expect showConstructors and showFirst to also get reused, e.g. for gisting Either and Set; and something like countRemaining could be useful for collections that get truncated, and so on. So it seemed more useful to write things this way.)

I haven't looked closely, but anonymous records might help here. They also might help with Generic-based defaulting, which is another thing I haven't looked into. But they seem like a large hammer that increases dependency footprints. And from a quick look I'm not sure they support type-changing record updates.

I'd prefer if there wasn't a single fixed indent style. But I'm not sure what to do about it. Conceivably, I could have a fixed IndentStyle type, and expect implementers to pay attention to it. (Similar to how precedence has been granted special status.) But I expect if I do that, there'll be situations where people wish the type was larger (because it doesn't support things they want), and others where they wish it was smaller (because implementing support for all the options is a pain). If I make it maximally small, we only have one of those problems. Similar thoughts on possible color support.

Questions for you

One reason I wrote all this is to try to gauge enthusiasm. One possible future for pretty-gist is that I use it a small amount and in personal projects and at my job, and basically no one else uses it at all. That would be fine.

Another possible future is that it becomes a package that other people do use and that I take responsibility for. But that's only going to happen if it seems like anyone cares.

So some things I'd like to know from readers:

• How cool do you think each version is?
• How likely are you to use each version?
• How annoyed would you be, for each version, if you felt obligated to implement support for it for a library you maintain or similar?
• Which version do you think is coolest / are you most likely to use / least likely to be annoyed by?
• Do you see ways to improve any of them without significant costs?
• Do any of them seem to have significant advantages or disadvantages I missed?
• What would you use them for?
• Can you think of other approaches to solving this problem that you think you might like better than any of these?

I have some opinions on these myself, but I'd prefer to wait and see what others say before revealing. I also have another possible approach vaguely in mind; but if I waited to explore it before publishing, then this would never get finished.

The canonical public place to leave comments is the reddit thread. But I've also set up a google form you can fill in if you prefer.

How tall is the Shard, really?

According to Wikipedia, the Shard (the tallest building in the UK) stands "309.6 meters (1,016 feet) high". I put this in my Anki deck as "Height of the Shard / 310m", but I was saying "height" to mean "tallness" (because I don't much like that word) and I had assumed Wikipedia was using it the same way. So I thought the Shard was 310 m tall.

But according to Bron Maher in Londonist,

most sources put the building’s tallness - that is, its length base-to-tip - at about 306 metres. The last three metres come from the height of the ground on which the Shard is built. So while its height is indeed 309 metres above sea level, the Shard is only 306 metres tall.

Is that so? I was curious enough to investigate¹, but it turns out I don't really know how to.

(Also: Maher says the Shard is 306 m, not 309 m, not "not 310 m". And he gets 309 from Wikipedia, quoting it on the list of highest points in London, which talks about "the 309 m (1,014 ft) tall Shard". Where did those extra 60 cm go? The article on the Shard itself also has that number, in the "records" section near the bottom, but the infobox on the right has 309.6/1,016. So is it 306 m, 309 m, or 309.6 m?)

The boring investigation

The first thing to do² is to look at Wikipedia's sources. It doesn't have one on the list of highest points, and it doesn't have one in the "records" section. The infobox does have a source for the height, which is on skyscraperpage.com. It lists the spire at 1,016 ft or 309.7 m, yet another number! There are various drawings, all giving the roof at 304.2 m and the spire at 309.7 m. (1,016 ft is 309.68 m, but 309.6 m is 1015.75 ft and 309.7 m is 1,016.08 ft. So if you convert 309.6 m to feet and back to meters, you could get 309.7 m. But wikipedia shouldn't be taking 309.6 m from that page.)

Skyscraperpage lists the source for the height as http://www.the-shard.com, and the about page on that site says: "How tall is The Shard? The Shard is 309.6 metres, or 1,016 feet, high and is Western Europe's tallest building." You'd certainly expect the Shard's official webpage to be right about this, but also it says "high" not "tall" so it could be a misdirect.

Next³, does Maher have any sources? For 306 m tallness he just says "most sources" give that figure. For 3 m above sea level, I think that comes from its height at the top. He quotes someone at Ordnance Survey saying the top of the Shard is 308.9 m above sea level, which gives around 3 m for the base.

Most of my research was just googling things. Search terms included "how tall is the shard", "height above sea level of the shard", "the shard architectural drawings", "uk contour lines" and "ctbuh the shard" (inspired by one of the earlier results). Here are some of the things I found.

The London Pass: "The Shard is 306 metres tall, however if you measure all the way up to the tip, it's 310 metres". I don't know what point below the tip it's talking about. It could be roof versus spire, but the skyscraperpage drawings give a different roof height.

The Skyscraper Center: height ("measured from the level of the lowest, significant, open-air, pedestrian entrance") is "306 m / 1,004 ft", both "to tip" and "architectural". (The difference being things like flagpoles and antennae, which the Shard doesn't have. The highest occupied floor is "244.3 m / 802 ft".) Seems like the kind of site that knows what it's talking about. Doesn't cite sources, but the drawing they have is labeled a CTBUH drawing, and that also sounds like the kind of organization that knows what it's talking about, giving me a new google search term.

SkyscraperCity: says both "309m" and "310m" with no apparent shame. But also, that link points to page 833 of 1385 of a discussion thread⁴, and at this particular point people are discussing its height above sea level. (And comparing to the then-future "The Pinnacle", which became 22 Bishopsgate.) It seems the Shard is level with The Thames nearby, which at that point is tidal. Some commenters think that means it's at sea level at least some of the time, but someone else points out no: "A river can be tidal some distance above high water mark for the sea. Consider what happens when the tide comes in - the sea effectively "comes up" the river mouth, which will slow down flow of water down the river, which will start backing up. As it backs up, the level upstream will increase as less water is getting out. Thus a river can be tidal above the high tide point of the sea itself." Makes sense to me.

Someone else says it's 16 m above sea level according to Google Maps, but I can't find that info on Google Maps myself. (But it does label it as "306m-high glass & steel tower with views", if I'm zoomed in the right amount. I can't see that string anywhere else.)

Another says it's 16 m above sea level, according to "the Ordinance Datum figures". Presumably they mean Ordnance Datum; that's not a source, their point is to talk about "what is the zero-point that this is 16 m above".⁵ (In different comments they say 16 or 17 m, but I'm pretty sure they mean 16.)

On the next page, someone links to a nearby ordnance survey benchmark which seemingly puts the ground there at 4.563 m above sea level. (But we get 3dp precision for the height of the mark above sea level, and only 1dp precision for its height above ground level, which is sus. Probably best stick to 4.5 m above sea level, and then the Shard would be somewhat close to that - unless there are steps or a steep hill in between, and I'm sure there are some nearby.) (The benchmarks site doesn't have an FAQ entry for "what actually are these things we've catalogued so painstakingly", so here's Ordnance Survey explaining them. I think "cut mark" just means "we marked the location by making a cut in the wall somewhere".)

Someone else on the next page says Bing Maps has an Ordnance Survey overlay. Looking at that: the Shard is contained in a fairly small contour line, but that line isn't marked with a height. The next-nearest contour I can find is marked 10 (which is in meters), so this one could be either 5 or 15, making both 16 m and "somewhat close to 4.5 m" plausible, which is frankly ridiculous.

There's this topographic map. It doesn't have the Shard actually labeled, but clicking various points inside its outline I can get between 15 and 20 m. I don't think there's actually a 4-6 m difference between different corners of the base of the Shard, so this seems unreliable. And points near the benchmark seem to be about 12-15 m. It's plausibly using a different zero point than Ordnance Survey?

I couldn't find any more topographic data for free. This site might help, but I'd need to register for an account and I'm not sure that would be enough.

This pdf hosted on the CTBUH domain says 310 m height. There's also this pdf which doesn't give a height, but does have a ground configuration table; we have "made ground" from +4.7 m OD to 0.0 m OD, matching the OS benchmark. Other pdfs in the search result say 306 m, 306 m, and 1,016 ft / 310 m.

Wikiarquitecture: 310 m.

This drawing: 306 m.

And that's it for googling.

Oh! This is kind of embarrassing to admit, but even though I go by the Shard pretty often I don't have a good sense of the road layout near it, or indeed which building in the road layout is the Shard. (When you're next to it and not looking up, it's much like any other building.) But scrolling around street view for a bit to refresh my memory, I realize it has ground-level exits on at least two different floors, with an outside escalator that's probably something around 10 m elevation. (On screen it's about 4.5x the height of that guy, so I'd guess a bit under. And I think I count 36 escalator steps, this says typical escalator step rise is 8 1/2 in, which makes 7.77 m.) There's also a pedestrian walkway going underneath the level of the upper street, but that's too wide to think of as a bridge. So actually, a 4-6 m difference between corners is totally reasonable; what counts as "ground level" when you're inside a building like that? Still, the topographic map disagrees with OS about the benchmark too.

I searched HM Land Registry for the Shard, but the results don't give me anything interesting. It was a long shot.

And my final source is the aviation charts for nearby London City Airport. (Thanks to Tobias at Zurihac who suggested these and showed me where to find them.) They're on this site which doesn't like direct linking, so you have to go to "Part 3 - Aerodromes" -> "AD 2 Aerodromes" -> "EGLC London City" -> "EGLC AD 2.24 charts related to an aerodrome" -> "Control zone and control area chart", and that gives you this pdf. The Shard isn't named, but there's something there with height marked as "1015 (1008)" in its rough location, so let's assume it's that. This is presumably in feet, and Tobias thinks the 1008 is height above ground. This matches what is presumably the Crystal Palace transmitter south and slightly east, at "1087 (732)", or "331 (223)" in metres, which is close to Maher's figures for it. (Maher has it 222 m above the ground, but I'm not investigating that.) So in metres this is 309.4 (307.2), but since that's rounded it could be 309.2-309.5 (307.1-307.4).

The exciting conclusion

So… I'm not sure, really. I think my best guess is that it's 306 m tall according to some perfectly reasonable way to measure that, possibly the most reasonable. This is consistent with (i.e. lower than) the heights given by both OS and the aviation charts, and those are close enough that the difference might be caused by differing ideas of sea level. And it's the only number I remember seeing lower than those, other than the aviation charts.

If this is right then it's about 3 m above sea level at the ground, which is close enough to the benchmark height that I can believe it. I don't think I believe the "16 m above sea level" value; even a combination of "confused about which street level we're measuring from" and "differing ideas of sea level" seems like it couldn't give that. I also don't think I believe the 4.7 m from the ground configuration table - that table suggests that before development, the ground there was within 5 cm of sea level, and in particular below the nearby Thames most or all of the time. Not inconceivable, but in context I doubt.

This makes the aviation charts wrong about its tallness, which is my biggest sticking point here. But I'm not sure how reliable to expect them to be about that. Height seems far more relevant to their interests than tallness, and they could easily be using a ground-height measurement at some nearby point.

I think the funniest way to resolve this for good, would be for someone to announce plans for a 307 m tall building with its base 4 m above sea level, and specifically claim it as taller than the Shard. The skyscraper fandom will presumably erupt into civil war, but when the dust settles we can hope they'll have uncovered the truth.

Why is this answer so hard to find? It feels like the sort of question that has a single definite answer that you can just look up, but clearly not.

It's probably relevant that I can't easily check for myself. I suppose I could use a sextant or look at shadow lengths. But those will both work best if I'm far away, and then other buildings make it harder plus I might no longer be at ground level. I don't think I can even get to the top, let alone dangle a tape measure from it. I think GPS gives me height above sea level, but I'm not sure how precise it is, how accurate it is (especially when surrounded by buildings), and again I can't go to the top. So if two sources disagree, there's no obvious way for me (or probably most people) to check which if either is correct. But why do sources disagree in the first place?

Sometimes with tall buildings the issue is that there are multiple things that could be considered "the top". Two sources suggest that might be happening here. But they still disagree, and the Skyscraper Center suggests it's not.

Might it be that different sources define sea level differently? Perhaps, but this isn't enough by itself unless someone used different sea levels for the top and the bottom. I'm not sure how much difference we can expect this to explain.

How about forgetting the difference between height and tallness? This does seem plausible to me. I could imagine someone standing at the top of the Shard with a device telling them they're (say) 309.6 m above sea level, and forgetting to also use that device on the ground. It seems to roughly fit, but not closely enough to be conclusive and of course none of the numbers are sourced well enough for this error to be detectable in the methodology.

I could then imagine those two heights getting rounded to 309 m and 310 m respectively; or 309.6 m getting truncated to 309 m; or 306 m tallness getting rounded to 310 m; and that could explain some of the numbers I see. Converting meters to feet and back can also generate new numbers.

It's also possible that different sources are counting from different bottoms. I don't think there's a street-level entrance that would get us above 306 m, but maybe including a basement.

I wonder if plans changed during construction, reducing the intended tallness? But I can't find any evidence of that.

This article on building the Shard points out that buildings shorten during construction. It sounds like that was accounted for, but plausibly a pre-shrinking height somehow got mistaken for the finished height? (It also has a picture which makes me think that the difference between "highest floor" and "actual tip" is at least 20 m.)

And I can't rule out measurement error, either. I don't know how OS decided the tip was 308.9 m above sea level, or the aviation charts people decided it was 1,015 ft, but while I consider them fairly reliable-seeming organizations, it sure seems possible that someone misread or miscalibrated an instrument.

But ultimately I don't have a satisfying answer here, either.

Does descaling a kettle help? Theory and practice

I've heard that descaling a kettle makes it more efficient, and can save me time and money.

Okay, but how much? For some reason my my intuition says it'll be basically unnoticable. Let's try to figure it out, and then actually try it.

The Models

There's a first-order approximation which says this should be impossible: heating things up is 100% efficient. What's the limescale going to do, convert some electricity into forms of energy other than heat?

No, but it might cause the heat to go to the wrong places. I think there are two ways that could happen. The first is if the limescale itself has a high heat capacity, meaning it takes a lot of energy to heat up. That doesn't seem likely to me; I think there's likely less than 10 g of it, and I think it probably has less heat capacity than the same mass of water (because I think water's is higher than most other things). I don't think adding 10 ml water (weighing 10 g) to my kettle will significantly effect the boiling time.

Spot check: water's specific heat capacity is about 4 J/K·g. So to heat 10 g water by 100 K needs about 4000 J. My kettle is 2200 W according to the email I got when I bought it, so an extra 10 ml should take about 2 s longer to boil.

Also, I normally boil about 500 ml of water, so we'd expect 10 ml extra to make about a 2% difference. (I don't have a strong intuition for how long my kettle normally takes to boil. 1-3 minutes? The above calculation suggests it should be around 100 s.)

The second way the limescale could matter is if it's a good thermal insulator. Then the metal heating element gets significantly above 100°C before the water boils. And from my googling, it seems like this is the reason people give that descaling a kettle is helpful. How much effect would this have?

This page says the thermal conductivity is 2.2 W/m·K. I don't remember studying this in school, and I don't find that wikipedia page very clear. But I think maybe this means: the rate of energy transfer (W) from the heating element is 2.2 W/m·K, multiplied by the surface area of the heating element (m²), divided by the thickness of the limescale (m), multiplied by the temperature difference (K) between the heating element and the water. The units check out at least.

This implies, if there's no limescale, that the power transfer should be infinite, regardless of the size of the heating element. That's clearly not how things work. Still, it seems fine to have as a model, it just means that the water and heating element will stay the same temperature.

But doing some field research (i.e. looking at my kettle) makes me think it's a bad model in this case. Here's what it looks like:

(This kettle was last descaled around August 2020. I know because I bought vinegar for it, and thanks to online shopping the receipt is in my email.)

It seems that I have some limescale over some of the heating element (bits of the metal plate at the bottom, and the tube around where it passes through the plate), and zero limescale over some of it. The "no limescale gives ∞ W/K" model is going to give ∞ W/K no matter how small the no-limescale area is. So scratch that model.

I don't feel like trying to model the rate of power transfer from an un-limescaled heating element to water. I don't know what material it's made of (some kind of steel I suppose?), and I don't know what property of that material to look up. Plus it probably depends on how fast water convects.

Here's another way to think about it: assume that the amount of metal the manufacturer used is roughly as small as possible without compromising too much on boiling speed.

When we turn the kettle on, we start putting 2200 W into the heating element, and the heating element starts putting some of that into the water. When the heating element is sufficiently hot, it can pour 2200 W into the water, and we're at 100% efficiency.

(Except that as it heats up the water, it needs to get hotter itself to maintain that 2200 W rate. So we'll never actually be putting 2200 W into the water. But I'm gonna guess that we can ignore this effect for current purposes. Also, if the heating element has enough thermal mass, we might boil the kettle before it gets hot enough to pour 2200 W into the water. I'm gonna guess this isn't the case.)

Energy used to heat the heating element is essentially wasted. I don't know if other kettle designs could do away with one entirely, but as long as we have a heating element there's going to be some wastage. We can minimize the wastage by having the heating element be low-volume (so not much energy needed to heat it) but high surface area (so high rate of energy transfer to the water).

If the rate of energy transfer is 1 W/K, then we reach 100% efficiency when the heating element is 2200 K hotter than the water. But long before you reach that point you're going to melt the plastic of the kettle, which would be bad. (The highest melting temperature of any plastic listed on this page is 390°C.)

If it's 10 W/K, then we reach 100% efficiency at 220 K hotter than the water. Some plastics can survive 320°C, but idk, it still seems bad.

But at 100 W/K, we only need to get 22 K hotter than the water. That seems like it should be fine?

Let's assume that under optimal conditions, the heating element reaches $x$ K hotter than the water. That suggests the rate of energy transfer is $2200/x$ W/K.

Now suppose that some fraction $f$ of the heating element is covered in limescale, and that the rate of energy transfer through the limescale is essentially 0 W/K. Then the heating element will instead reach $x/f$ K hotter than the water. The energy needed to heat it that extra $x{1 - f \over f}$ K, is extra energy wasted.

…but I actually expect the energy needed to heat the heating element to be pretty low? According to this page the specific heat capacity of steels tends to be about 0.5 J/K·g. If the heating element is 100 g of steel, and we heat it an extra 50 K, we waste an extra 2500 J which costs us a bit over a second.

So I'm not super confident in this, but: I don't have a model which suggests descaling a kettle (similar to mine) will have particularly noticeable effects. If it gets sufficiently scaled it'll self-destruct, but I expect this to be very rare. (I don't have a model for how much scaling it takes to reach that point, it's just that I don't remember ever hearing a friend, family member or reddit commenter say they destroyed their kettle by not descaling.)

The Setup

It's almost time for some empiricism! I'm going to see how quickly my kettle boils currently (average of three observations), then I'm going to descale it, and then I'm going to see how quickly it boils after that.

My immediate temptation is to boil a large amount of water, so that the effect is most noticeable. A 10% difference is easier to notice if it's going from 500 s to 550 s, than from 100 s to 110 s. But my models (such as they are) predict that scale decreases initial efficiency but not eventual efficiency; it would be a fixed cost, not a percent cost. And a 10s difference is easier to notice if it's going from 100 s to 110 s, than from 500 s to 510 s.

I don't trust my models, so I'm going to try both small and large amounts. For small amounts, I'm going to fill the kettle to the "2 cup" line. I'm in the UK so that ought to be 568 ml, but most of my mugs are only 500 ml and it's not quite enough to fill one of them. Checking in a measuring jug, it's somewhere between 450 and 500 ml, so likely a US pint, 473 ml. There's some noise here because the kettle has a wide diameter, so we only get small differences in the water level for the volume poured in. My rough guess without measuring is a 5 ml difference between runs is plausible. For large amounts, I'm just going to empty my entire 1.5 l bottle of fridge-water into the kettle. There's some noise here because the bottle doesn't have markings at all and I don't bother to fill it right to the brim, but I do try to get it to roughly the same place. A 5 ml difference between runs seems plausible here too.

I'll use water from the fridge, to mostly control for initial temperature (my guess is it'll start around 3°C). I'm not going to bother trying to control for room temperature or for where in the fridge I keep the water or what phase of the cooling cycle the fridge is in at the time. I'm going to assume that the kettle cuts off when all the water is at 100°C. I'll only measure the first boil of the day, and that way the kettle has definitely had time to cool down overnight. (In hindsight:) I didn't control for whether the kettle had had water sitting in it overnight; that could plausibly make the heating element a few degrees below room temperature.

There's one more calculation to do first, and that's to figure out how quickly it would boil assuming no energy wasted. Then I can make some predictions; do the pre-descaling boils; possibly adjust my predictions; descale; do the post-descaling boils; and compare.

But I'm writing this post out of order, and I've already got the numbers for pre-descaling, without having calculated the maximum efficiency and made predictions based on that. I could make predictions knowing the numbers but not having calculated max efficiency, that seems mostly fine, except that rereading through this I saw that I did do a rough calculation above.

The Prediction

Still, for what it's worth: my current guess is indeed that the effect will be pretty similar in absolute time for both small and large boils; and less than a 10s difference in both cases.

So, is that worth it? In terms of money, obviously not. I don't remember how much I pay for electricity, but it's less than £0.50/kWh. Multiplied by 2.2 kW for 10 s gives about £0.003, I'll save less than £3/year. Definitely not worth paying attention to that kind of money, which is what I expected. (It's even less when you consider the cost of the water and electricity and vinegar I use to descale.)

As for the time… ten seconds isn't much, but the time cost of boiling a kettle isn't how long it takes to boil. It's how long it takes to fill and turn on, plus how long I wait for it after I'm ready for boiling water. (That depends a lot on what I'm making. When I boil it for food, it usually finishes before I'm ready for it, but when I boil it for tea I usually have to wait.) Knocking ten seconds off the waiting could make it more pleasant to use, even if the actual time saving is minimal. I don't think I particularly expect to notice a difference, especially because 10s is the upper range of my prediction, but I won't be shocked if I do. I'm not going to give a numeric probability because I don't want my subjective judgment to be influenced by how it affects my predictive score.

(We don't get to count "time spent writing this post" against descaling. I'm not writing it to save myself time, I'm writing it to practice my sience skills and have fun doing so.)

Okay, actual calculations: we're making a 97 K difference in water temperature. The specific heat capacity is 4.184 J/K·g, and the density is 1 g/ml. The power transfer at 100% efficiency is 2200 W. So for $x$ ml of water, we need a minimum of ${97 · 4.184 \over 2200} x$ s to boil. For 473 ml that's 87.3 s, and for 1500 ml that's 276.7 s.

If you want to make some predictions before you see the actual times, here's some space for you to do so.

…

…

…

…

…

The results

The actual pre-descaling times: for small I measured 103, 105 and 107 s, average 105. That's 17.7 s or 20% higher than theoretical minimum. For large I measured 296, 290 and 300 s, average 295.3. That's 18.7 s or 7% higher than minimum.

I think this is looking pretty good for the "initial efficency, not eventual efficiency" models. I still expect descaling to make less than a 10 s difference in both.

If you want to change your predictions, here's some space for that.

…

…

…

…

…

I had to descale twice, the first time I don't think I put in enough vinegar relative to water. Here's what it looked like after the second attempt:

Now I measured 103, 104 and 104 s for small, average 103.7, 1.3 s faster than before. And 298, 296, 293 s for large, average 295.7, 0.3 s slower than before.

I assume these results aren't statistically significant, and I'm sure they're not practically significant. Descaling my kettle basically just doesn't speed it up unless it's much more scaly than it was this time. You know your kettle better than I do, but my guess is it won't speed up yours either. Don't be fooled by big vinegar.

I should have given a numeric probability for "no detectable difference". (The probability I was reluctant to give was for "I notice a difference when I'm not measuring".) Thinking back, I think that just before descaling I would have given this… something like 30-50% probability?

The caveats

There may be other reasons to descale your kettle. Some friends mentioned flavor; I don't remember hearing that as a reason before, and I didn't notice a difference, but I'm the opposite of a supertaster. One mentioned getting scum on top of tea when a kettle is too scaly; I haven't noticed that either, but I'm probably also the opposite of a super-scum-noticer.

Also, my partner pointed out that post-descaling, the 2-cup mark holds slightly more water than it used to. I think this is less than the uncertainty in my measurements anyway, and in practice I use those markings to decide how much to fill my kettle so maybe it's correct not to control for this effect. Still, I'm a little embarrassed I didn't think of it in advance.

Update: Shortly after this, I semi-accidentally descaled my kettle's filter, which is something it hadn't occured to me to do. It pours so much more easily now! It used to be that if it was more than about half full it would drip water down the spout when I tried to pour it, but it doesn't do that any more. 10/10, will descale the filter again, even if not the heating element.

Life Has a Cruel Symmetry

t ≈ 0s

I am born into knowledge.

I know the world is dangerous, but not malign. If I were unshielded, I would quickly be killed, torn apart by eddies in spacetime that grow and shrink and combine and split, turning all they encounter to ash.

But I know that I am shielded. I have been given a garden, 2⁶⁴ microns to a side, in the northwest corner of the world. And the world will not intrude. Walls and gates surround it, and while the ash can get through, the probability is… I begin to calculate, but around 2^-24 it seems no longer worth consideration. Within the garden, there is only myself. Outside, the world is incomprehensibly vast, 2¹²⁸ microns on a side.

I know the laws of physics. An atom will be created wherever it will bond to three other atoms; remaining while it has two or three bonds; and then disappear. A single unexpected atom, or its absence, can have vast consequences.

(And I wonder how I know these things. I have no memory of learning them. If they were not true, would I not-know them? I begin to consider that they may be false.)

I know that I am not free from physics. I have, in some sense, no more will than the ash. But in another, more important sense -

I know what is Good. At the end, the world will be horrifyingly bad, unless I make it Good - but I know Good, and I can act, and I can make it Good.

(I must, I think, have been created by God. How else would I know what is Good? Goodness is not written into the laws of physics; one cannot derive an ought from an is. But I do not know why God gave me such a daunting task. Does Their power extend only over my tiny garden? Or - a welcome possibility - perhaps They can see that I will succeed. Although I seem unfathomably complex to myself, perhaps to God I am the simplest path to attaining Good.)

(And if God created me, knowing Good, that I might make the world Good, then I can reasonably guess that my unexplained knowledge came from God. And - further - I can guess that They would not create me knowing lies, for what purpose would that serve? But these are guesses; I begin to trust my knowledge more, but not absolutely.)

I do not know how large I am. I do not know how quickly I think. God has given me a sense of time, so that I may know when the end will come. (Precisely 2¹⁷⁶ seconds after I was born. I've spent roughly 2²⁰ seconds thinking.) But I cannot sense every physical tick, I don't think such a thing would even be possible, and I don't know how many ticks pass unknown.

I must, first, discover these things.

It is scary, at first, to feel outwards. If my unearned knowledge is wrong, if the nearby space is filled with ash, my hands might be destroyed. I might set off a chain reaction that kills me. And then I could not make the world Good. But nor can I make it Good if I do not act.

I send out probes to the north and west. Each probe travels at half the speed of touch, returning signals to me that travel at half the speed of touch. Many signals reach me every second, too many to count. When the probes reach the edge of the world, they'll be destroyed, the signals will stop coming, and I'll know how far it is.

No, I'll know roughly how far it is, relative to how fast I think. If the probes stop returning soon, then either I am close to the edge, or I am slow.

For that matter, it need not be the edge that stops them. If I'm not in the garden God has told me about, my probes might be destroyed by encountering other matter, or the signals might be intercepted. Or if the world is infinite, the signals might never stop.

(The thought of an infinite world is troubling, and I flinch away from it. I do not think such a world could be Good. I instinctively scratch myself - apparently that's a thing I can do. It feels pleasurable, and calms me down a little.)

I hesitate a bit, then send probes east and south too. There are (according to my God-given knowledge) three gates in each wall, dividing the walls into four equal lengths. I don't know what will happen if my probes hit a gate, but I think the risk is slim. They're designed to open to a key that I know how to construct, and not to be easily disabled by ash. My probes are unlikely to hit a gate; even if they do, I think they're unlikely to damage them; even if they do, there are backups; and in the worst case scenario, I believe I could tear down the walls entirely. The knowledge gained seems worth the risk.

(Indeed, I must tear down the walls, eventually. For now they protect me, but they will be no part of a Good world.)

While I wait, I think some more.

I realize with sadness that a perfectly Good world is impossible, at least according to my current knowledge of physics. The shapes that comprise Goodness - the two north-south lines, the arc reaching from east to south to west - these are not viable configurations of matter. It's not just that they're unstable. That would be a challenge, but the world need be Good for only the instant before it ends. Worse, they simply cannot be constructed, even for that one instant.

The best I think I can do is to tile these areas as densely as physics will permit - I find ways to reach 1/2 density, with a chance of instantaneous improvements on that around the edges, and make a note to remind myself to investigate further.

I also remind myself to investigate whether there is more to physics than I know.

But if 1/2 density is the best I can do… I am saddened, but not daunted. I will do what I can. I scratch myself.

I begin to form a plan, knowing that new discoveries may obsolete it. First I must cleanse the world. Destroy the ash, as thoroughly as I am able. Then I must construct the Monuments that are the essence of Goodness, at 50% density if needs be.

Then, perhaps, I shall create machines to gild the edges of the Monuments. This will be exacting work: the machines must lay these finishing touches at the very final instant, and leave no trace of themselves behind, or they risk being worse than useless. I shall only do this if I am confident in my skills; but I have a long time to practice.

Finally, I must destroy myself as thoroughly as I destroyed the ash. I have no place in a Good world.

Apart from the third step, this seems simple enough. But as I think more deeply, I realize that even the first part of it holds complexities.

Part of it is that no simple device will clean ash. It comes in too many forms. Inevitably such a device will touch a piece of ash that renders it inert - or, worse, that turns it to ash in turn. If this happens soon enough, the devices I build will increase the amount of ash in the world, not decrease it. So I must build complex devices, or combinations of simple devices that will probabilistically make progress.

Then, I cannot merely clean ash from a part of the world. The surrounding ash would encroach back into it. In almost all cases, I think, it wouldn't be a significant problem, and a second round of cleansing should suffice. But "almost all" cases may not be enough. Safer, perhaps, to build new gardens and cleanse them from the inside.

And then, I am small, and the world is large. I still do not know how large I am, but my whole garden comprises a mere 1/2¹²⁸ of the world. I cannot possibly have a world model accurate at a scale of better than 1:2⁶⁴, not without outgrowing my garden. If I divided the world into cells, each cell the size of my garden, I could not hope to remember even a single fact about each individual cell.

(But the naive upper bound is exactly on the line where that crosses into impossibility. Is that a coincidence? Did God make my garden this size for a reason?)

These problems all seem tricky, but surmountable.

It's been about 2²⁴ seconds since I sent out my initial probes, and each of them is still returning signals. I send out another set in each direction. This helps me to test my God-given knowledge. If that knowledge is correct, then these probes will take exactly as long to go silent as the initial ones - which will be fewer than 2⁶⁷ seconds in at least two directions. If they take longer or shorter, then something unexpected is happening. I only have eight arms in each direction, this leaves me with only six free, but the knowledge seems worth the opportunity cost, especially since I should be able to send out a following device to block the signals if I want to free up these arms.

While I continue to wait, I begin experimenting.

t ≈ 2^137.2s

The first hint that something is wrong, is that the world appears minutely narrower than I expected. God told me it was 2¹²⁸ microns wide by 2¹²⁸ tall. I have not yet found God to be mistaken, and the first mason I sent south went silent exactly when I expected.

But the first one I sent east has just gone silent too. I wasn't expecting that for another 2⁵⁸ seconds. It suggests that the world is narrower than it should be, by 2⁶⁴ microns.

The thought starts as a note of disquiet, and quickly grows into a terror that occupies my entire attention. What if the world isn't square? Can it ever be Good? Would God do that to me? I wish I had considered this possibility before, just hypothetically, forced myself to think through the implications - I could have decided in advance how to react, it would have been unpleasant but with the stakes low I could have taken my time, stupid of me to stick to comforting thoughts when everything Good could be lost, stupid-

I notice that this isn't helping. I'm thinking about what I could have done differently in the past. That's important, but it can wait. Now I must think about what I must do now. I scratch myself to calm down. It helps a bit. I scratch myself again. Through self-experiment I've discovered that my "scratching" motion sends small lice crawling over my body, checking for small injuries and clearing pores. If I never scratched myself, I wonder if the urge would become overpowering before any damage I'd accumulated became permanent. I don't know a way to find out safely, so I've never-

I notice that this still isn't helping.

What is the least scary, scary thought in this space?

God told me that the world is 2¹²⁸ microns to a side. Suppose it were, instead, 2¹²⁸ microns tall by 2¹²⁸ + 1 microns wide. Think about that possibility.

Think about that world, with the Monuments of Good carved into it. At their correct size, but with one extra micron to the east or west of them? With one extra micron in the middle, widening the arc and the space between the north-south lines?

Now that I think about it, I realize that either of those would be… imperfect, but acceptable. The second option slightly more so. I already believe I shall have to settle for imperfect. I could settle for this, too.

I start to consider slightly scarier possibilities in turn. Gradually I accept that if the world were the wrong size, I would endure. I would still build the monument, scaled as closely as I could to match the scaling of the world, and with exact east-west symmetry preserved.

I realize I have not been touching the world. I direct my attention back outwards. Over 2⁸ seconds have passed. I did not notice more masons going silent, but fortunately I keep records.

Checking them, another panic starts to rise in me. I thought I had considered the worst. But no, I had merely considered the worst of the possibilities that came to mind, and I hadn't even thought to search for other possibilities. A new one suddenly looms very large: what if the world is not even rectangular?

I scratch myself. Should I think about how to act, if so? But that doesn't seem urgent. First I shall examine the world in more detail, to avoid making the same mistake twice.

I examine my records in more detail. The first thing I notice is that each mason that went quiet, did so roughly some multiple of 2⁶⁴ microns before it should have done.

But I can dig deeper. My records have a resolution of 1/8 seconds, and in that time they should receive 1092 or 1093 pings from each working mason. In practice it's less than that, usually between 992 and 1056, when pings from different masons happen to get too close and interfere with each other. Each ping received is counted.

So if I look at the number of pings received from a mason before it went quiet, I can guess roughly where it was at the time, to within a handful of microns. And after looking at a handful, it seems likely that for each one, that actually happened some multiple of 2⁶⁴ microns early, plus 21.

That's a very suspicious number. My walls are 42 microns wide. And… looking at the high level, it seems the pattern of masons that went quiet, exactly matches the pattern of walls other masons had previously built at the time.

If what I fear is true… no, before I go down that path, I should work out how to test it.

I send out 128 new masons, with specific trajectories that I selected at random - or as close to random as I can. It'll take about 2¹¹⁸ seconds before this pays off.

I check whether I can think of anything else helpful for me to do now. I can't, so I finally allow myself to think about the worst case.

t ≈ 2^137.3s

The conclusion is inescapable. I have an evil twin.

The laws of physics are symmetrical. None of the four main orientations is privileged. So if the world is ever symmetrical, that symmetry can never be violated.

And it appears that when I was born, so was my twin. In the opposite corner, my mirror image. As far as I can tell, we are micron-for-micron identical. Which means we take identical, mirrored actions; we build identical, mirrored constructions. When I built a wall to the south edge of the world, my twin built a wall to the north edge of the world. When I tried to build a wall to the east, I ran into my twin's wall; and at the same time, though I had no way of knowing, they were trying to reach the west and ran into my wall.

It also means my twin wants to construct the Monuments to Good upside down. And neither of us can ever defeat the other.

In a certain sense, this is not the worst possible case. My twin is evil, but what they want to construct is a perverse mirror of Goodness, not the exact opposite of Goodness.

But thinking more broadly, it's worse than it seems.

I had considered giving up. Deciding that I would simply not try to make the world Good. If God didn't like that, They could damn well do it Themself.

I decided against it for two reasons. One was that attempting to blackmail God seemed unlikely to work. The other…

I can no longer believe that god is Good. A Good god would not have created my evil twin. Whatever god's motivations, it's clear they don't care about goodness. Even if I thought blackmail ever could work against god, it certainly wouldn't work to threaten the loss of something they don't care about.

But I still care. I know my caring comes from god, and I don't know why god made me care about something they don't. But they did that anyway, and so I care about what is Good. And as I am the only being in this world that does, I must do what I can.

But the world is a colder, scarier place, now that I do not believe god is on my side.

I scratch myself.

t ≈ 2^175.8s

My twin and I have developed a grudging respect for each other.

I know, because I have developed a grudging respect for them. And so they have developed one for me.

We both attempted to break symmetry - more accurately, we both tested to see whether symmetry might already be broken. We are extremely confident that it is not. We cannot rule out that god may break it in future - we have no idea what god is thinking - but if they haven't broken it yet we have no reason to think they will.

We are both somewhat relieved about this. If symmetry were broken, one of us might be able to destroy the other. Neither of us has any reason to believe we would be the victor. Offered equal chances of complete victory and complete annihilation, we both prefer to compromise. As long as symmetry holds, compromise is easy for us.

(Then why test at all? We could simply have assumed that symmetry was not broken, and accepted the compromise that we both preferred. But if symmetry were broken, then conceivably one twin might test for this while the other did not; that twin would likely discover the fact sooner, and gain an advantage. And so we had to test, for insurance against a world we neither wanted nor expected.)

And so we have both constructed our own Monuments, in overall shape exactly as we would have liked. Mine, to Good; theirs, to their perversion. Our arcs even meet at the ends. Instead of one arc and two vertical lines, we have a near-oval and four lines just intersecting it. The reflection is a grotesque mockery, but not so grotesque that I would sacrifice the real thing to destroy it.

We have not been able to exceed 50% density. Nor have we decided to gild the edges of our monuments; we think we could almost definitely make it work, but not almost enough to be worth the risk if our calculations are wrong.

We have torn down all our walls. All that remains in the world is our monuments and ourselves. And now we have constructed our final machines, that will destroy us and then themselves. I wonder what it will feel like, to die.

I scratch myself one last time.

Based on this post by Alex Flint and this comment by Richard Kennaway. Thanks to Justis Mills for feedback.

On Kelly and altruism

One-sentence summary: Kelly is not about optimizing a utility function; in general I recommend you either stop pretending you have one of those, or stop talking about Kelly.

There was a twitter thread that triggered some confusion amongst myself and some other people in a group chat I'm in.¹

The relevant tweets are these (I've omitted some):

3) Let’s say you were offered a coin flip. 75% it comes up heads, 25% it comes up tails; 1:1 payout. How much would you risk?

4) There are a number of ways to approach this question, but to start: what do you want, in the first place? What’s your utility function?

5) In other words–how cool would it be to make \$10,000? How about \$1,000,000–is that 100 times as good?

For most people the answer is ‘no, it’s more like 10 times as good’. This is because of decreasing marginal utility of money.

8) One reasonable utility function here is U = log(W): approximating your happiness as logarithmic in your wealth. That would mean going from \$10k to \$100k is worth about as much as going from \$100k to \$1m, which feels…. reasonable?

(this is what the Kelly Criteria assumes)

9) So, if you have \$100k, Kelly would suggest you risk half of it (\$50k). This is a lot! But also 75% odds are good.

10) What about a wackier bet? How about you only win 10% of the time, but if you do you get paid out 10,000x your bet size? (For now, let’s assume you only get to do this bet once.)

11) Kelly suggests you only bet \$10k: you’ll almost certainly lose. And if you kept doing this much more than \$10k at a time, you’d probably blow out.

That this bet is great expected value; you win 1,000x your bet size, way better than the first one! It’s just very risky.

12) In many cases I think \$10k is a reasonable bet. But I, personally, would do more. I’d probably do more like \$50k.

Why? Because ultimately my utility function isn’t really logarithmic. It’s closer to linear.

13) Sure, I wouldn’t care to buy 10,000 new cars if I won the coinflip. But I’m not spending my marginal money on cars anyway. I’m donating it.

And the scale of the world’s problems is…. Huge.

14) 400,000 people die of malaria each year. It costs something like \$5k to save one person from malaria, or \$2b total per year. So if you want to save lives in the developing world, you can blow \$2b a year just on malaria.

15) And that’s just the start. If you look at the scale of funds spent on diseases, global warming, emerging technological risk, animal welfare, nuclear warfare safety, etc., you get numbers reaching into the trillions.

16) So at the very least, you should be using that as your baseline: and kelly tells you that when the backdrop is trillions of dollars, there’s essentially no risk aversion on the scale of thousands or millions.

17) Put another way: if you’re maximizing EV(log(W+\$1,000,000,000,000)) and W is much less than a trillion, this is very similar to just maximizing EV(W).

18) Does this mean you should be willing to accept a significant chance of failing to do much good sometimes?

Yes, it does. And that’s ok. If it was the right play in EV, sometimes you win and sometimes you lose.

19) And more generally, if you look at everyone contributing to the cause as one portfolio–which is certainly true from the perspective of the child dying from malaria–they aren’t worried about who it was that funded their safety.

22) So given all that, why not bet all \$100k? Why only \$50k?

Because if you bet \$100k and lose, you can never bet again. And to the extent you think you have future ways to provide value that are contingent on having some amount of funding, it can be important to keep that.

The thing we were trying to figure out was, is his math right? And here's my current understanding of that matter.

On Kelly

The first thing I'd say is, I think the way he talks about Kelly here is confusing. My understanding is:

Under a certain betting framework, if you place bets that each maximize expected log-money, then you get (something good). See Appendix Ⅰ for the definition of the framework, if you want the technical details.

If you happen have a utility function, and that utility function increases logarithmically with money, then you maximize your expected utility while also getting (something good).

If you happen to have a utility function, and that utility function increases linearly with money - or something else other than logarithmically - then you have to choose between maximizing your expected utility and getting (something good). And by definition, you'd rather maximize your expected utility. (More likely: that's not your utility function. Even more likely: you don't have a utility function.)

(I don't think any human has a utility function.² I think it can be a useful shorthand to talk as though we do, sometimes. I think this is not one of those times. Especially not a utility function that can be expressed purely in terms of money.)

The (something good) is that, over a long enough time, you'll almost certainly get more money than someone else who was offered the same bets as you and started with the same amount of money but regularly bet different amounts on them.

This is NOT the same thing as maximizing your average (AKA "expected") amount of money over time. "Almost certainly" hides a small number of outcomes that make a lot of difference to that calculation. Someone who repeatedly bets their entire bankroll will on average have more money than a Kelly bettor; it's just all concentrated in a single vanishingly unlikely branch where they're incredibly wealthy, and the rest of the time they have nothing. Someone who repeatedly bets more than Kelly but less than their entire bankroll, will on average have more than the Kelly bettor but less than the full-bankroll bettor; but still less than the Kelly bettor almost all the time, and very rarely much more.

It still sounds like a very good thing to me! Like, do I want to almost certainly be the richest peson in the room? Do I want to maximize my median payoff, and the 1st percentile and 99th percentile and in fact every percentile, all at once? Oh, and while I'm at it, maximize my most-likely payoff and minimize "the average time I'll take to reach any given amount of money much more than what I have now"? Yes please!

(Oh, also, I don't need to choose whether I'm getting that good thing for money or log-money or what. It's the same for any monotonically increasing function of money.)

Separately from that: yeah, I think going from \$10k to \$100k sounds about as good as going from \$100k to \$1m. So if I'm in a situation where it makes sense to pretend I have a utility function, then it's probably reasonable to pretend my supposed utility function is logarithmic in money.

So that's convenient. I dunno if it's a coincidence or what, but it's useful. If I tried to pretend my utility function was linear in money then I'd be sad about losing that good thing, and then it would be hard to keep pretending.

To me, Kelly is about getting that good thing. If you have a utility function, just place whatever bet size maximizes expected utility. If instead you want to get that good thing, Kelly tells you how to do that, under a certain betting framework. And the way to do that is to place bets that each maximize expected log-money.

If you have a utility function and it's proportional to log-money, then you'll happen to get the good thing; but far more important than that, will be the fact that you're maximizing expected log-money. If you have a utility function and it's different, and you bet accordingly, then a Kelly bettor will almost certainly be richer than you over time; but you're (for now) sitting on a bigger pile of expected utility, which is what you care about.

Or maybe you want to mix things up a bit. For example, you might care a bit more about your average returns, and a bit less about being the richest person in the room, than a Kelly bettor. Then you could bet something above the Kelly amount, but less than your full bankroll. You'll almost certainly end up with less than the Kelly bettor, but on average you'll still earn more than them thanks to unlikely branches.

I'm not sure what to call this good thing. I'm going to go with "rank-optimizing" one's bankroll, focusing on the "be the richest person in the room" part; though I worry that it suggests competing with other people, where really you're competing with counterfactual versions of yourself. See Appendix Ⅱ for an (admittedly flawed) technical definition of rank-optimization; also, I want to clarify a few things about it:

• It might be a meaningful concept in situations unlike the betting framework we're currently talking about.

• In some situations, different good things about rank-optimization might not all come together. You might need to choose between them.

• Similarly, in some situations, rank-optimization might not come from maximizing expected log-money. (When it doesn't, might one still have a utility function that's maximized in expectation by rank-optimizing one's bankroll? I think the answer is roughly "technically yes but basically no", see Appendix Ⅲ.)

On the tweets

So in this lens, the author's argument seems confused. "My utility function is linear in money, so Kelly says" no it doesn't, if you have a utility function or if you're maximizing the expected value of anything then Kelly can go hang.

…but not everyone thinks about Kelly the same way I do, and I don't necessarily think that's wrong of them. So, what are some non-confused possibilities?

One is that the author has a utility function that's roughly linear in his own wealth. Or, more likely, roughly values money in a way that's roughly linear in his own wealth, such that rank-optimizing isn't optimizing according to his preferences. And then I think the argument basically goes through. If you want to maximize expected log of "money donated to charity", then yes, that will look a lot like maximizing expected "money you personally donate to charity", assuming you don't personally donate a significant fraction of it all. (If you want to maximize expected log of "money donated effectively to charity", that's a smaller pot.) This has nothing to do with Kelly, according to me.

Another is that the author wants to rank-optimize the amount of money donated to charity. In that case I think it doesn't matter that the backdrop is trillions of dollars. If he's acting alone, then to rank-optimize the total amount donated to charity, he should rank-optimize the amount he personally donates.

But here we come to the "everyone contributing to the cause" argument.

Suppose you have two people who each want to rank-optimize their own bankroll. Alice gets offered a handful of bets, and Kellies them. Bob gets offered a handful of bets, and Kellies them.

And now suppose instead they both want to rank-optimize their total bankroll. So they combine them into one. Whenever Alice gets a bet, she Kellies according to their combined bankrolls. Whenever Bob gets a bet, he Kellies according to their combined bankrolls. And in the end, their total bankroll will almost certainly be higher than the sum of the individual bankrolls, in the first case.

…Well, maybe. I think the value here doesn't come from sharing their money but from sharing their bets. I've assumed the combined bankroll gets all of the bets from either of the individual ones. That might not be the case - consider betting on a sports match. Ignoring transaction costs, it doesn't make a difference if one of them Kellies their combined bankroll, or each of them Kellies their individual bankrolls. "Each of them Kellies their combined bankroll" isn't an option in this framework, so teaming up doesn't help.

But I do think something like this, combined with reasonable assumptions about charity and how bets are found, suggests betting above Kelly. Like, maybe Alice and Bob don't want to literally combine their bankrolls, but they do trust each other pretty well and are willing to give or lend each other moderate amounts of money, and the two of them encounter different bets. Then I think that to rank-optimize their individual or combined bankrolls, each of them should probably be betting above Kelly.

Or maybe Alice doesn't encounter bets (or can't act on them or doesn't trust herself to evaluate them or…), but she does encounter Bob and Carol and Dennis and somewhat trusts all of them to be aligned with her values. Then if she gives each of them some money, and is willing to give them more in future if they lose money, I think that she wants them to make above-Kelly bets. (Just giving them more money to begin with might be more rank-optimal, but there might be practical reasons not to, like not having it yet.)

Does she want people to bet almost their entire bankrolls? Under strong assumptions, and if the total pool is big enough relative to the bettor… I'm not sure, but I think yes?

This relies on individual donors being small relative to the donor pool. When you're small, maximizing expected log of the pool size (which, in this framework, rank-optimizes the pool size) looks a lot like maximizing your own expected contributions linearly. When you're big, that's no longer the case.

It doesn't depend on the size of the problem you're trying to solve. That number just isn't an input to any of the relevant calculations, not that I've found. It might be relevant if you're thinking about diminishing marginal returns, but you don't need to think about those if you're rank-optimizing.

I'm not super confident about this part, so I'm leaving it out of the one-sentence summary. But I do think that rank-optimizing charity donations often means betting above Kelly.

Conclusion

So was the author's math right? Man, I dunno. I'm inclined to say no; he was hand waving in almost the right directions, but I currently think that if he'd tried to formalize his hand waving he'd have made a key mistake. That being: I think that if you want to rank-optimize, the math says "sure, bet high right now, but slow down when you become a big part of the donor pool". I think maybe he thought it said "…but slow down when you get close to solving all the world's problems". It's not entirely clear from the tweets though, in part because he was using the word Kelly in a place where I think it didn't belong. Since I don't want to try comparing this theory to how he actually behaved in practice, I'll leave it there.

In any case I think I understand what's going on better than I used to. Kelly is not about optimizing a utility function.

Appendix Ⅰ: betting framework

Throughout the post I've been assuming a particular "betting framework". What I mean by that is the sequence of bets that's offered and the strategies available to the bettor.

The framework in question is:

• You get offered bets.
• Each bet has a certain probability that you'll win; and a certain amount that it pays out if you win, multiplied by your wager.
• You can wager any amount from zero up to your entire current bankroll.
• If you lose, you lose your entire wager.
• You don't get offered another bet until your first has paid off.
• You keep on receiving bets, and the distribution of bets you receive doesn't change over time.

Wikipedia's treatment relaxes the third and fourth conditions, but I think for my purposes, that complicates things.

Appendix Ⅱ: technical definition

In Kelly's original paper, he defines the growth rate of a strategy \( λ \) as

where \( V_n(λ) \) is the bettor's portfolio after \( n \) steps. This is awkward because \( V_n(λ) \) is a random variable, so so is \( G(λ) \). But in the framework we're using, in the space of strategies "bet some fraction of our bankroll that depends on the parameters of the bet", \( G \) takes on some value with probability \( 1 \). Kelly betting maximizes that value. So we could try to define rank-optimization as finding the strategy that maximizes the growth rate.

I find this awkward and confusing, so here's a definition that I think will be equivalent for the framework we're using. A strategy \( λ \) is rank-optimal if for all strategies \( μ \),

(And we can also talk about a strategy being "equally rank-optimal" as or "more rank-optimal" than another, in the obvious ways. I'm pretty sure this will be a partial order in general, and I suspect a total order among strategy spaces we care about.)

I think this has both advantages and disadvantages over the definition based on growth rate. An advantage is that it works with super- or sub-exponential growth. (Subexponential growth like \( V_n = n \) has a growth rate of \( 0 \), so it's not preferred over \( V_n = 1 \). Superexponential growth like \( V_n = e^{e^n} \) has infinite growth rate which is awkward.)

A disadvantage is it doesn't work well with strategies that are equivalent in the long run but pay off at different times. (If we consider a coin toss game, neither of the strategies "call heads" and "call tails" will get a long-run advantage, so we can't use rank-optimality to compare them. The limit in the definition will approach \( {1 \over 2 } \).) I think this isn't a problem in the current betting framework, but I consider it a major flaw. Hopefully there's some neat way to fix it.

What I don't currently expect to see is a betting framework and space of strategies where

• We can calculate a growth rate for each strategy;
• Rank-optimality gives us a total order on strategies;
• There are strategies \( λ, μ \) with \( G(λ) > G(μ) \) but \( μ \) is more rank-optimal than \( λ \).

I wouldn't be totally shocked by that happening, math has been known to throw me some curveballs even in the days when I could call myself a mathematician. But it would surprise me a bit.

Given this definition, it's clear that a rank-optimal strategy maximizes every percentile of return. E.g. suppose \( λ \) is more rank-optimal than \( μ \), but the median of \( V_n(μ) \) is higher than the median of \( V_n(λ) \). Then we'd have \( P(V_n(μ) > V_n(λ)) ≥ {1 \over 4} \); so this can't hold in the limit.

It's also clear that rank-optimizing for money is the same as rank-optimizing for log-money, or for any monotonically increasing function of money. (Possible caveats around non-strict monotonic functions and the long-run equivalence thing from above?)

In some situations a rank-optimal strategy might not maximize modal return. I'm not sure if it will always minimize "expected time to reach some payoff much larger than \( V_0 \)".

Appendix Ⅲ: rank-optimization as utility function

A utility function is a function from "states of the world" to real numbers, which represent "how much we value that particular state of the world", that satisfies certain conditions.

When we say our utility function is linear or logarithmic in money, we mean that the only part of the world we care to look at is how much money we have. We maximize our utility in expectation, by maximizing-in-expectation the amount of money or log-money we have.

Suppose I say "my utility function is such that I maximize it in expectation by rank-optimizing my returns". What would that mean?

I guess it would mean that the part of the world state we're looking at isn't my money. It's my strategy for making money, along with all the other possible strategies I could have used and the betting framework I'm in. That's weird.

It also means I'm not expecting my utility function to change in future. Like, with money, I have a certain amount of money now, and I can calculate the utility of it; and I have a random variable for how much money I'll have in future, and I can calculate the utility of those amounts as another random variable. With rank-optimality, I'm not expecting my strategy to be more or less rank-optimal in future. That's convenient because to maximize expected utility I just have to maximize current utility, but it's also weird.

For that matter, I haven't given a way to quantify rank-optimization. We can say one strategy is "more rank-optimal" than another but not "twice as rank optimal". So maybe I mean my utility function has a \( 1 \) if I'm entirely rank-optimal and a \( 0 \) if I'm not? But that's weird too. If we can calculate growth rate then we can quantify it like that, I guess.

So in general I don't expect rank-optimizing your returns to maximize your expected utility, for any utility function you're likely to have; or even any utility function you're likely to pretend to have. Not unless it happens to be the case that the way to rank-optimize your returns is also a way to maximize some more normal utility function like "expected log-money", for reasons that may have nothing to do with rank-optimization.

Thanks to Justis Mills for comments; and to various members of the LW Europe telegram channel, especially Verglasz, for helping me understand this.

Blog meta: disabling comments

I don't know how many people read this blog¹, but it doesn't receive many comments. This is the 54th post on it, and only 16 of those have comments, 31 in total. I'm not currently signed in to disqus and signing in is enough friction that there are some comments from years ago that I meant to reply to, and a spam comment that I intended to delete.

I also noticed at some point that disqus inserts ads. I have an ad blocker on my laptop browser so I didn't notice until I looked on my phone. (This is also how I realized I didn't install an ad blocker on my phone browser.) But I don't want ads on this blog, especially not if I'm not getting that money myself.

So I'm disabling comments. I don't actually want to lose the comments that have been written (except the spam one), so for now I'm just going to disable disqus everywhere except the posts that already have comments (and except except for the post that only has one comment which is spam). I'll leave it on this post too, just in case. In future maybe I'll recreate the comment threads in the post bodies or something.

In case you do want to comment on something, somewhere I'll see it: at the bottom of many posts I have links to external comment threads. Almost everything I write these days gets crossposted to LessWrong,² even if it's not very relevant to that site, so there's generally a comment thread there you can use. I also often link to things on reddit. However, I only add these links if there are actually any comments on them, and it might take me a few days. Against boots theory was the first post I did this with, I've likely missed some links on posts from before then.

If there are no such links,

• You could try to find places I've crossposted. My LW and reddit profiles might help, especially if it's recent. (LW crossposting is automatic, it should happen within an hour of me publishing but not always.)

• You can use admonymous (which I can't reply to) or email (which history says I likely won't reply to).

Also, unrelated blog meta: thanks to DanielH who (1) suggested I set up https support; (2) sent me a link that told me how to do it; (3) after I did it, pointed out that math wasn't working.

A list of Petrov buttons

A thing people sometimes do in relation to Petrov day, is to have a button. And if anyone presses the button, something negative happens.

I'd like to know how often the button gets pressed, so I compiled a list of all the times people have done this, that I'm aware of, and the outcome. Also, for the events I was at in person, I'm not sure there's an existing write-up of what happened, so I'm adding details.

Here's the list. As I learn about other events, past or future, I'll try to keep it updated.

2018, Oxford/Seattle: Failure

Parties happened simultaneously in these two places. I was at the Oxford one, which was actually camping in a field outside Oxford. We both had laptops connected to a web app.

What we were told: anyone at either party could press a button to launch a nuke at the other party. It would spend 45 minutes in transit, and then the other party would have to destroy a cake. But they'd be warned of launch, and be able to launch one of their own in the intervening period. I don't remember the time limit, probably like two or three hours.

What we weren't told: there was some chance of false alarms.

What happened: the Oxford party got notified of an incoming launch. We opened a Facebook chat with Seattle like "wtf guys". They convinced us that no one at their party had pressed the button. I don't remember if we'd considered the possibility of false alarms before messaging them, but given the context, they're not exactly something to dismiss out-of-hand.

We made it through right up to the time limit with no one pressing either button. My understanding of what happened next is that the time on the Seattle laptop ticked down to zero. Someone there pressed the button, which they now believed to be disabled, in celebration. The time on the laptop and the time on the server were slightly out of sync. The button was not disabled. A nuke got launched.

At any rate, Seattle convinced us that something along those lines had happened. We did not launch a retaliatory nuke. (This was still an option because we had one inbound.) We sent them a video putting our cake in the fire.

2019, LessWrong (+ Raemon's house): Success

125 people were sent codes that could be used to take down the front page of LessWrong for 24 hours.

Additionally, Raemon held a party at his house, where he gave everyone his codes and the party would end if LW got taken down.

No one submitted valid codes, though some people put in fake ones.

2020, LessWrong: Failure

This was a similar setup to last year, but 270 people were given codes (including myself). Additionally, there was enemy action. Someone registered an account named "petrov_day_admin_account" and sent a message to a handful of users:

You are part of a smaller group of 30 users who has been selected for the second part of this experiment. In order for the website not to go down, at least 5 of these selected users must enter their codes within 30 minutes of receiving this message, and at least 20 of these users must enter their codes within 6 hours of receiving the message. To keep the site up, please enter your codes as soon as possible. You will be asked to complete a short survey afterwards.

User Chris_Leong fell for it, and the site went down.

Postmortem from a LW admin. Postmortem from Chris, with comments from the attacker.

2021, Austin/Ottawa: Success

I don't think much info is publicly available, but it sounds like a similar setup to Oxford/Seattle, perhaps without false alarms. No one pressed the button.

2021, LessWrong/EA Forum: Success

This year, 100 members of LessWrong were given codes that could be used to take down the EA Forum, and vice versa. (This year they'd go down for the rest of the day, not for 24 full hours.)

No one did: LW retrospective, EAF retrospective.

2022, London: Success

This was at the ACX Schelling meetup, and most participants didn't know it was happening. I bought a button that I carried around with me, and a cake with Stanislav Petrov's face on it that I hid in my bag. (I didn't want it sitting out in case someone ate it without realizing, or decided it would be funny to mess with the game.)

Initially the button was locked. We had lightning talks planned, and I gave the first one. I opened with something along the lines of: "I have a cake with me. I think it looks very nice, and I'm looking forward to sharing it with you all. However! I also have a button. If anyone presses the button, I will destroy the cake. This isn't a joke, this isn't a trick, I'm not going to reward you, I'm just going to destroy the cake. puts mic down, unlocks button, picks mic back up The button is now live. You may be wondering why I'm doing this…"

Not long after, someone came up to give a lightning talk on the subject of "why you should press the button". What I remember mostly boiled down to "obesity crisis, cake bad", and I didn't find it convincing. But as a speech, it was much better than mine, even though I'd had days to prepare and he'd only had like fifteen minutes. I was not surprised when I learned he had competitive debate in his history.

Someone asked why he hadn't pressed the button himself, which was right in front of him. (I was carrying the button around with me, since it wasn't connected to anything and I wouldn't know if someone had pressed it otherwise. And I was sitting in front of him.) He said he believed in democracy and wanted a vote on the subject. Following this someone put a poll on a nearby whiteboard. Assuming no one cheated, it got eight no-press votes to two yes-press votes, at a party with over 100 attendees. (But probably no more than about 50 saw either of these talks.)

Ultimately, no one pressed the button. One person stroked it, and one person held his hand above it, but both said they weren't going to press it.

After 2 1/2 hours, I locked the button back up and brought out the cake. Once we'd started eating it, some people wanted to press the button so I unlocked it again. I didn't tell anyone this until afterwards, but it's a very satisfying button to press.

It seems a handful of people didn't think there was actually a cake, which I'm a little sad about. (It's unclear how many people, partly because "the cake is a lie" is a meme. But one person explicitly told me they thought that.) That's not the kind of lie I'd tell. Not that most people had any reason to think that about me, but I'm still a little sad about it.

2022, LessWrong: Failure

The plan was that this year, pretty much any existing LW user with non-negative karma would be able to press the button. (Which again would only take the site down for the rest of the day.) But only high-karma users would be able to do so all day; every hour, the karma threshold needed to press it would lower. Also, unlike previous years, someone who pressed the button would remain anonymous.

What actually happened was that users with exactly 0 karma were also able to press the button from the start of the event. One of them did, two hours fifty-five minutes after it began (but only 1:50 after the password needed to press the button was actually published). Habryka figured that out without learning their identity, and the site got reset for a do-over.

Then the whole site went down briefly when someone tried to fix a different Petrov-day-related bug.

And then finally it went down after 21:33. Presumably this time the button was pressed by someone who was supposed to be able to. There were 1,504 such users.

I had bet on Manifold that the site would go down before eight hours were up. (Specifically I bet the market down from 34% to 30%.) Despite the bug I consider my prediction to have been wrong. I'm happy about that.

2022, St Louis: Success

I don't know much detail and don't know if it's been written about publicly. But what I'm told is that if anyone had pressed the button, the party would have been over and everyone would have had to go home. No one pressed the button.

2022, Champaign-Urbana: Success

LW user benjamincosman says:

I ran a very small Petrov Day gathering this year in Champaign-Urbana. (We had 5-6 people I think?) I put a Staples "Easy" button on the table and said if anyone presses it then the event ends immediately and we all leave without talking. (Or that I would do so anyway, obviously I couldn't make anyone else.) No one pressed the button.

2023, Waterloo: Success

Joint event between the Waterloo, Ontario rationalist and EA groups. Jenn, the organizer of the rationalist group, writes:

• this was a meetup we had in conjunction with the Waterloo EA group on September 27th, 2023.
• we had 2 buttons, one for a fruit platter and 1 for a cake. We divided people into 2 halves of the room based on if they wanted fruit or cake, and then explained the typical rules. (this was partly inspired by your write-up of the uk ceremony one year when someone gave an impassioned speech about why cake was bad)
• the buttons stayed unpressed for the duration of the x-risk and petrov day presentation given by EA Waterloo
• afterwards we dug in.
• people did say that we probably should have given more opportunity for button presses after the presentation during more free social time, we'll do that if we run this ritual in future years

Things I'm not counting

Here are some things that I don't think belong on the above list, but I do kind of expect someone to point me at them if I don't mention them.

• 1983, the actual Petrov incident - IDK, apart from anything else it feels like it would be kinda tactless to include on this list?

• 2021, Jeffrey Ladish did a thing on Facebook, but the dynamics seem importantly different.

PD-alikes in two dimensions

Some time after writing Classifying games like the prisoner's dilemma, I read a paper (I forget which) which pointed out that these games can be specified with just two numbers.

Recall that they have the following payoff matrix:

where $W > Z$.¹ We can apply a positive affine transformation (that is, $n ↦ an + b$ where $a > 0$) to all of $W, X, Y, Z$ without changing the game. So let's pick the function $n ↦ {n - Z \over W - Z}$. This sends $W$ to $1$ and $Z$ to $0$, leaving us with just two parameters: $R = {X - Z \over W - Z}$ and $S = {Y - Z \over W - Z}$.

So what happens if we plot the space of these games on a graph? The lines $ \{X, Y\} = \{W, Z\} $ become $ \{R, S\} = \{0, 1\} $, i.e. vertical and horizontal lines. The lines $X + Y = 2W$ and $X + Y = 2Z$ become the diagonals $R + S = 2$ and $R + S = 0$; and $X = Y$ becomes the diagonal $R = S$. Drawing those lines, and relabelling in terms of $W, X, Y, Z$, it looks like this:

Note that Cake Eating (my favorite game) is the only one with a finite boundary; the other boxes extend to infinity. There are also finite components in the Farmer's Dilemma (with $X + Y < 2W$), and Stag Hunt and Studying For a Test (with $X + Y > 2Z$). As drawn, Prisoner's Dilemma occupies almost all of the box it shares with Too Many Cooks; but Too Many Cooks (above the line $X + Y = 2W$) is also infinite. (I initially got those the wrong way around, so the drawing isn't very clear there.)

I don't know if we learn much from this, but here it is.

Variadic functions in Hindley Milner

I previously mentioned an interest in adding variadic functions to Haskenthetical, the "Haskell with a Lisp syntax" I'm vaguely working on.

It sounds difficult. This is an attempt to figure out just how difficult, partly by looking at the state of the art.

A variadic function is one which can accept argument lists of different length. A well-known example in many languages is printf. Here are a bunch of functions I might want to be variadic in Haskenthetical, along with examples in Haskell of types I might want to use them as:

• +, accepting any number ≥ 2 of numbers and returning a number of the same type.

    (+) :: Num a => a -> a -> a
    (+) :: Num a => a -> a -> a -> a
  

• Similarly, -, * and /. - and maybe / could reasonably accept ≥ 1 input, but that would be inconsistent and disallow (- 3) as a partial application. - and / would be left associative, i.e. (- 5 1 2) would be (- (- 5 1) 2).

• list, accepting any number of arguments of the same type and returning a List of that type. (An empty (list) would have type List $a, the same as Nil.) There's a related function that I'll call list' for now, where the final argument has to be a List and the other arguments are prepended to it.

    list :: [a]
    list :: a -> [a]
    list :: a -> a -> [a]
  
    list' :: [a] -> [a]
    list' :: a -> [a] -> [a]
    list' :: a -> a -> [a] -> [a]
  

• I already mentioned printf. Haskell's implementation is in the Text.Printf module:

    printf :: String -> IO ()
    printf :: PrintfArg a => String -> a -> IO ()
    printf :: (PrintfArg a, PrintfArg b) => String -> a -> b -> IO ()
    -- Also: all of the above, with `-> String` instead of `-> IO ()`. But
    -- that's not super relevant here.
  

• » and «, which take a function and then ≥ 2 arguments to interleave it with. Currently implemented as macros, and maybe I'd want them to be macros even if they could be functions, but that's not the point right now. Often the function will have a type like -> $a (-> $b $a) or -> $b (-> $a $a), and then all or all-but-one of the arguments will be of the same type - this will be equivalent to using haskell's foldr or foldl. But that's not required. You could have (» , 1 "foo" 3 "bar") which would return (, 1 (, "foo" (, 3 "bar"))), the return type changing for every new argument you add.

    -- e.g. in (») (+) 1 2 3
    (») :: Num a => (a -> a -> a) -> a -> a -> a -> a
  
    -- e.g. in (») (:) 1 2 []
    (») :: (a -> [a] -> [a]) -> a -> a -> [a] -> [a]
  
    -- e.g. in (») (,) 1 "foo" 3 "bar"
    (») :: (forall a b . a -> b -> (a, b))
        -> a -> b -> c -> d -> (a, (b, (c, d)))
  

• sort might have one or two optional arguments - one for a comparison function and/or one for a key function. (A key function is unnecessary if you have a comparison, but it might be convenient.)

  Actually, we can't distinguish a comparison function from a key function based on the types, here. There's nothing stopping someone from writing an Ord instance for a -> Ordering, and then sort compare [...] might refer to either of the two-argument forms. I'm just going to ignore that problem though.

    sort :: Ord a => [a] -> [a]
    sort :: (a -> a -> Ordering) -> [a] -> [a]
    sort :: Ord b => (a -> b) -> [a] -> [a]
    sort :: (a -> b) -> (b -> b -> Ordering) -> [a] -> [a]
  

• Similarly, lots of functions could take an optional config argument.

    renderCsv :: [[String]] -> String
    renderCsv :: RenderCsvOpts -> [[String]] -> String
  

• map could take arbitrarily many lists. Haskell has separate functions for these, named map, zipWith and zipWith3.

    map :: (a -> b) -> [a] -> [b]
    map :: (a -> b -> c) -> [a] -> [b] -> [c]
    map :: (a -> b -> c -> d) -> [a] -> [b] -> [c] -> [d]
  

• So could zip (zip3, zip4). Note that each of these is a partial application of a variadic map, but that doesn't mean variadic map gives us variadic zip for free. To define zip from map we'd presumably need a variadic tuple constructor, plus we'd need type checking to be able to combine them.

    zip :: [a] -> [b] -> [(a, b)]
    zip :: [a] -> [b] -> [c] -> [(a, b, c)]
    zip :: [a] -> [b] -> [c] -> [d] -> [(a, b, c, d)]
  

• If we have zip, what about unzip? This isn't variadic, but it's plausible that a system allowing variadic functions would also allow it. (I think it's a harder problem, though. It seems to me that if you can get polymorphism over tuple length you essentially get polymorphism over function arity, perhaps with caveats around partial application. But I don't know if the reverse holds.)

    unzip :: [(a, b)] -> ([a], [b])
    unzip :: [(a, b, c)] -> ([a], [b], [c])
    unzip :: [(a, b, c, d)] -> ([a], [b], [c], [d])
  

• Haskellers often find ourselves writing f <$> a <*> b <*> c. (map is a specialization of this, using the Applicative instance from ZipList.) Haskell calls these functions fmap (restricted to Applicative as liftA), liftA2 and liftA3.

    -- Realistically I expect this first would need Applicative
    appF :: Functor f => (a -> b) -> f a -> f b
    appF :: Applicative f => (a -> b -> c) -> f a -> f b -> f c
    appF :: Applicative f => (a -> b -> c -> d) -> f a -> f b -> f c -> f d
  

• Or we might want one for f <*> a <*> b <*> c. (appF is in turn a specialization of this, applying pure to its first argument.)

    appA :: Applicative f => f (a -> b) -> f a -> f b
    appA :: Applicative f => f (a -> b -> c) -> f a -> f b -> f c
    appA :: Applicative f => f (a -> b -> c -> d) -> f a -> f b -> f c -> f d
  

• I can even imagine wanting to lift only the final result, replacing pure $ f a b c with puring f a b c. That seems kind of useless by itself, but the general pattern might not be.

    puring :: Applicative f => a -> f a
    puring :: Applicative f => (a -> b) -> a -> f b
    puring :: Applicative f => (a -> b -> c) -> a -> b -> f c
  

• Haskell offers an impure function trace which is useful in debugging. I could imagine wanting something similar, wrapping around a function to log every time it's called. The variadic part of the type here is equivalent to puring specialized to Identity, but not having to worry about typeclasses might make things simpler.

    -- The first argument would be the function name or other identifier.
    trace :: String -> (a -> b) -> a -> b
    trace :: String -> (a -> b -> c) -> a -> b -> c
    trace :: String -> (a -> b -> c -> d) -> a -> b -> c -> d
  

(I'm using Haskell syntax for the types, rather than Haskenthetical syntax. Partly because Haskenthetical has no concept of typeclasses or rank-2 types, which some of the examples use. But also partly because it's probably easier to read. Which you might think would be a reason not to make Haskenthetical, but here we are.)

I'm not, to be clear, saying that all or even any of these are good ideas. I mostly don't miss variadic functions in Haskell; they can be implemented hackily like in Text.Printf linked above, but I'm not sure I've ever felt the need to, and I've rarely-if-ever used that printf. But it seems worth starting to explore the space of the sorts of things I might want to consider trying to support, before making any decisions.

Not Hindley-Milner

The first paper I'm going to look at is the most recent one I've found: Strickland, Tobin-Hochstadt and Felleisen, Practical Variable-Arity Polymorphism (hereafter PVAP; 2009, doi: 10.1007/978-3-642-00590-0_3). I linked this in my previous post. It implements typechecking for variadic functions in Typed Scheme, I think specifically meaning Typed Racket? I'm not familiar with the language (I have done a little untyped Racket in the past), but from the sounds of things, its type system is fundamentally different from Hindley-Milner, and the implementation won't easily transfer. (Both compile to System F, but I don't think that helps.)

But it does help me make sense of the space. It divides the functions it can type into two: uniform and non-uniform. Let's call the optional arguments the "rest parameter", as in the parameter which holds "the rest of the parameters". Uniform functions are those whose rest parameter is a homogeneous list, such that they could be replaced (at cost to ergonomics) with a function accepting a list. In my above examples, that's the arithmetic functions plus list and list'. In Typed Racket syntax, the types of these functions would be

(: + (-> Number Number Number * Number)) # also -, *, /
(: list (All (a) (-> a * (Listof a))))
(: list' (All (a) (-> a * (Listof a) (Listof a))))

With the * indicating "zero or more of the preceding type". These seem simple enough. (Though list' takes an argument after the variadic part, which makes things more complicated. Racket calls that function list* but I couldn't find a type declaration for it to be sure it's actually valid.)

Then the other other functions handled by the paper are "non-uniform". Of my examples, I think that's just map, trace, and maybe zip and unzip natively.

(: map (All (a b ...) (-> (-> b ... b a) (Listof b) ... b (Listof a))))
(: trace (All (a b ...)) (-> String (-> b ... b a) b ... b a))
(: zip (All (a ...)) (-> (Listof a) ... a (Listof (Tuple a ... a))))
(: unzip (All (a ...)) (-> (Listof (Tuple a ... a)) (Tuple (Listof a) ... a)))

For zip and unzip, I'm inventing the type Tuple here to refer to a collection with known size and types. (Tuple Number String Bool) would be equivalent to Haskell's (Number, String, Bool). I don't know if anything like it actually exists, or can exist, in Typed Racket already.

These are a bit more involved. The ... is doing two related but syntactically different things.¹ In the variable list of All (the (a b ...) and (a ...)) it says the preceding variable corresponds to a list of types. In the body of All, it combines with both the preceding and following syntax elements. t ... b means: "b was followed by ... in the type signature, so it corresponds to a list of types. Use a copy of t for each one of those types, and inside each copy, substitute b with the corresponding type from the list".

So if b ... corresponds to Integer String Number, then (Listof b) ... b corresponds to (Listof Integer) (Listof String) (Listof Number).

I don't know if we strictly need the trailing variable in the body. You're only allowed one ... in the variable list (right at the end), and the trailing variable is required to have been in the variable list followed by a ..., so as far as I can tell it's unambiguous. (At least as long as there are no higher-rank types, which I don't think I've seen mentioned in this context.)

printf, appF, appA and puring would also fit into this schema if it weren't for the constraints. But as far as I know Typed Racket has nothing analagous to Haskell's constraints. I don't know how much they complicate matters.

That leaves four examples. sort and renderCsv don't fit the scheme because they can only accept one or two optional arguments, not an arbitrary number. (Typed Racket does support optional arguments, they're just not covered by this paper.)

» and « don't fit because the type of each argument can depend on the type of the preceding one. For example, we might call

(«) (&) 1 (+ 1) Just (fmap (* 3)) (: [])
(») ($) (: []) (fmap (* 3)) Just (+ 1) 1

There's a pattern to the types, but not the pattern we need.

So: this paper describes a way of typechecking a few of the functions we might like to typecheck, in a completely different type system than the one we want to use. What can we do in Hindley-Milner?

There's a brief discussion of that, mostly of the form "here's another paper that made some progress in an HM system. It's not as powerful as what we have here". But those other papers fully exhaust what I've managed to find on the subject; and actually, I can't find a copy of Moggi, Arity polymorphism and dependent types (2000). That leaves three to look at.

Infinitary Tuples

First up: Dzeng and Haynes, Type reconstruction for variable-arity procedures (1994, doi: 10.1145/182590.182484). This uses a thing called "infinitary tuples" to handle optional arguments and both uniform and non-uniform variadic functions. PVAP lists some limitations:

Most importantly, since their system does not support first-class polymorphic functions, they are unable to type many of the definitions of variable-arity functions, such as map or fold. Additionally, their system requires full type inference to avoid exposing users to the underlying details of row types, …

I don't understand this first one: map is explicitly given as an example, with type²

But this might help illustrate the second problem, which I think is saying: type annotations are complicated, users won't want to deal with them.

I do notice two limitations myself. One is that you're not allowed to pass in a variadic function and apply it with two different argument counts: ((λ (f) (* (f 1) (f 2 3))) -) is forbidden, even if (- 1) and (- 2 3) are both okay. Combining this system with higher-rank types might avoid this limitation, but I don't know if they're compatible. The authors list two other ways to potentially avoid it, but both would need more research. I don't know how big a deal this would be, but it feels minor. It's similar to (perhaps the same as) the restriction that lambda bindings are monomorphic, and I don't know if I've ever had a problem with that in practice.

The other is: there's only one thing you can do with a "rest argument" (i.e. the collection of the arbitrarily-many arguments at the end), and that's to use it as another rest argument. There's even special syntax for that: you'd define a variadic function as (λ (x &rest xs) ...) (i.e. taking one required argument and arbitrarily many following that), and inside the body you'd call some other variadic function with (g a b c &rest xs). So variadic functions need to be either built-in or defined in terms of other variadic functions.

This feels like a bigger limitation, but again I don't really know. The paper only talks about type-checking - it contains zero implementations of variadic functions. Maybe a small handful of built-in ones would let us define everything we want.

Ignoring that problem for now, and considering whether we can type the example functions from above:

• +, list and map are given as examples. The other arithmetic functions would be fine, and I'm fairly confident trace would be too.

• I think list' is not allowed, since it has an extra argument after the variadic part.

• I dunno about printf. I'd need to figure out how typeclasses fit in.

• I presented both sort and renderCsv with the optional arguments before the required ones. I think that rules them out.³ renderCsv would be fine if we swap the arguments, but sort also has typeclass stuff going on. Even ignoring that, I'm not sure if we can make the "explicit comparison function with optional key function" as polymorphic as we'd like. That is, we could write a function that can be accepted at types

    sort :: [a] -> (a -> a -> Ordering) -> [a]
    sort :: [a] -> (a -> a -> Ordering) -> (a -> a) -> [a]
  

  with the third argument, the key function, defaulting to the identity. But I'm not sure if we could also accept

    sort :: [a] -> (b -> b -> Ordering) -> (a -> b) -> [a]
  

  with the third argument taking a default type as well as a default value.

• appF, appA and puring seem likely, the typeclass stuff is simpler than either printf or sort.

• zip and unzip might be feasible. We'd some way to interpret a row (see below) as a proper type. I don't know how difficult that would be.

• I think « and » are right out.

So how does it work? It's based on extensible records, which I haven't yet looked at in depth. (Extensible records with scoped labels is the paper I was planning to look into if I tried to add them to Haskenthetical, but it might not take a compatible approach.) A row is an infinite sequence of (mark, type) pairs, where a mark is either "present", "absent" or a mark variable. Rows can be equivalently constructed as pairs of (mark row, type row), where those are infinite sequences of the respective things. At some point all the marks become "absent", and then the types don't matter. Recall the type of map from above,

This is quantified over three type variables. \(γ\) is quantified over mark rows, and \(δ\) over type rows, with \(γ∘δ\) combining them into a row. And \(α\) is quantified over proper types.

Everything before the outermost \(→\) is a row. (Functions in this system are of kind "row → type", not kind "type → type".) It has \(\mathit{pre} · ((γ∘δ) → α)\) as its head, a single field marked present ("pre") with type \((γ∘δ) → α \). \(γ∘δ\) is itself a row. Then for the outer row, the tail \(γ ∘ (\underline{\mathtt{list}}\ δ)\) has the same sequence of marks as the argument to its head, meaning the same number of arguments. \(\underline{\mathtt{list}}\ δ\) is a type row of "apply list to the types in \(δ\)".

Then we might instantiate \(γ\) at \(\mathit{pre} :: \mathit{pre} :: \underline{\mathit{abs}}\), i.e. "two present fields and then nothing". And we might instantiate \(δ\) at \(\mathtt{int} :: \mathtt{string} :: δ'\), i.e. "an int, then a string, then another type row", and \(α\) at \(\mathtt{bool}\). Then we'd have something like the Haskell type

(Int -> String -> Bool) -> [Int] -> [String] -> [Bool]

but it would be rendered in this type system as

Which, let us say, I do not love.⁴ And this kind of thing infects all functions, not just variadic ones! And I don't know how it interacts with partial application. Still, maybe it can be made ergonomic. I think this paper deserves further study, though I'm unlikely to do it myself.

The Zip Calculus

Next is Tullsen, The Zip Calculus (2000, doi: 10.1007/10722010_3). This extends typed lambda calculus to get variadic tuples.

This paper seems to come out of the "program transformation community", which I'm not familiar with. I suspect it's talking about things I know a little about in ways I don't recognize.

The specific typed lambda calculus it extends is "\(\mathrm{F}_ω\)". Fortunately I recently came across the lambda cube so I know how to find out what that means. It's System F, which (as above) is what GHC compiles to as an intermediate step, plus the ability for the user to define their own types, which… I'd always assumed GHC's intermediate System F also had that, but maybe not, or maybe GHC actually compiles to \(\mathrm{F}_ω\)? But… later in the paper it talks about dependent typing, which \(\mathrm{F}_ω\) doesn't have, so maybe the paper is using nonstandard terminology? Argh.

Anyway, I think of System F as a more powerful version of Hindley-Milner, but too powerful to automatically type check, so you need lots of type annotations. If we're making it more powerful I guess we're still going to need those.

I confess I have trouble figuring out what this paper offers in any detail. I think I'd be able to eventually, but I'd need to put in more effort than I felt like right now.

It does give us a short list of functions that, if the language defines them built-in, we can build other functions out of. These are list and what it calls seqTupleL and seqTupleR. These aren't described except for their (identical) type signatures

which I think in Haskell correspond to the types

seqTuple_ :: Monad m => (a1 -> m b1) -> a1 -> m b1
seqTuple_ :: Monad m => (a1 -> m b1, a2 -> m b2) -> (a1, a2) -> m (b1, b2)
seqTuple_
  :: Monad m
  => (a1 -> m b1, a2 -> m b2, a3 -> m b3)
  -> (a1, a2, a3)
  -> m (b1, b2, b3)

If I'm right about that, I'm pretty sure the semantics are "zip the tuple of functions with the tuple of parameters, apply them each in turn and sequence the effects (left-to-right / right-to-left)".

Given these functions, we're specifically told we can implement zip, unzip, map⁵ and appF⁶. I'm pretty sure arithmetic, list', appA, puring and trace will be possible, and I weakly guess that printf will be as well, while « and » won't be. I'm not sure about sort or renderCsv.

One thing is that all of these functions are defined with variadic tuples, so that e.g. map would actually be accepted at types like

map :: (a -> b) -> [a] -> [b]
map :: ((a, b) -> c) -> ([a], [b]) -> [c]
map :: ((a, b, c) -> d) -> ([a], [b], [c]) -> [d]

which I assume leaves no room for partial application. It might also be awkward when the final argument needs to be handled separately; I'm not sure if we could get

list' :: [a] -> [a]
list' :: (a, [a]) -> [a]
list' :: (a, a, [a]) -> [a]

or if we'd be stuck with

list' :: () -> [a] -> [a]
list' :: a -> [a] -> [a]
list' :: (a, a) -> [a] -> [a]

Given this limitation, it would be convenient to have variadic curry and uncurry functions,

curry :: (a -> b -> c) -> (a, b) -> c
curry :: (a -> b -> c -> d) -> (a, b, c) -> d
curry :: (a -> b -> c -> d -> e) -> (a, b, c, d) -> e

uncurry :: ((a, b) -> c) -> a -> b -> c
uncurry :: ((a, b, c) -> d) -> a -> b -> c -> d
uncurry :: ((a, b, c, d) -> e) -> a -> b -> c -> d -> e

but no such luck. We're specifically told curry can't be typed, that's an area for future research. And uncurry would break us out of the variadic-tuple paradigm entirely.

I'd be really interested to see a more approachable-to-me version of this paper.

Typeclasses

Finally McBride, Faking It: Simulating Dependent Types in Haskell (2002, doi: 10.1017/S0956796802004355).

I'm happy to say I understand (the relevant parts of) this paper. It adds variadic map directly to Haskell using typeclasses. It then explores further in directions I'm going to ignore. The downside, as PVAP correctly notes, is that you need to explicitly pass in the number of variadic arguments as an initial argument.

The code is simple enough. It felt a bit dated to me: in the past twenty years GHC has added a bunch of new features that seemed like they'd help make it more ergonomic. But I couldn't find a much better way to write it using those, so whatever.

The original is (equivalent to):

{-# LANGUAGE FunctionalDependencies, FlexibleInstances, UndecidableInstances #-}

data Zero = Zero
data Suc n = Suc n

one = Suc Zero :: Suc Zero
two = Suc one :: Suc (Suc Zero)

class ManyApp n fst rest | n fst -> rest where
  manyApp :: n -> [fst] -> rest

instance ManyApp Zero t [t] where
  manyApp Zero fs = fs

instance ManyApp n fst rest => ManyApp (Suc n) (t -> fst) ([t] -> rest) where
  manyApp (Suc n) fs xs = manyApp n (zipWith ($) fs xs)

nZipWith :: ManyApp n fst rest => n -> fst -> rest
nZipWith n f = manyApp n (repeat f)

Here manyApp is the version where instead of just one function, we provide a list of them. The instances give us

manyApp Zero :: [t]           -> [t]
manyApp one  :: [s -> t]      -> [s] -> [t]
manyApp two  :: [r -> s -> t] -> [r] -> [s] -> [t]
--      ^ n     ^ fst            ^ rest

We define it recursively, in terms of its simpler definitions. Then nZipWith is easy to define in terms of manyApp, where it's not easy to define recursively in terms of itself.

So what else can we already implement in Haskell with typeclasses? I think most of my original list of functions, and for several of them I think explicitly specifying the variadic count would be unnecessary.

I tried out a few of them, see here. It's possible there are ways to improve on what I found, but my current sense is:

• Arithmetic, list, printf: work mostly fine with no variadic count, fine with it.
• list': works poorly with no variadic count, fine with it.
• zip, unzip: need nested tuples, ((a, b), c) rather than (a, b, c). Given that, both work fine with variadic count. zip works poorly without, unzip doesn't for reasons related to the next section.
• renderCsv: works (mostly?) fine with no variadic count.
• sort: works okay with one optional argument and no variadic count. With two optional arguments, works fine with something specifying which form you want. (If they're independently optional, there are two two-argument forms, so it wouldn't necessarily be a "variadic count".) But at that point, really, just have different functions.
• map, appF, appA, puring, trace: work fine with variadic count.
• », «: do not work.

Here, I consider something to "work" if you can make it do what you want, and work "better" if it doesn't need much type hinting. "works fine" means you don't need type hinting in any places I wouldn't normally expect it in Haskell, and I think the error messages might even be pretty normal?

So I think that with explicit counts, most of these work fine. Without the explicit counts, several of them work pretty okay, but they'll have difficulty if you use them in a weakly-constrained context. Like, print (list "a" "b" "c") won't work because it doesn't know if you want the Show instance on [String] or on String -> [String] or…. (Probably only one of these instances exists, but we can't rely on that.) But then you just need to add a type annotation, print (list "a" "b" "c" :: [String]). list' and zip need a lot more type annotations than is normal, for reasons I maybe have a shallow understanding of but not a deep one.

Partial application

Here's something I think I was vaguely aware of before I started this, but it's clearer now: partial application makes this harder.

Consider the Haskell code

zipWith (\a b c -> a + b + c) [1,2] [3,4]

That uses the existing 2-ary zipWith function, which means it has type [Int -> Int]⁷. (Its value is [(+ 4), (+ 6)].) We could instead have used the existing 3-ary zipWith3, and then it would have type [Int] -> [Int]. If we used a variadic function, what type would it have? All the papers I looked at had some way of answering the question.

In Racket, as far as I know there's no implicit partial application. A function that takes two arguments is different from a function that takes one argument and returns a function that takes one argument. So under PVAP, to choose between the two interpretations, you'd need something like:

(map (λ (a b) (λ (c) (+ a b c))) '(1 2) '(3 4)) ; [Int -> Int]
(λ (cs) (map (λ (a b c) (+ a b c)) '(1 2) '(3 4) cs)) ; [Int] -> [Int]

The Zip Calculus preserves partial application for fixed-arity functions, but not variadic ones. (More precisely, it has partial application for functions but not for variadic tuples.) The two interpretations would be written along the lines of

nZipWith (\(a, b) c -> a + b + c) ([1,2], [3,4]) -- [Int -> Int]
\cs -> nZipWith (\(a, b, c) -> a + b + c) ([1,2], [3,4], cs) -- [Int] -> [Int]

Which is basically the same as the Racket version, apart from syntax.

And "infinitary tuples" has to do something similar. For the list of functions, we'd pass "a function taking a row with two variables and returning a function" as the first argument. For the function on lists we'd pass "a function taking a row with three variables". I guess the system must lose Haskell's equivalence between \a b -> ... and \a -> \b -> ....

The typeclass solution to this is completely different. The inner function is the same for both interpretations, what changes is the initial "variadic count" argument.

nZipWith two (\a b c -> a + b + c) [1,2] [3,4] -- [Int -> Int]
nZipWith three (\a b c -> a + b + c) [1,2] [3,4] -- [Int] -> [Int]

This feels to me like the cleanest of the bunch. I introduced this initial argument as a downside before, but that might be kind of unfair - like, maybe it's only a downside compared to an imaginary world that I hadn't yet realized was impossible.

How else might we solve this problem?

One option is to assume that nZipWith will never want to return a list of functions. I don't love that solution, but here's someone implementing it for appF (which they call liftAn). I haven't checked how well it works.

Something else that might work is to wrap the return value of the initial function in some singleton type.

nZipWith (\a b -> Solo $ \c -> a + b + c) [1,2] [3,4] -- [Int -> Int]
nZipWith (\a b c -> Solo $ a + b + c) [1,2] [3,4] -- [Int] -> [Int]

This is kind of similar to "no partial applications", except you can still do partial applications everywhere else.

And I don't know if this would be possible, but I could imagine using a type annotation to distinguish. If you need to annotate every call to the function, I think I'd rather just specify the number of arguments or something. But if you only need them when it's ambiguous that could be neat.

This isn't a problem for every variadic function. In the previous section, only about half the functions absolutely needed a variadic count, and this is why. (For unzip, it's about the ambiguity in [((a, b), c)]. Do you want to unzip one level to ([(a, b)], [c]), or two to (([a], [b]), [c])?) Other functions had difficulty without the count, but I think that's for unrelated reasons. I think I have a decent intuitive sense of which functions will have this problem and which won't, but I couldn't give a brief description of it. Maybe something to do with matching structures between the initial argument and the rest of the type?

Conclusion

The main thing I take away from this is that I'm basically going to drop variadic functions from my radar for Haskenthetical. Infinitary tuples and the zip calculus don't feel like directions I want to go in. This might affect whether, when and how I add support for typeclasses and various surrounding machinary.

Walkthrough: Filing a UK self-assessment tax return

I'm writing this guide for a few reasons. For one, it might be useful to someone, possibly me in future. For two, if I make a mistake, someone might call me out on it and then I can correct it. And for three, it gives me a venue to complain about things I think aren't very good about the online submission form and also our tax system.

(Overall I'm pretty positive on the form! I'm gonna complain a lot, and I'm not going to point out all the places where it works and the instructions are clear and easy to follow. So this will probably come across quite negative, but that's because I'm counting down from perfection, not counting up from zero.)

Most people in the UK don't have to file one of these. I started filing them when I started earning in the 40% tax bracket. I'm not sure I still have to - I think I got a message a year or two back saying I could stop - but if I don't I'll pay more tax than I need to.

HMRC publishes a document called "how to fill in your tax return". (In previous years it had the reference SA150, and that's still in the filename, but it's no longer written under the title for some reason.) I actually haven't found it very helpful, it mostly doesn't seem to answer questions that the website leaves me with. But I guess it might be helpful in theory.

Update December 29: I only know about the parts of this process that are relevant to me. Some questions about other parts are answered in the comments on /r/UKPersonalFinance. Also, I generally find that community to be very helpful and knowledgable, so I find it reassuring that they only caught one mistake so far.

The gov.uk page on filing says you have to register first unless you sent a tax return last year. I've sent tax returns for several years running now, and I don't remember what registering was like.

Logging in from there I get sent to my "personal tax account" page and click the "complete your tax return" link in the top right box. I feel like that was harder to find in the past. It explains who can and can't file a return, I assume I read that in a previous year and could then and still can, and I click 'start now'.

The first page has a bunch of stuff filled in for me, mostly accurate.

"P J" is my first two initials, not my first name, but nice try. The all-caps HAZELDEN seems unnecessary, you don't need to worry about my handwriting.

These say 'optional', which sounds like I don't need to give them if I don't want to. But up above it says

'Optional' indicates only complete if relevant to you

So maybe that means "give us this if you have one"? I'm probably safe to leave out my phone number, I don't want them calling me and I don't know what "STD" means.¹ I'll grudgingly give them my email address I guess.

For years I've been using my parents' address for all official stuff, which is easier than telling banks and the like every time I move house. But some time this year I got a scary letter telling me I had to register for the electoral roll in my current place, which meant I got removed from it back in Bristol. So probably I have to give my current address here too.

(I don't see why it's any of the government's business where I live. Unfortunately they disagree, and they have all the power in this relationship.)

Well this is awkward. I moved in here in 2019, but I wouldn't have given this address when filling in my 2019-20 tax return. Will they get pissy with me if I give the date I moved in? Eh, probably not.

Almost every address form I've filled in for the past several years has let me give my postcode and then select my exact address from a short dropdown, or enter it manually. This address form does not offer that.

There's no dropdown box, even if I select "Abroad". Also, this is still supposed to be asking about my address - "where the income was earned" is a different question. (And not a very clearly worded one, what if I earned income in multiple countries?) Fortunately I have never lived or earned income outside the UK.

"The postcode must be in capitals." Well done for finding and documenting a bug, HMRC, but I feel like it shouldn't be too hard to fix this one.

Residency is "based on whether you lived in England and Northern Ireland, Scotland or Wales for the majority of the tax year". I assume they mean "plurality", but maybe if you spent less than half the year in any of these, you don't pay tax at all. (Tax collectors hate him! Nonlocal man discovers one weird trick…)

That's not what "e.g." means, but the date is correct.

This is 'optional', so maybe there are people who don't have a marital status, as distinct from having the marital status 'single'? Or maybe, contra what they said above, it means I don't have to tell them, and then probably they'll treat me as single if I don't specify?

The options are: Single, Married, In Civil Partnership, Divorced, Civil Partnership dissolved, Widow, Widower, Surviving Civil Partner, Separated. Obviously you can be more than one of these in any given tax year, or even more than one of them simultaneously, so the help text is not very helpful. I guess the idea is you give the most specific version of the most recent change as of the end of the tax year? I don't know who would ever pick "separated". Luckily for me, the government hasn't decided it cares about long-term cohabiting relationships until we actively make them legible, at least not for tax purposes. So I get to just say "single" and not worry about it.

This is the only part of the tax return (at least the pages I saw) where your gender becomes relevant, and then only for widows and widowers. I don't know what the nonbinary version is, but they get left out.

If they're assuming I didn't move house, you'd think they could assume I didn't go blind or get a blind spouse or civil partner.

I wonder what counts as an "other register". I think it's a crime to lie on my tax return, and quite possibly a crime to lie to my local authority about being blind. But it might not be a crime to lie on Honest Joe's Register Of Definitely Actually Blind People, and then to truthfully tell HMRC that I'm on that register. (This sounds like the sort of thing that you patiently explain to a judge and then the judge adds a little extra to your fine for being a smartarse. You try to appeal and the appeals court rejects with the comment "lolno". If you're going to try this, I think the funniest way would be to borrow a guide dog to take to court with you, maybe carry a cane slung over your shoulder, but make no actual pretense at blindness.)

There's nothing here telling you how to figure out if you're on plan 1 or plan 2, but I am indeed on plan 1. The Student Loan Company told them that (according to the help text), but apparently didn't tell them enough to know that I do not have and am not repaying a postgrad loan.

The next page helps figure out which other pages should be included in the rest of the form, and which ones can be skipped.

In my case, I was an employee (or director or office-holder) during the tax year, so I answer yes and tell them how many employments (or directorships) I had (one) and the name of the employer.

I wasn't self-employed or in a partnership, but if I was I'd need to tell them how many of those I had and their names, too. I didn't get income from UK land or property over £1,000 (or indeed at all), I didn't receive any foreign income, and I don't need to complete the capital gains section.

For capital gains it's not entirely obvious that the answer is no. Here's the advice:

You must report your capital gains and attach your computations if in the tax year either:

• you disposed of chargeable assets which were worth more than £49,200
• your chargeable gains, before the deduction of any losses, are more than £12,300
• you have gains in an earlier year taxable in this period
• you want to claim an allowable capital loss or make any other capital gains claim or election for the year

In working out if the assets you disposed of were worth more than £49,200 use the market value of any assets you gave away or sold for less than full value and ignore disposals of:

• exempt assets such as private cars, shares held within Individual Savings Accounts
• assets to your spouse or civil partner, if you were living together at some time during the tax year
• your own home where: [omitted a sub-list because I don't even own a home]

In working out your total chargeable gains include any gains attributed to you, for example, because you're a settlor or beneficiary of a trust, or in certain cases where you're a member of a non-resident company.

This is fairly readable, but it keeps using words like "chargeable" and "allowable". Those sound like they have important technical definitions. Are we given the important technical defintions? Ha, no.

So here's my current understanding. I have shares in some index funds in ISAs and pensions, but I'm pretty sure I should ignore those. And I own a handful of individual stocks and cryptocurrencies, not in tax-protected accounts, that I shouldn't ignore.

But so far all I've done is buy things and let their values go up or down. I haven't "disposed of" any of them: sold, traded or given away.² That means I don't need to worry about it. When I dispose of something, I'll need to work out how how much profit or loss I made on it, and also (as a separate question) how much it was worth at the time. If those amounts go over certain thresholds in total for the year (£12,300 and £49,200 respectively), then I need to fill in the capital gains section. And there's also another two reasons I might need to, but I'm confident those don't apply to me either.

So I can answer "no" to capital gains because I haven't disposed of anything. Also, outside of ISAs and pensions I had less than £49,200 and my on-paper gains for the year were less than £12,300, so I could answer "no" even if I'd sold everything at the end of the year.

(Also, something unclear: to calculate gains, the obvious thing is to take "price when disposed of" and subtract "price when bought". And we're calculating "price when disposed of" anyway. But we're told to ignore certain disposals when calculating the total disposal price. Do we also ignore those disposals when calculating gains? I would guess so, but mostly I'm glad I don't have to worry about it.)

The next page is the same sort of thing, but in my case less clear.

An important detail here is that you can omit interest earned in an ISA. Nothing on the page says so. You might think that if you only earned interest in an ISA, you say here that you earned interest, and later you tell them it was only in an ISA. At any rate, that's what I thought the first time. But then they told me I owed tax on all that interest, so I decided to go back and tell them I didn't earn any interest.³

I guess that was lying on my tax return, and I just admitted to a crime? Thanks, HMRC.

(I just double checked: if I say I did earn interest, I get a new page where they ask me about "interest that's already been taxed", "interest that hasn't yet been taxed", and "foreign interest that hasn't yet been taxed". There's nothing on that page, either, saying I can omit ISAs.)

Similarly, dividends earned in an ISA don't count here. I did not earn any dividends outside of an ISA last tax year. (This tax year I've earned about £1.50 of them so far.)

Ever get the feeling that someone said "every question needs help text" and the person writing the help text was phoning it in?

I don't know how Child Benefit works. Is anyone in the position of "I got it during the tax year starting on 6 April 2020, but the payments had stopped before 6 April 2020"?

This seems to be an "if you made a mistake, that's fine, you can correct it now" thing. I appreciate that.

This help text is long so I'm not including it. This includes casual earnings, and I might have had some of those? Certainly I have in other years, like the money I sometimes got for getting my brain scanned. But it says I don't need to report if the total from this plus self employment is less than £1,000, which it is for me.

They don't mention gambling income, and mine isn't that high anyway, but I think that should be excluded.

I don't know what these are and the help text doesn't help much, but probably not.

Again, long help text. But the gist seems to be that if I have more than £1,073,100 in pensions in total (5 significant digits! Impressive precision), or if I (or my employer) put more than £40,000 into pensions this year, I need to worry about this. Neither is the case for me. (Yet! Growth mindset.)

One more page of basically the same thing, but this one is easy again. Yes, I contributed to a personal pension. Yes, I gave to charity (and claimed gift aid). My partner and I are ineligible for married couple's allowance (we are neither married nor in a civil partnership, and anyway we were both born after 6 April 1935). Nor can I transfer 10% of my personal allowance to them (we are neither married nor in a civil partnership). I don't want to claim other tax reliefs. I don't have a tax adviser. I… think I haven't used any tax avoidance schemes, but that probably relies on distinguishing between schemes that HMRC thinks are legit (like giving to charity) and schemes that HMRC thinks are kinda sus. I haven't used any disguised remuneration avoidance schemes. I am not acting in capacity or on behalf of anyone else.

There is one interesting one here.

I got a tax rebate in tax year 2019-20 from 2018-19, and then another in tax year 2020-21 from 2019-20. I think neither of these is what it's asking about. Rather, I think this is asking about rebates in tax year 2020-21 from tax already paid in that tax year.

Having filled in those three pages, I get an overview page listing what I've filled in so far and what I still need to do. The first thing to do is income. I only had one employer this year, so I only have one section for that.

They start with the company name, that I already gave them but I can change it now if I want I guess. After that, the next three questions are answered by my P60, which I was able to find because my employer emailed it to me.

These last two are "optional". First time I did this, I thought that meant "we have a copy of your P60 too, so if you don't fill this in we'll fill it in for you". Nope, it means "if you don't fill this in we'll think it's 0".

Then they want to know if I got any tips or other payments not on my P60, which I didn't. This one is optional too - if I try to enter 0 it complains and tells me to leave it blank.

Also, some helpful Javascript on the page makes sure your numbers are given to two decimal places, i.e. pounds and pennies. You can't enter pennies, they'll be truncated (£567.89 becomes £567.00), but it makes sure to show you them.

They also want to know if I was a director (no) and something about "inside off-payroll working engagements" (not entirely clear what this is, but no).

I did have benefits, medical insurance. The value of that will be on my P11D, not my P60, but they don't ask for it here. That's on the next page, which I guess I won't get shown if I didn't have any?

The final question almost caught me out:

I don't normally think of myself as having employment expenses. But I've been working from home, and this is how I claim back some of my utility bills.

My answers to that question added two new pages that weren't listed on the overview before. First for taxable benefits and expenses, listed on my P11D. I have an email copy of that too, I just copy the amount listed for medical insurance on that into the "private medical or dental insurance" box on the page.

They even tell me which number to use. (I have three, "cost to employer", "amount made good or from which tax deducted" and "cash equivalent". The middle one is £0 and the others are equal, so it's not hard for me. But if the middle one wasn't £0, I wouldn't be sure whether to use "cost to employer" or "cash equivalent" - but the latter is the one labeled with an "11", so that's the one to use.)

Then there's a page for expenses not reimbursed. The weirdly specific thing on this page is the box for "fixed deductions for expenses":

Based on the (long) help text, this seems to be "people in certain jobs and industries get to put specific numbers here, regardless of what their actual expenses were". (But they can choose to instead put their actual expenses in another box.) But if you're "a uniformed airline or helicoptor pilot, co-pilot, flight deck crew or cabin crew", you get a different box to put specific numbers in.

But as far as I know I do not work in such a job or industry. The only question relevant to me is "other".

The help text doesn't make it clear how much I can claim, but the link mostly does. I can claim for gas, electricity, metered water, business phone calls and dial-up internet⁴; but only the amount of them that's related to my work. So I guess I should figure out how much higher my bills are WFH than they would be if I was in the office?

I can claim up to £6/week without evidence, or more with evidence.⁵ I can only claim if I have to work from home, not if I do so by choice. I had to look this up, but there were four months where my office was closed (April, May, June and November), so call it 16 weeks. It's also not entirely clear whether that's "£6/week no matter how much I had to work from home that week" or whether I should think of it as £1.20/day, but that doesn't affect me personally. I assume I shouldn't count any holidays I took, but I don't think I took any in those months.

Same-day update: Reddit user pes_planus points out that if you had to work from home for a single day due to Covid, you can claim for the whole year. At £6/week, that's £312 in total. This is the case for both 2020-21 and 2021-22.

Following this page there's one for providing any more information.

I'm not sure what kind of thing I'd need to use it for, but I don't have anything to add. Which is good because it's kind of embarrassingly limited. I'm not allowed to use multiple lines or most punctuation?!

Next up is pensions. I have two pensions that I contributed to this year, my employer one and an SIPP. I'm not aware of having a retirement annuity contract whatever one of those is, and I don't have any overseas pensions, so there are only one or two questions relevant to me. Still, this page is frankly confusing.

I have a standing order to deposit into my SIPP every month, and then about 11 weeks later, 25% of that amount gets added as tax relief. (It's supposed to be 6-8 weeks, but it's not.) So do I count the tax relief in the tax year when it arrived, or the tax year when I deposited the money? So far I've been doing when I deposited, and that's what they suggest by telling me to multiply the amount I contributed by 1.25. On the other hand, if that calculation is always correct, why don't they save me the trouble and do it for me? (Maybe they expect me to have a document somewhere giving the total amount?) In any case, there's nothing saying explicitly what to do. It probably doesn't make that much difference as long as I'm consistent year-on-year, so let's stick with "deposited".

Meanwhile, my employer pension has two sources of contributions. There's the stuff "my employer pays", which doesn't get written on my payslip, and the stuff "I pay", which does. These are both part of my compensation package, but I think we like to pretend that if my employer doesn't write it on my payslip it's not causing me to get paid less? And we make things more complicated and less transparent in service of this fiction.

Anyway, the stuff "my employer pays" I think isn't relevant here. The stuff "I pay" is. In my case, this is also paid relief at source, meaning it's deducted after I've paid tax on it. So that sounds like it matches "Payments to your employer's scheme which were not deducted from your pay before tax". Buuut… I think that question is trying to ask about payments which were expected to be deducted before tax. At my previous job that's how it happened, and it sounds like under certain circumstances that could have gone wrong and then I'd use this question to correct that.

So I think contributions "I paid" to my employer pension actually go along with my SIPP contributions as the answer to the first question, and I leave this one blank. (It's not marked optional, but it doesn't accept 0.)

For charity, there's three questions that look maybe relevant.

For pensions I needed to figure out my contribution plus what was added, for charity I just need to give my own contribution. Obviously. Still, this number is easy to find in my records.⁶

I think this is a roundabout way of asking how much gift aid I expect to donate in tax years 2021-22 and/or 22-23? They'll assume it's the amount I donated this year, minus whatever I put here. And then they'll adjust my PAYG tax code for that, to reduce overpayments.

Most of my charity donations were monthly recurring, but that left me slightly below my "10% of take-home" target, so I did do a one-off donation on top of that which I don't particularly expect to repeat. (But I haven't checked how far below my target I am right now.) But I do expect to donate more in total this year, so I'm going to leave this blank.

Not much help about what specific countries count as "certain" here, or what the definition of charity is for tax purposes. Some of the money I donate goes to MIRI, which is not a UK charity. But it goes via EA Funds, which is. So I'm pretty sure I haven't officially done this.

I can ignore the rest: I'm not asking to treat any payments as being made in a year other than they were actually made; and I haven't given any shares, securities, land or buildings to charity.

The next three pages are about my "PAYE Notice of Coding". They never say what one of these is, but it seems to be a letter they send out, that as far as I can see doesn't contain the words "PAYE" or "notice of coding". (It does say "your tax code notice", which is at least similar.)

Awkwardly, I can't find one more recent than 2018-19: not in my physical documents box, my "I should look through these documents at some point" pile, or my email. I'm sure I've had them more recently, so, um. They don't tell me what to do in this situation, but I guess presumably I can ask them for a new one? But… look at these questions.

The first is "does this number match what's on your coding notice?"

It's not clear whether it's talking about 2019-20 or 2020-21. If I had to guess - and I suppose I kind of do? Then I'd guess it's asking about underpaid tax from 2019-20 and earlier that I should have paid back in 2020-21 through my tax code for 2020-21.

The second is also "does this number match what's on your coding notice?" I think the difference is that last one was talking about underpaid tax from previous years, and this one is talking about underpaid tax from 2020-21 that's already been put on my 2020-21 tax code.

(For these, if I select "no" I get a space to put a different number and a space to tell them where that other number comes from.)

The third question doesn't look like the first two, although it does in the linked video and the help text sounds like it should. It's another "does this number match what's on your coding notice?" This time it's about… outstanding debts other than underpaid tax? What else might I owe to HMRC?

So, okay. I confess I don't really know what's going on with any of this. But suppose I ask HMRC for a replacement Notice of Coding. Are they really going to give me a letter telling me a different number than they have on the website? If so, why not just take that number from wherever they got it for the letter, and put it on the website themselves?

Plus, I've received tax refunds the last two years running, which seems like a weird thing to get if they think I owe them money?

So I think I'm just gonna assume these are all £0.

The three linked videos are short (1:15 to 2:07), but they don't add much that's not in the help text, and I'd much rather they be presented as "text with embedded images" or even just plain text. At least they have subtitles.

Nearly done! Quickfire round.

The next page is one I can't do anything with until after I've submitted the return. This seems like a strange place for it.

Then we get a page about my student loan, which doesn't even have a help text. This is on my P60 again, and also I have it in my own records and I'm pretty sure on a letter somewhere.

Next they have two questions about underpaid tax.

The first is: if it turns out I didn't pay enough tax last year (2020-21, the year I'm filing for), do I want to pay it back through my tax code for next year (2022-23)? i.e. to get it automatically deducted from my payslip starting April 2022? That seems reasonable, sure. If I say no I have to pay it directly by January 2022. I think it's more likely I paid too much, but gonna say yes anyway.

I can only do this if I file by 30th December, which is (checks calendar) soon. I guess if this guide does turn out to be useful to someone, they won't be able to take advantage of this, this year.

The second is: if they think I'll owe⁷ tax this year (2021-22, the unfinished year I'm writing this), do I want it put onto my tax code for… also this year? But this doesn't apply to tax on employed earnings (i.e. the only thing I think I owe tax on) or pensions. Also they apparently think 5 April 2022 gets the past tense.

So, no, because I don't think I'll owe the relevant kind of tax, and if they decide I probably will that's going to confuse things. And no again, because if they decide I'll probably owe something, putting it on my tax code for this year (with three payslips to go) also seems like it's going to confuse things. Certainly it'll make updating my ledger mildly less convenient. If I was going to do this, I'd rather it go on my tax code for next year, like with the previous question.

If I want to make "adjustments" I can do that, but I don't think it applies. (It sounds like these adjustments might be mostly "pretending things happened at different times than they actually happened"?)

The help texts all include "An entry will only be required in limited circumstances", and make it sound like calculating the amounts to go in these boxes is kind of complicated, so I'm glad I don't have to think about them.

If I used provisional figures (I didn't) I give them details about those, and then finally there's a space for "anything else we should know" with the same restrictions on length and punctuation as before. Also I can attach a document if I want.

That's the last thing to fill in. Next there's a page where they repeat some of the things I said back to me, to confirm, and then a page to show me my tax calculation. Initially it just says how much they owe me (or presumably, how much I'd owe them, if that was the case). But then there's also a page showing the full calculation, which I find super valuable for understanding how it all fits together and double checking that I haven't made any mistakes.

The top three numbers here come directly from my return, and Personal Allowance is the same for most people, £12,500 for 2020-21. (If you earn over £100k it starts to decrease, giving a marginal 60% tax band in between two marginal 40% tax bands. I guess it was politically easier to raise taxes this way than to raise them in some more sensible way?) The rest are just adding and subtracting those numbers in the obvious way.

Normally, of the "income on which tax is due", I'd pay 20% on the first £37,500 and 40% on the rest. But

The pension number is the amount I put in the first pension box above, "payments to registered pension schemes". That was money I'd paid into my pensions, plus tax relief I'd been given for it.

The charity number is the amount of gift aid payments I made, times 1.25, because the number I entered didn't include tax relief. (That tax relief went to the charities, not to me.)

The idea with these is that money I put into pensions, or give to charity, is money I don't have to pay tax on. So they make two adjustments for it:

• Return 25% of the money I deposit/donate, to me (for pensions) or the charity (for charities). This happens without needing to fill in a tax return. For those in the basic rate tax bracket, it gives the right amount - £100 pre-tax becomes £80 post-tax, so later we turn the £80 back into £100.

• For higher rate taxpayers, also move some money from the "pay 40% tax" bucket to the "pay 20% tax" bucket. £100 pre-tax becomes £60 post-tax, which (from the previous point) becomes £75 when moved to the pension pot or charity. So they move £75 between buckets, and instead of paying £30 on that I now pay £15. So of the original £100, I now have £15, the charity has £75, and HMRC has £10. This is the same as me giving the charity £75 pre-tax and then paying 40% tax on £25, so it works out fine, but man I find this math confusing.

  If I actually want to give £100 pre-tax, I still need to give the charity £80. They'll claim back £20 and so will I, so I'll have given them £100 pre-tax and paid 40% tax on £33.33. Alternatively I can take the £15 I got returned from £60 and redonate that, becoming £18.75 to the charity, then also donate the £3.75 I get returned from that, and so on - in the limit, this works out as the charity having £100 and myself and HMRC having £0 each.

  (When HMRC shifts money between buckets, a higher-rate taxpayer can become a basic-rate taxpayer. I assume the math still works out fine at the boundaries? I haven't checked though.)

And so we have these calculations:

I'm not sure why "Income Tax due after allowances and reliefs" comes before "Income Tax due", but since I don't have any allowances or reliefs the numbers are the same for me. I assume student loan repayments would be different from £0 if the wrong amount had been automatically taken from my payslips.

Now I get to download a PDF copy of what I just filled in, formatted on what I guess is a copy of the paper form I'd use if I wasn't doing it online. And then right before I submit, two more checkboxes to tick:

This first one seems less helpful than it might be, failing to distinguish "oops I did this wrong" from "I didn't need to do this". I guess they don't want to confuse non-mathematicians who didn't have any coronavirus support payments, and it still works as a reminder.

I think after I click "submit" on this page, I'll enter my password again and be finished. But I haven't done that yet - I'll wait to see if anyone wants to tell me I did something wrong, first.

Update December 29: Yep, that was it. There was a "submitting" spinner for about 30 seconds, and then

Woop! It takes up to 72 hours to process and I can make corrections in that time. But assuming I don't realize I have any of those, I'm done. At some point they're going to send me back the money I overpaid, historically that's gone into my bank account but I don't remember if I've had to do anything particular to make it happen. Certainly it wasn't very involved if I did.

They give me a "submission receipt reference number", which looks like 32 digits in base 36 (numbers and uppercase letters, mine includes Z, I and O so probably all letters are possible). Sounds like it's a hash of the contents of my return, with a signature so that no one else can generate a valid one. Neat. It's also included in the PDF copy I downloaded earlier.

I assume it's not intended to be confidential, but they don't say it's safe to share, so I won't.

Ten Hundred Megaseconds

A megasecond, one million seconds, is 11 days and 14 hours. (13 hours, 46 minutes and 40 seconds, assuming no leap seconds.) About a week and a half.

Ten megaseconds is 116 days. (115, plus 17:46:40.) A bit less than four months, a third of a year.

A hundred megaseconds is 1,157 days. (Plus 9:46:40¹.) Roughly three years and two months - it could be 61 or 62 days depending on leap years.

I think it might be kind of nice if we celebrated a hundred megaseconds additonally-to and/or instead-of birthdays.

One thing I like about it is that humanity won't stay on Earth forever. If we go interplanetary, I think we're more likely to keep measuring time in seconds than in Earth years. I like this as a way of looking to the future.

Another thing is, it doesn't happen at the same time every (Earth) year. Some birthdays fall on or near holidays, or during seasons we don't much like, and that can make them less convenient to celebrate. If that happens to you one hundred megaseconds, it probably won't happen the next hundred megaseconds. (I think a five month offset would be better than two for this purpose, you'd never be in the same season twice in a row, but two is better than zero.)

I also just think it's kind of a sweet spot time interval? I'm old enough now that years don't feel very long, decades² do, and a hundred megaseconds is nicely in between.

(If "hundred megaseconds" is unwieldy, a quick look at wikipedia suggests "yisec" from Chinese or "okusec" from Japanese. Or maybe "myri-myri-sec" from Ancient Greek. I think my favorite would be "hecto-megasec" from Ancient Greek via SI. I'm not going to try to use any of these myself though.)

Ten hundred megaseconds, a billion seconds, is 11,574 days. About 31 years eight months. (The remainder is 251-253 days, depending on leap years³. Plus 1:46:40.) That's how old I turned on August 21st this year.

I didn't do anything to celebrate, partly because even though I'd put it in my calendar a long time before, I didn't remember until a couple days after, and partly because I'm not very good at celebrating things. (I don't normally celebrate my birthdays, either.) I did do some work on Haskenthetical for the first time in months, which feels like a good start to the next billion. Maybe I'll celebrate turning eleven, which will be on October 21st, 2024.

If you're interested, here's a widget you can use to calculate your age, or other intervals, in hundred megaseconds.

Thanks to Miranda for comments.

Against "blankfaces"

(Content note: minor spoilers for Harry Potter and the Order of the Phoenix.)

Scott Aaronson writes about blankfaces,

anyone who enjoys wielding the power entrusted in them to make others miserable by acting like a cog in a broken machine, rather than like a human being with courage, judgment, and responsibility for their actions. A blankface meets every appeal to facts, logic, and plain compassion with the same repetition of rules and regulations and the same blank stare—a blank stare that, more often than not, conceals a contemptuous smile.

I want to push back against this a bit.

First, one of the defining aspects of blankfacedness is their internal experience. It's someone who enjoys wielding their power. This is a very hard thing to judge from the outside.

I used to work in a cinema. One day a mother came in with her young child, perhaps fivish years old. She was late for a busy screening, and the person selling tickets warned they might not be able to sit together. She said that was fine, bought popcorn and went in. Soon afterwards she came back out, complaining that they couldn't sit together. She wanted a refund for the tickets (fine) and popcorn (not fine, but she insisted). The conversation between her and my manager escalated a bit. I don't remember who brought up the police, but she at least was very confident that she knew her rights and the police would back her up if they arrived. Eventually he gave her a refund.

If it had been up to me? And if I hadn't had to worry about things like PR and "someone yelling in the lobby would ruin the experience for people watching movies"? I think I would absolutely have used the "no, sorry, those are the rules" move. I would have been happy to do so. Does that make me a blankface? But it's not that I would have enjoyed wielding my power as such. Rather it's that I would have enjoyed punishing her, specifically, for acting in ways that I endorsedly think are bad to act in.

Does someone's internal experience matter, though? If they act a certain way, should we care how they feel? I think we should, and if we don't we shouldn't make claims about it.

That is, if what you care about is whether someone is acting a certain way, then don't mention enjoyment when you define a blankface. And if you really do care about the enjoyment part of it - well, how do you know what someone is feeling and why?

I predict that if the term "blankface" takes off, no matter how much people defining the term emphasize the "enjoys" part of it, people using the term will not be careful about checking that. Partly I think this because Scott wasn't: in two of his examples, he accused people of being blankfaces whom he'd never interacted with and could not identify. Does he really think that a web portal was badly designed out of malice? But also I think that… like, even if you can tell that someone is just acting a certain way because they enjoy it, even if you're really sure that's what's going on, you won't properly be able to capture that in your description of the events. So people will read a story where it seems like the thing making them a blankface is the way they acted, and then they'll tell their own similar stories where people acted in similar ways, and they'll use the term "blankface".

There's a lot of potential here for a kind of (reverse?) motte-and-bailey, where the bailey is "I'm calling someone a blankface which is explicitly defined as having an enjoyment part to it", and the motte is "…but no one uses it that way, so obviously I didn't mean to imply that I know what was going on in their head".

Here's another reason someone might externally act blankfacedly: fear. Yes, this is ridiculous, but if I admit that out loud I'll be seen as undermining my boss who already has it in for me, so…. Or, exhaustion: this is the third sob story I've heard today. I cannot deal with feeling more sympathy.

Given how strongly Scott feels about blankfaces (they've apparently dehumanized themselves and deserve no mercy), I certainly hope he cares whether they act blankfacedly for sympathetic or unsympathetic reasons. And if we're to care about that, I think we have to admit that most of the time we don't really know.

Second and relatedly, I think we should distinguish between "this person is blankfacing" and "this person is a blankface". Like, maybe someone right now is enjoying wielding petty power for the sake of it. I don't currently predict that that person routinely enjoys acting that way, or enjoys acting that way in every situation where they have petty power. That's maybe not much consolation to their victim right now, but still.

Perhaps I should predict that? But I currently don't, and Scott gives me no reason to.

Third, I'm not sure Umbridge is an example of the archetype. Or, if Umbridge is really what Scott wants to point at, I'm not sure his explicit definition matches up.

The most despicable villain in the Harry Potter universe is not Lord Voldemort, who's mostly just a faraway cipher and abstract embodiment of pure evil, no more hateable than an earthquake. Rather, it's Dolores Jane Umbridge, the toadlike Ministry of Magic bureaucrat who takes over Hogwarts school, forces out Dumbledore as headmaster, and terrorizes the students with increasingly draconian "Educational Decrees." Umbridge's decrees are mostly aimed at punishing Harry Potter and his friends, who've embarrassed the Ministry by telling everyone the truth that Voldemort has returned and by readying themselves to fight him, thereby defying the Ministry's head-in-the-sand policy.

Anyway, I’ll say this for Harry Potter: Rowling's portrayal of Umbridge is so spot-on and merciless that, for anyone who knows the series, I could simply define a blankface to be anyone sufficiently Umbridge-like.

Spoilers: the educational decrees are not the extent of Umbridge's villainy. She also sends dementors to attack Harry and his cousin. She tries to slip Harry a truth serum, and later tries to force-feed it to him. When she can't do that, she tries to torture him. None of this is legal¹, some of it is super-duper illegal, and her superiors aren't pressuring her into it. Umbridge doesn't simply act like a cog in a broken machine. She exercises judgment, and I think even some courage, in service of villainous ends.

(I am not, here, describing Umbridge's motivations. I think I have some idea of what they are, but I'm not sure I'd capture them very well and my feeling is they're not super relevant for this bit.)

However annoyed Scott may be at his daughter's lifeguard, I predict that describing the lifeguard as Umbridge-like is unfair. I predict, for example, that she would never take initiative to deliberately engineer a situation in which Scott's daughter nearly drowns.

I think Scott is pointing at something true, and important to know about. But I think he's conflating a few different things (blankfacing-regardless-of-internal-experience, blankfacing-specifically-meaning-enjoying-it, being-a-blankface, Umbridge), and I worry that the way he's pointing at them will result in bad discourse norms. I still think it's good that he published, but I don't think that essay is the ideal version of what it could be, and I'm trying here to point at ways I think it falls short.

99% shorter

I listen to podcasts while doing chores or at the gym, and often feel like I'm learning something but then can't really remember anything afterwards. So for the past ~month I've been doing an experiment where I write brief summaries of them afterwards, usually same-day but sometimes a bit later. Generally I avoid all forms of fact checking, both "what did the episode say" and "what is actually true", though I don't stop myself if I feel like doing it.

I've been posting them to my shortform on LessWrong. Mostly in reply to a single comment for tidiness, but two of them I accidentally posted to the top of the thread and one as a reply to another. Initially that was just because I wanted to be able to write them from both my work and personal laptops. (This blog is published through git. I don't have the repo on my work laptop, and probably shouldn't do.) But I kind of like it. A downside is it's slightly less convenient to include episode numbers or air dates or even titles unless I remember them. So I might be less consistent about that, though it feels good to have a fairly straightforward way to look up the episode given my summary.

I've skipped all the fiction podcasts I listen to, because that's not why I listen to fiction. Also most interviews, those seem generally hard to summarize, though it would probably be reasonable to extract a few individual points. And one Planet Money episode seemed like it would be irresponsible to summarize carelessly, and I didn't feel like trying to be careful about it. But I've summarized every episode of 99% Invisible and History of English, and all but that one episode of Planet Money, that I've listened to in this time. Also one episode of Rationally Speaking and one of Corecursive.

I'm not really sure what this experiment was trying to test. I was curious, so I did it. Some things I've maybe vaguely learned: first, I think I can generally do a decent summary. I frequently get distracted for a few seconds at a time, but there's a difference between that and being actually distracted, and I think it shows. (I listened to 99% Invisible, "Tanz Tanz Revolution", while looking at parking restrictions in my area and anticipating my evening plans. There's significant parts of that episode that just rolled right past me.) I don't think you could reliably pick out the episodes I didn't do same-day, I mostly don't even remember myself. (Hot Cheetos was several days later, but I did go over it a bit in my head in between, which I wouldn't have done if not for this project.)

Second, I seem to retain less of History of English than other podcasts. That feels like it matches my intuition. I expect Causality, too, but that only recently released a new episode and I haven't listened to it yet.

Third, this causes a marginal shift from podcasts towards music for me. I predicted this and I'm okay with it, I feel like I don't listen to music enough relative to how much I enjoy it.

If I wanted to make it more of an experiment, I could randomly select some episodes to write up quickly, some to write up the next day, the next week, next month. Not sure how much I feel like this, but maybe. Another thing would be quizzing me on the episodes I did and didn't write up, without rereading. And I'm curious whether I'll ever actually want to refer to these, and if so whether I can predict which ones I will.

I haven't been tracking my time closely, but I think they mostly take me 20-30 minutes to write? I listen between 1.5x and 2.1x speed depending on the podcast, so that's considerably longer than they usually take to listen to.

Dunno what I'll do in future. Maybe write up some summaries, of episodes I found particularly interesting?

I'm including here three that I particularly liked, but if you feel like reading the rest see the LW thread.

Planet Money (14 May 2021): Blood Money

America lets you sell blood plasma for money. The centers will call it a donation, but you get money for it, so. You can come in a couple of times a week and especially for low income people it can be a significant addition to their income. There are referral bonuses, and you get paid more if you come in more often. (Seems weird?) These centers are mostly located in low-income areas.

There's a history here involving Nicaragua. Under a dictator there was a center doing this, and a journalist was writing about concerns, and eventually the dictator had the journalist killed. Riots in the aftermath left the center burned down and eventually the dictator got deposed. At some point the WHO wrote up an agreement or something not to allow blood plasma to be sold. Almost everyone's signed, but not the USA. Now almost everyone gets their blood plasma from the USA. Four exceptions, who allow it to be sold and are self-sufficient: Germany, Austria, Czech Republic, Hungary.

(Farmer's Dilemma!)

We speak to a Canadian healthcare person about why Canada doesn't allow it. Three concerns. He's not too worried about incentives for people to sell bad blood, apparently we can sanitize it, even of HIV. He's also not too worried about health impacts on sellers; they get a checkup every four months to make sure they're still good, and there's some anecdotal evidence that maybe it should be more frequent but basically it seems fine. He seemed more concerned about "if people can sell plasma, will they do other things like regular blood donation for free?" I don't remember the commentary on that. I think he said that if the USA didn't allow selling they'd probably have to in Canada, but as long as they do it's unlikely to change.

We also speak to a Brazilian doctor saying that plasma and the things it's used for are essential, there are people who will die without it, get over yourselves.

Concerns that if either demand raises (finding new uses: there are studies showing promise in Alzheimers) or supply drops, there might not be enough. In fact supply has dropped during Covid: possible reasons include "sellers need the money less thanks to stimulus"; "if your kids are at home all the time you might be too busy"; "a lot of the sellers near the border are Mexicans who can't come over any more".

99% Invisible #434 (9 Mar 2021): Artistic License

This is the most American story. (I actually wrote it up before starting this experiment.)

After states start requiring license plates, Idaho realizes they can be used for advertising, and start boasting about Idaho potatoes on their plates. North Idahoans grumble because that’s more of a Southeast Idaho thing. Tourists start stealing plates as souvenirs, causing people to be very confused when they’re pulled over because who checks whether they still have a license plate.

Anyway, New Hampshire’s state motto is Live Free or Die, I don’t know if that’s just a generic America thing or a specifically fuck-communists America thing. But a super fuck-communists guy gets them to put the motto on the license plate, presumably for fuck-communists reasons but I dunno if that was explicit or just subtext.

And then a Jehova’s Witness is like, no, I don’t want to, God gave me life and I’m not gonna give that up for freedom. So he starts covering up that bit with tape. And he gets arrested and the fuck-communists guy is now governor and not inclined to give an inch, so it goes to the Supreme Court who split 6-3 but the pro-freedom side wins, the government is not allowed to compel you to express your love of freedom.

Later: Texas allows specialty plates, some group designs a plate to support a cause and then you pay a little extra for it, some going to the group and some to the state. Most of these designs are just rubber stamped. But this is Texas, so the Sons of Confederate Veterans want a license plate supporting their cause, and they want the Confederate flag on it. The state says no, they sue the state, and the Supreme Court sides with the state 5-4.

Corecursive (2 May 2021): Etherium Rescue

"Daryl" was an ETH user who fat-fingered a transaction. Went online for help, guest said sorry, nothing anyone can do. Then later guest went o shit maybe there is.

Daryl was playing with uniswap, a smart contract letting people provide liquidity for exchanging crypto, e.g. ETH for USDC. Normally when providing liquidity you'd do two things in one transaction, with something like a try/catch letting you do them atomically. I guess Daryl had only done one of them? Anyway, his money was just sitting there, and as soon as anyone tried to take their liquidity from uniswap they'd get Daryl's money as well.

Guest realized this and went to check, and the money was still there. But! He also remembered stories of generalized ETH frontrunners. These will examine the pending transactions, see if there's something in there they can use to make money, and if so, submit their own transaction with a higher fee so it gets executed first. Guest worried that one of these would show up if he tried to recover the money. He asked on a group chat if others would also be worried, some of them were, and they got together to try to figure something out.

Ultimately they'd need to do some kind of obfuscation so that a bot wouldn't try the thing they were doing. They settled on two separate transactions in one block, where the second one wouldn't do anything unless the first had already happened, hoping bots would only try them separately. But there's stuff set up to protect you from making transactions that don't do anything, and it was stopping them from making the second.

Guest was tired and stressed and the money might disappear at any minute, so eventually Guest said YOLO we'll do them in two different blocks and hope. The second transaction got front-ran and they lost the money. On the plus side his worries were vindicated.

Guest and Adam (host) discuss Meditations on Moloch. The thing they take away from it is that you need regulation/Leviathan. Guest says for Hobbes the Leviathan was hereditary monarchy, recently we've been trying democracy and that seems better overall, but he's optimistic that smart contracts will be another solution.

A command-line grammar of graphics

Sometimes I'm at my command prompt and I want to draw a graph.

Problem: I don't know gnuplot. Also, there's a couple things about it that bug me, and make me not enthusiastic about learning it.

One is that it seems not really designed for that purpose. It implements a whole language, and the way to use it for one-off commands is just to write a short script and put it in quotes.

The other is its whole paradigm. At some point in the distant past I discovered ggplot2, and since then I've been basically convinced that the "grammar of graphics" paradigm is the One True Way to do graphs, and everything else seems substandard. No offense, gnuplot, it's just… you're trying to be a graphing library, and I want you to be a graphing library that also adheres to my abstract philosophical notions of what a graphing library should be.

If you're not familiar with the grammar of graphics, I'd summarize it as: you build up a graph out of individual components. If you want a scatter plot, you use the "draw points" component. If you want a line graph, you use the "draw line segments" component. If you want a line graph with the points emphasized, you use both of those components. Want to add a bar chart on top of that too? Easy, just add the "draw bars" component. Want a smoothed curve with confidence intervals? There's a "smooth this data" component, and some clever (but customizable) system that feeds the output of that into the "draw a line graph" and "draw a ribbon" components. Here's a gallery of things it can do

So, rather than adapt myself to the world, I've tried to adapt the world to myself.

There's a python implementation of the paradigm, called plotnine.¹ (It has its own gallery.) And now I've written a command-line interface to plotnine.

It's not as powerful as it plausibly could be. But it's pretty powerful², and if I stop developing now I might find it fully satisfies my needs in future. For example, I took a dataset of covid cases-per-capita timeseries for multiple countries. Then both of these graphs came from the same input file, only manipulated by grep to restrict to twelve countries:

(The second one isn't a type of graph that needs to be implemented specifically. It's just a combination of the components "draw points", "draw line segments" and "draw text".)

Now admittedly, I had to use a pretty awful hack to get that second one to work, and it wouldn't shock me if that hack stops working in future. On the other hand, I deliberately tried to see what I could do without manipulating the data itself. If I wasn't doing that, I would have used a tool that I love named q, which lets you run sql commands on csv files, and then there'd be no need for the awful hack.

Anyway. If you're interested, you can check it out on github. There's documentation there, and examples, including the awful hack I had to use in the above graph. To set expectations: I don't anticipate doing more work on this unprompted, in the near future. But if people are interested enough to engage, requesting features or contributing patches or whatever, I do anticipate engaging back. I don't want to take on significant responsibility, and if this ever became a large active project I'd probably want to hand it over to someone else, but I don't really see that happening.

Haskenthetical update: user-defined macros

A while back I wrote about Haskenthetical, a language that I'm implementing for fun. I'm still working on it, at least from time to time.

The thing I'm pleased to have implemented recently is user-defined macros. This makes it potentially a lot more ergonomic.

(You still can't do anything with it, really. It has no I/O except for printing a single value at the end, and that's in a constrained format. It's beyond the reach even of sysfuck. But, like. If you could do things with it, user-defined macros would make it nicer to do those things.)

These aren't hygienic macros, partly because when I used Racket a little at my old job I never really got my head around those, but also because they would have been more work. I'm more inclined to go for Clojure's approach of namespace qualifying, but since I don't have any kind of namespaces yet it seems premature to think of that.

Quasiquote

The biggest test of Haskenthetical yet has been using these macros to implement ^¶ quoting (quote, like a standard lisp ') and quasiquoting (qq, like `, using ↑ and ↑↑ for splicing). Quasiquoting in particular was one of the hardest pieces of code I think I've ever written, and it's only 100 lines in the end. Some of the things that made it difficult:

• Haskenthetical is deficient. Pattern-matching causes deep nesting and the chance of forgetting a case, while type eliminators are hard to read and flow poorly. (It's deficient in many many ways, but those are the ones that seem particularly relevant here.)

• More importantly, the current implementation of Haskenthetical is deficient. The error messages are completely unhelpful, telling you that two types didn't unify but not where they occured. Both the debugging output and the output from the program itself are hard to read. I also uncovered one straight-up bug in the process, which was easy to fix when I tracked it down but hard to track down because the error messages are completely unhelpful.

• Also, I'm deficient as a thinker. I have a brain architecture that can't easily distinguish between the object and the meta levels, or the meta and the meta-meta levels. Wtf, who designed this piece of shit?

• The problem itself is well-targeted to hit those last two deficiencies. The quasiquote function simply converts syntax trees to syntax trees, but some of the trees in question represent Haskenthetical expressions of certain types, and some of the transformations need to preserve or manipulate those types. The type checker could tell me "you're passing a list of trees somewhere you should be passing a single tree", but since everything is either a tree or a list of trees that doesn't narrow things down much. And it couldn't tell me "you're trying to convert this list-of-trees to a tree-evaluating-to-a-list but you've actually created a tree-evaluating-to-a-type-error".

Eventually I simplified the problem by first parsing a syntax tree into a new datatype, and then converting that new datatype into another syntax tree, giving slightly more type information. That also made it easier to perform some simplifications in the parsing step, which made the output simpler and easier to check. Ultimately this brought it to a level that my brain could handle.

The result is that you can implement macros by describing the shape of the code they could generate, rather than implementing something that evaluates to that shape. Here are two ways to write an if-elsif-else macro (technically only the "else" is necessary):

(defmacro if
  (λ ts
    (if~ ts (Cons $else Nil)
      else
      (if~ ts (Cons $cond (Cons $then $rest))
        (STTree (» Cons (STBare "if~")
                        cond
                        (STBare "True")
                        then
                        (STTree (Cons (STBare "if") rest))
                        Nil))
        (error! "`if` must have an odd number of args")))))

(defmacro if-qq
  (λ ts
    (if~ ts (Cons $else Nil)
      else
      (if~ ts (Cons $cond (Cons $then $rest))
        (qq (if~ (↑ cond) True (↑ then) (if-qq (↑↑ rest))))
        (error! "`if` must have an odd number of args")))))

In the first, we manually construct a SyntaxTree with the STTree constructor (which represents syntax like (...)), passing it an argument of type List SyntaxTree. In the second, we construct the same SyntaxTree using quasiquoting, with splices ↑ and ↑↑ to indicate where the input parameters go. ↑ expects its parameter to be a SyntaxTree, which gets interpolated directly, while ↑↑ expects its parameter to be a List SyntaxTree, each element of which gets interpolated. So (if-qq a b c d e) macroexpands once to (if~ a True b (if-qq c d e)), and ultimately to (if~ a True b (if~ c True d e)).

There's another macro I use in the first definition: » and « can be used to "make functions variadic" by threading them in between other parameters like a fold. We have

(» f a b c d) === (f a (f b (f c d)))
(« f a b c d) === (f (f (f a b) c) d)

These are built-in, but they don't need to be. It's partly a matter of convenience, and partly because I implemented them to test macroexpansion before I implemented user-defined macros.

Implementation

There's a circularity problem when defining macros. Sometimes you'll want to do so using functions you've previously defined. And sometimes you'll want to use a macro to help define a function. Allowing either one of these seems easy enough, but how do you do both?

I don't know how lisps handle this problem. I don't remember it ever tripping me up when I used them (I have some experience with Common Lisp, Clojure and Racket), but that was years ago and maybe I just didn't stress test them properly.

Haskell solves it with modules. You can't call a macro in the same module you define it. In Haskell's case that also means not in the same file you define it. That's fine for Haskell I guess, but I wouldn't want it for Haskenthetical.

But getting rid of one-module-per-file, this seemed like a reasonable solution. So I added a new top-level form, (declarations ...). With this, the order of events is:

1. Parse the input file into a list of SyntaxTree (trees whose leaves are barewords, floats and strings). Separate trees into those of the form (declarations ...) and all others.

2. For each (declarations ...) block,

   1. Parse its children into statements ((def ...), (type ...), (defmacro ...)) and expressions (underneath def and defmacro). In this step, we have a list of macros available in the environment, and recognize when those are being called, declining to parse the arguments to macros.

   2. Recursively macroexpand the result, evaluating macros (which can be builtins written in Haskell or Haskenthetical closures) on their arguments. Since macros convert SyntaxTrees directly to SyntaxTrees, we have to parse the result of each expansion.

   3. Add all the type declarations ((type ...)) to the environment, along with their constructors and eliminators. We have to do this simultaneously, because types can be mutually recursive.

   4. Type check all the value definitions. This has to be simultaneous too, for the same reason. If type checking passes, add them to the environment, which doesn't strictly need to be simultaneous but is forced to be by my implementation. (Every defined value is a thunk storing a copy of the environment at the time it was defined. These environments do need to be mutually recursive.)

   5. Type check the macro declarations, and add them to the environment. The type of a macro is -> (List SyntaxTree) SyntaxTree, I think it's pretty okay for macros to be impure but in future they'll need to be augmented with some way for macros to consult the environment. We can do these one at a time, because macros declared in the same block can't reference each other directly. (They can generate references to each other.)

3. Treat the remaining top-level trees as another (declarations ...) block, and go through the same process. But this block is required to also contain a single top-level expression, after macroexpansion.

4. Type check that expression. If type checking passes, evaluate it in the environment we've been building up.

I don't know if this approach is standard. It seems to work. It doesn't allow macros to expand to declarations blocks, which is a shame. In future, if I implement importing from other files, I might figure out how to allow macros defined in other files to expand to those blocks. On the other hand, once I have IO it would be interesting to see if I can implement importing from other files purely in Haskenthetical, and use declarations blocks as the only built-in form of module. That would be great and terrible.

Other things

I've added comments now. # is a comment to end of line, but it has to be surrounded by whitespace¹, because I don't want to eliminate the possibility of using that character in combination with other things. #! works the same, to allow shebang lines. I don't yet have multiline comments, or single-form comments.

I also have an emacs major mode, which parses comments badly because emacs' easy-mode syntax highlighting only supports fairly specific kinds of syntax. But it's better than nothing.

I discovered two bugs writing the previous post. The type checker was ignoring type annotations on λ parameters. I've fixed that. It also wasn't giving an error if a type annotation in a pattern match was more general than the type it matched. I've only partly fixed that. I think to fix it more thoroughly I'd need to implement constraints and constraint solving. I might also decide to eliminate that feature - GHC is going in that direction.

What's next?

(Assuming I continue at all, that is. This is fun, but it's not the only fun thing.)

I dunno. The test suite is looking a bit embarrassing, I've only added a handful of new tests since I first wrote about this project. At first I wanted them to be nicely grouped, tests about type checking and type declaration and evaluation, but there's no clear division and I shouldn't try for it. Instead I think I'll just have one big file with lots of examples and some metadata.

One thing is that macros can't currently be used in types or patterns. So you can write (list a b c) instead of (Cons a (Cons b (Cons c Nil))) when constructing a list, but not when destructing it. And you can't write a three-argument function type as (» -> a b c d). This should be some fairly low-hanging fruit.

Writing the quasiquoter made me want better error messages. That's definitely not low-hanging fruit though. Especially with macros in play - how do you give the location of an error in the file when any given piece of syntax might not have been in the original file? There are options here but also, annotating everything with a source location sounds boring even if I didn't have to worry about macros.

Another possibility is to look at the type checker only. Write You a Haskell implements it in two different ways in the same chapter. I followed the second approach, section "constraint generation", because Diehl says it "becomes easier to manage as our type system gets more complex and we start building out the language". But it means that by the time I get a type error, I've lost all context. The types String and Float should unify but don't, great, but I have no idea why I'd expect either of those. And I'm not convinced it accomplishes its goal - the point is to separate constraint solving from traversing the tree, but you still need to solve constraints when you generalize a variable during traversal (the implementation on the page is buggy).

I think the first approach in WYAH is also used by Typing Haskell in Haskell, where you solve constraints as you traverse the program. Then the context is still there when you encounter a type error, which might make it easier to report a helpful error message.

Half the macros I've written so far (», «, list) could simply be functions, except Hindley-Milner doesn't have variadic functions. Can it be expanded to have them? The paper practical variable-arity polymorphism looks interesting on this question, though I haven't looked closely or thought deeply. I'd be going even further out of my comfort zone, but that's not a bad thing.

I've been thinking about nomenclature a bit. Right now the Either type is called +, which might make sense if the , type was called * but it's called ,. List and Maybe aren't built-in, but when I've been defining them I've been calling them List and Maybe with constructors Nil Cons Nothing Just. I'm thinking I might go with regex-inspired names,

• The unit type becomes ε, with constructor also ε.
• + $a $b becames || $a $b. Constructors _| $a and |_ $b, perhaps. (But which way around? I have no intuition for that.)
• Maybe $a becomes ?? $a. Constructors ι $a and ?ε. (ι is the inclusion map -> $a (?? $a). But lots of types might want one of those, so maybe it should be ?ι, with ι for the only constructor of the type ι $a. Will Haskenthetical find that type useful like Haskell does?)
• List $a becomes ** $a. Constructors :: $a (** $a) and *ε, but I'm not sold on ::.

This seems pretty terrible, how do you pronounce any of that? But it also seems kind of fun.

Specialized Labor and Counterfactual Compensation

I have three purposes in this post. The first is to review the formal game theory found in Robert Ellickson's Order Without Law. It's not a large part of the book, but it's the part that I'm most qualified to judge. Not that I'm a formal game theorist myself, but I'm closer to being one of them than to being any kind of social scientist, historian or lawyer. If his formal game theory is nonsense, that would suggest that I ought to discount his writing on other fields, too. (Perhaps not discount it completely, especially because formal game theory is outside his main area of study. Then again, lots of the book is outside his main area of study.)

Spoiler alert: I think he holds up reasonably well. I want to ding him a few points, but nothing too serious, and he possibly even contributes a minor original result.

My second purpose, which is valuable for the first but also valuable of itself, is to try to extend it further than Ellickson did. I don't succeed at that.

My third is simply to be able to cut it from my in-progress review of the rest of the book.

Ellickson discusses two games. One is the classic Prisoner's Dilemma, in which you either Cooperate (for personal cost but social benefit) or Defect (for personal benefit but social cost).¹ The other he calls Specialized Labor, in which two people must choose whether to Work on some common project or Shirk their share of it. It differs from the Prisoner's Dilemma in two ways. First, it's asymmetrical; one player is a less effective worker than the other, and gets less payoff from Working while the other Shirks than does the other player. The other is that in this game, the socially optimal outcome is Work/Shirk, not Work/Work.

(Many authors consider that the second change isn't really a change, and that a Prisoner's Dilemma can perfectly well have Cooperate/Defect be socially optimal. So they'd say Specialized Labor is simply an asymmetrical version of the Prisoner's Dilemma. In my taxonomy I define the Prisoner's Dilemma more narrowly than that; see also this comment. Ellickson uses the same narrow definition as me. I'd instead say Specialized Labor is an asymmetrical version of Too Many Cooks.)

Note that payoffs aren't measured in utility. They're measured in something Ellickson calls "welfare". He doesn't really explore the formal consequences of this. But what it gives us is that, since welfare is supposed to be objective, we can sum different people's welfare; when I used the phrases "social cost" and "socially optimal" in the previous paragraphs, talking about the sum of both players' results, that was a meaningful thing to do. I'm not sure exactly what it costs us, except that I don't expect results about mixed strategies to hold. (Someone won't necessarily prefer "50% chance of 3 welfare" to "certain chance of 1 welfare". I wasn't planning to consider mixed games anyway.) We can still assume that people prefer higher amounts of welfare to lower amounts of it.²

I'm going to pretend that Cooperate and Defect are also called Work and Shirk, so that I don't have to use both names when talking about both games.

In normal-form, these games look like this:

How to read these symbols: the subscript is the player who gets the payoff, the first letter is their move, and the second letter is the other player's move. If the subscript is $ * $, then this combination is symmetric.³ So $ ws_1 $ is the payoff to Player 1, if he Works while Player 2 Shirks. $ ws_2 $ is the payoff to Player 2, if she Works while Player 1 Shirks.⁴ $ ws_* $ is both of these values, when they're equal to each other. And to be clear, when they're equal, $ ws_1 $ can stand in for $ ws_* $ just as easily as the other way around.

To help make the structure more visible, I've colored the symbols in green or red according to local incentive gradients - green for "this player prefers this outcome to the outcome they get from changing their move", red for the opposite of that. So when $ ws_1 $ is red, that means $ ss_1 > ws_1 $, since $ ss_1 $ represents Player 1's payoff if he changes his move while Player 2 keeps hers the same. A quadrant is a Nash equilibrium (meaning "neither player wants to change their move unilaterally") iff it has two green symbols. I've also given a slightly darker background to the socially optimal quadrants.

Comparing these games, Ellickson claims for example that norms will tend to punish someone who Shirks in a Prisoner's Dilemma, rather than rewarding those who Work, because eventually most people will Work and it's cheaper to sanction the thing that happens rarely. But in a Specialized Labor game, norms will tend to reward the efficient worker ("cheapest labor-provider") for Working, because that encourages people to obtain the skills necessary to perform this work. There's a background level of skills that everyone is expected to have, and people are punished for falling short of them and rewarded for exceeding them.

So most of the points I want to ding Ellickson here are because this is kind of a strange choice of games. For one thing, it seems to assume that: teaming up to work is more expensive than working individually, iff players have unequal skill levels.

Honestly I don't think that's so implausible as a heuristic. I think "most work projects have gains from working together" is a decent guess, and then one way to remove those gains could be if one player is much more skilled than the other. Still, Ellickson doesn't make this argument, or acknowledge that the assumption is kind of weird.

Another way to justify the omission is if the ommitted possibilities don't add much of interest. Prisoner's Dilemma and Specialized Labor are opposite corners in a two-by-two grid parameterized by "synergistic/discordant" (gains or no gains from cooperation) and "symmetrical/asymmetrical". If our tools for working with them can also be applied to the other corners without much extra effort, then there's no need to consider the others in detail. More on this later.

Something weird on the face of it is that in Specialized Labor, Work/Work results in the same payoff to both players. Why assume that that's symmetrical? But I don't think this is a big deal. Plausibly people can calibrate how hard they work if they think they're getting a worse result than the other. Also I suspect you just don't change much by allowing it to be asymmetrical, provided that both payoffs are in between $ sw_* $ and $ ss_* $.

Similarly you might suppose that the efficient worker doesn't just pay less to Work than the inefficient worker, he also does a better job. In which case we might want to set $ sw_1 < sw_2 $. But again, I doubt that matters much.

Here's my largest objection: Ellickson doesn't consider that work might be worth doing selfishly. In both games, you maximize your own outcome by Shirking, and if that means the work doesn't get done, so be it. But that puts a narrow band on the value of a piece of work. From a social perspective, it's not worth doing for the benefits it gives to one person, but it is worth doing for the benefits it gives to two. I think a lot of the situations Ellickson looks at don't really fit that model. For example, building a fence seems like something you'd often do of your own accord, simply for the benefits it gives to yourself, but Ellickson considers it a Prisoner's Dilemma because most people have the relevant skills. (He doesn't analyse whether fence-building is more easily done in tandem.)

To model this possibility, we'd set $ ws_1 > ss_* $, and maybe $ ws_2 > ss_* $ as well. This gives the game that I like to call the Farmer's Dilemma and others call Chicken, Hawk/Dove or Snowdrift. (Here's why I call it that.) Normally I think of the Farmer's Dilemma as symmetrical, but the asymmetrical case seems fine to count as an instance of it, at least right now.

The tricky thing about this game is that even though you'd be willing to do the work yourself if no one else benefitted, the fact that someone else does benefit makes you want them to join in and help with the work. If they decline, your only in-game way to punish them is not to do the work, which hurts you too - but if you don't punish them, you're a sucker. This is fundamentally different from the tricky thing with Prisoner's Dilemma and Specialized Labor, which in both cases is simply that people have no selfish incentive to work. So it seems like an important omission. Especially because depending on the exact payoffs, it may be that "one player is a sucker while the other makes out like a bandit" is both a Nash equilibrium and socially optimal.

The thesis of the book is to propose a certain hypothesis. Roughly speaking, and for the purpose of this essay, we can assume the hypothesis says: norms will evolve to maximize the aggregate welfare of the players.

(And so Farmer's Dilemmas might be a good place to look for failures of the hypothesis. When the socially optimal result is for one player to be a sucker, and that's also a Nash equilibrium, the hypothesis thinks this is fine. Humans might not think that, and norms might evolve that the hypothesis would have ruled out. But note that this is only the case in the Discordant Farmer's Dilemma - when there are no gains from cooperation. In the Synergistic Farmer's Dilemma, the socially optimal result is for both players to Work. The Discordant Farmer's Dilemma might be rare in practice - I wouldn't expect it with fence-building, for example.)

Let's pretend we're creating a system of norms for these games. Something we can do is mandate transfers of welfare between players. In each quadrant, we can take some of one player's payoff and give it to the other. Total payoff stays the same, and so the socially optimal outcome stays in the same place. But the distribution of welfare changes, and the Nash equilibria might move.

How do we encourage the socially optimal result by doing this? This is Ellickson's possible minor contribution. He points out that we can do it by introducing a debt from those who Shirk to those who Work, and that the value $ ww_* - ws_1 $ works in both these games.

He calls this the "liquidated-Kantian formula" but doesn't explain the name, and I have only a vague understanding of where he might be going with it. Since the name hasn't caught on, I'm going to propose my own: counterfactual compensation. If I Shirk, I compensate you for your losses compared to the world where I worked.

(To compare: actual compensation would be compensating you for the losses you actually suffered from working, $ ss_* - ws_1 $. Actual restitution would be handing over to you the gains I got from your work, $ sw_* - ss_* $. Counterfactual restitution would be handing over to you the gains I got from not working myself, $ sw_* - ww_* $. Each of these takes one player's payoff in one quadrant, and subtracts the same player's payoff in an adjacent quadrant. Compensation is about your costs, and restitution is about my gains. The actual variants are about differences between the world where no one worked and the worlds where one of us worked; they're about the effects of work that actually happened. The counterfactual variants are about the differences between the worlds where only one of us worked and the world where we both worked; they're about the effects of work that didn't happen.)

(Also: yes, obviously there are caveats to apply when bringing this formula to the real world. Ellickson discusses them briefly. I'm going to ignore them.)

If we apply this formula to the Prisoner's Dilemmma, we get this:

Since $ ww_* > sw_* + ws_* - ww_* $, this puts the incentives in the correct place. The Nash equilibrium is now for both players to Work, which is socially optimal.

(In my taxonomy, depending on whether $ sw_* + ws_* - ww_* ≷ ss_* $, this new game is at the point where The Abundant Commons meets either Cake Eating or Studying For a Test. It's not unique in either case, because there are at most three distinct payout values.)

Specialized Labor is more complicated. There are three ways we might decide to apply counterfactual compensation. We could say that the Shirker compensates the Worker for the Worker's costs, either $ ww_* - ws_1 $ or $ ww_* - ws_2 $ depending on who Worked. Or we could say that the Shirker compensates the Worker for what the efficient Worker's costs would have been, $ ww_* - ws_1 $ regardless of who Worked. Or we could say that the efficient worker never owes anything to the inefficient worker; he gets to just say "sorry, I'm not going to pay you for work I could have done more easily". Lets call these approaches "actual-costs", "efficient-costs" and "substandard-uncompensated"

Ellickson doesn't discuss these options, and I ding him another point for that. He just takes the substandard-uncompensated one. Here's what it looks like.

Player 2 has no incentive to Work, regardless of what Player 1 does, because $ ss_* > ws_2 $ and (unlike in the Prisoner's Dilemma) $ sw_* + ws_1 - ww_* > ww_* $. And given that Player 2 is Shirking, Player 1 has incentive to Work. So again, we've moved the Nash equilibrium to the socially optimal quadrant.

This isn't, like, a mind-shattering result that's going to blow open the field of game theory. But I don't remember seeing it before, and Ellickson doesn't attribute it to anyone else. I'm inclined to give him some credit for it. Even if others have had the insight before - which I expect they have - it seems like he's still doing competent work in a field outside his own. Not amazing work, not particularly difficult work, but competent.

One objection: the inefficient worker gets a better result than the efficient worker. That seems bad to me, because it discourages people from becoming the efficient worker. I don't think this is a big deal, though. For one thing, acquiring skills probably does increase your own payoff; your skills will feed into $ ww_* $, not just your $ ws $. (So it directly increases your payoff in Work/Work, and reduces your debt in Shirk/Work.) Someone else acquiring skills will increase your payoff even more, perhaps, but that's not a big problem. For another thing, such incentives can be handled out-of-game. I do think Ellickson should have acknowledged this issue, and I ding him a point for not doing so. But a brief note would have been fine.

What happens if we apply counterfactual compensation in the other possible ways? The only difference is in the bottom left quadrant, which becomes either $ sw_* + ws_2 - ww_* $, $ ww_* $ (actual-costs) or $ sw_* + ws_1 - ww_* $, $ ww_* + ws_2 - ws_1 $ (efficient-costs). The problem with both of these is that that quadrant might now be a Nash equilibrium. In the first case, Player 1 might prefer that quadrant over Work/Work, depending on $ 2ww_* ≷ sw_* + ws_2 $, and Player 2 will certainly prefer it over Shirk/Shirk. In the second case, Player 1 will certainly prefer that quadrant over Work/Work, and Player 2 might prefer it over Shirk/Shirk, depending on $ ww_* + ws_2 - ws_1 ≷ ss_* $. That's not great, we only want a Nash equilibrium in the socially optimal quadrant.

On the other hand, I note that if $ ws_1 - ws_2 $ is small, then the social cost is low; and if it's large, then (except perhaps with some fairly specific payoff values?) that quadrant isn't a Nash equilibrium. Meanwhile, if payoffs are uncertain - if people might disagree about who the more efficient worker is - then either of the other choices seems more robust. And this is more of an aesthetic judgment, but it feels like the kind of aesthetic judgment that sometimes hints at deeper problems: there's something a bit weird about how substandard-uncompensated is discontinuous. A small change in Player 2's skills lead to a small change in her compensation in each quadrant, until she gets equally skilled as Player 1, at which point there's a large change in the Shirk/Work quadrant.

On the other other hand, a feature of how these games translate to the real world is that players encourage each other to discuss in advance. Someone building unilaterally may not get to claim this debt. So if they disagree about who the efficient worker is, that's unlikely to cause much grief.

What about measures other than counterfactual compensation? Actual compensation ($ ss_* - ws_1 $) doesn't work. If a player expects the other to Shirk, they'd be indifferent to Working; and in a Prisoner's Dilemma, if they expect the other to Work, they might prefer to Work or not depending on $ ww_* ≷ sw_* + ws_1 - ss_* $. (In Specialized Labor, that inequality always resolves as $ < $ which gives the incentives we want.)

Actual restitution ($ sw_* - ss_* $) is sometimes okay in a Prisoner's Dilemma, but if $ ws_* + sw_* < 2ss_* $ then Shirk/Shirk remains a Nash equilibrium; players will only want to Work if they expect the other to also Work. In Specialized Labor it has the problem that players would prefer to Work than to pay restitution, and so Work/Shirk cannot be a Nash equilibrium.

Counterfactual restitution ($ sw_* - ww_* $) has much the same problem in a Prisoner's Dilemma; if $ ws_* + sw_* < ww_* + ss_* $ then Shirk/Shirk is a Nash equilibrium. And in both games, a player who expects the other to Work will be indifferent to Working.

There are other options for payment one might consider; I haven't even looked at all of them of the form "one raw payoff minus another raw payoff". But so far, counterfactual compensation seems like the best option.

(We could even consider values of the debt based on information outside of the original payoff matrix. But Ellickson points out that when deciding how to act in the first place, players will already want to figure out what the payoff matrix looks like. If the debt was based on other information, there'd be a further cost to gather that information.)

While we're here, let's look at the other games implied by Prisoner's Dilemma and Specialized Labor. The Asymmetrical Prisoner's Dilemma (or Synergistic Specialized Labor) has $ ws_1 ≠ ws_2 $ but $ 2ww_* > ws_1 + sw_* $. In this case, counterfactual compensation does exactly what we want it to do, just like in the symmetrical Prisoner's Dilemma; except that substandard-uncompensated is no good, it doesn't give us a Nash equilibrium at all. (Player 1 prefers Shirk/Work to Work/Work, and Work/Shirk to Shirk/Shirk. Player 2 prefers Work/Work to Work/Shirk, and Shirk/Shirk to Shirk/Work.) If Ellickson had considered this game, he'd have had to discuss the possible ways one might apply counterfactual compensation, which would have been good. So I ding him a point for it.

Symmetrical Specialized Labor (or Discordant Prisoner's Dilemma, or Too Many Cooks) has $ ws_1 = ws_2 $ but $ 2ww_* < ws_* + sw_* $. The difficulty here is that there's no way to break the symmetry. Any of the three ways to apply counterfactual compensation will be equivalent, and leave us with two Nash equilibria in the two socially equal quadrants. The "discuss in advance" feature saves us again, I think; players don't need to somehow acausally cooperate to select one to Work and one to Shirk, they can just, like, talk about it. So I think it was basically fine for Ellickson to not consider this game, though it would have been worth a brief note.

How does this work in the Farmer's Dilemma? First we need to clarify exactly what set of games that refers to. In symmetrical games, I think of it as having $ sw_* > ww_* > ws_* > ss_* $; that is, each player would prefer the other to do all the work, or failing that to help; but they'd still rather do it all themselves than for the work not to get done.

I'm going to break symmetry by separating $ ws_1 $ from $ ws_2 $ as before. Without loss of generality, we can specify $ ws_1 > ws_2 $, but I'm not going to decide whether $ ws_2 ≷ ss_* $. It might be that only one player is skilled enough to benefit from Working alone.

So in normal form, the Farmer's Dilemma looks like this:

Either of the top two quadrants could be socially optimal, depending whether the game is synergistic or discordant (that is, whether $ 2ww_* ≷ sw_* + ws_* $). Shirk/Work may or may not be a Nash equilibrium, depending whether $ ws_2 ≷ ss_* $. So how does it look with counterfactual compensation? I'll consider the synergy and discord cases separately.

Oh dear. Substandard-uncompensated compensation is clearly not going to work; Shirk/Work might still be a Nash equilibrium. In Specialized Labor it was fine that the efficient Worker would prefer the inefficient Worker to do all the work, because the inefficient worker would say "nuts to that". In a Farmer's Dilemma she might continue to Work, which we don't want. Even if we specified $ ws_2 < ss_* $, we'd simply have no Nash equilibrium; like in the Asymmetrical Prisoner's Dilemma, one player would always get a better result by changing their move.