You know Advent of Code, right? Of course you do. If you don’t, go check it out right now, you’re in for a treat.
I love Advent of Code. The only imperfection in Advent of Code is the puzzle unlocking’s scheduling, as I haven’t quite gotten around to relocating to accommodate for it.
Here are the key factors I think make Advent of Code the monument it is:
- puzzle quality
- the two-part thing
- language agnosticity
- puzzle quality being so reliably spread out across the puzzle base
For a comparison, Google Code Jam used to have all of that going for it too, only lacking a bit on the “evenly spread out” bit. But now that it’s lost language agnosticity, it’s not exactly horrific, but I don’t exactly love it anymore either.
But back to Advent of Code.
Its creator Eric Wastl was a speaker at FOSDEM 2021. I recommend1 watching his entire talk, but I’d like to highlight a very specific slide, as it’s got much to share with the problem set we commonly discuss around here.
The slide is titled “How do you make a good puzzle?”
- Avoid ambiguity
- Confusing sentence structure
- Multiple interpretations
- Avoid expectations of outside information
- The user may not have taken a data structures course (or any CS course)
- The user might not have the same domain knowledge as you
- Avoid requiring the user to make assumptions
- repeat and highlight important details
- for every sentence, there is a user that skipped only that sentence
- Have other people test the puzzle
There’s so much to take away there. (and so much more if you actually listen to the corresponding part of the talk)
Avoid ambiguity. It seems obvious. Nobody would write an ambiguous statement on purpose, right? Well, it’s not that simple. As Eric goes on, simply using pronouns, like the infamous “it”, can easily result in multiple interpretations.
So I’d extend that to: write simply structured sentences. Have an actual term available for any concept you’d be tempted to “it” out. An unambiguous one, please.
Avoid expectations of outside information. I wish it were obvious to everybody. Yes, I’m looking at you, JPS.
Eric mentions the user may not have a CS background. I’m tempted to think CG may2 differ in that matter. Where it definitely doesn’t is for the next item: the user might not have the same domain knowledge as you.
The user doesn’t know what a composite number is. The user doesn’t know a pawn can capture without colliding. The user doesn’t know how tennis scoring works.3
Define, define, define. If it makes your puzzle too long, maybe your puzzle isn’t that good. Simplify. Extract an easier puzzle out of it.
I’ll add a word about pictures. As in: in most cases, you don’t have them. Many were tempted to link to one externally. My usual argument against that used to be that if a picture is necessary, your concept isn’t clear to begin with. Add words. Better words. “Just look” doesn’t cut it, if you weren’t able to express it with words, chances are, it’s still going to be ambiguous with an image, just somewhere you hadn’t thought of.
And there’s an even better reason later down the talk. Accessibility. Some users are visually-impaired, or even straight blind. They don’t have images.
An image is allowed to help. It’s not allowed to be the statement.
Avoid requiring the user to make assumptions. This item I’d say is mostly for clash and contests, though it never hurts to keep in mind. For the common puzzle, I’d refine it by adding “when going from test to validator”.
Have others test the puzzle. It’s ok if you din’t get it right on the first try. Nobody does. Clarity is a learned skill. You can practice for it. The catch is, you can’t practice for it alone. How would you know you’re getting better at communicating your ideas to others if you don’t have others?
Don’t dismiss rude comments. Well, ok, dismiss them, but be able to read the actual content inside of them. The commenter got it all wrong, the actual explanation is in the statement. Yet the statement failed to make itself understood on first reading. Sometimes because of too many assumptions. Sometimes because of too many pronouns. Sometimes because of not enough assumptions. Sometimes because you actually forgot something.
That last point of Eric’s makes for a nice recap. Your puzzle’s not ambiguous to you. Of course not. You even wrote a solver for it, it’s that good. Nope, it’s ambiguous in multiple places you didn’t even consider as “places”, and you’re not seeing them because your solver is overspecialized. Have someone read it.
Your puzzle’s self-contained to you, you just wrote it as a single piece. No it isn’t, it makes use of many concepts you’re familiar with because you think everybody is. But they aren’t. They’re from a different culture. They write dates differently, spell some words another way. Nobody agrees whether zero is positive, or greater than itself. Be inclusive. Have others read it.
You’ll have noticed all items are “avoid”s, not “don’t”s.
A good puzzle is a work of balance. Any rule can be bent, if there’s a good reason to. The sole acceptable good reason being: it makes for a better puzzle.
For example, being fully unambiguous is often at odds with being simple. As I’ve said before, for puzzles I prioritize non-repetitive clarity, but for clashes a certain amount of additional non-ambiguousness might be in order. Awareness is key.
Later in the talk, a few other points come up that I’d like to highlight. Paraphrased:
- “Making a puzzle everybody loves is impossible.” So settle for making good puzzles, and make them varied. This way, everybody will like at least one of them.
- “Good puzzles are language-agnostic.” So be able to solve them with multiple classes of languages. This is trickier than it looks, as the platform appears to treat them differently. Have the tests cut off the desired complexity, and ensure it’s feasible in slower languages. Never have the validation cut off more strictly than tests.
- “Inputs are pregenerated and filtered for interestingness.” Nothing to add, really. And sadly.
- “I’m the sole author, I want no copyright issues.” I wish that were the case in our queue…
Thank you Eric for prompting me to write up at least some of my thoughts on the topic!
Naturally, I’m fully confident and trusting of the community to take heed of any statement guidelines to write good puzzles; I have no doubt every single one of the 113 in the queue right now is high quality, the result of a long and fruitful reflection and not at all an attempt to grab who-knows-how-much XP from the quest and contribution, which is why I still race to have the opportunity to moderate each and every one of them.