2018-08-19: On being on a speedrun marathon run selection committee

As part of staffing Big Bad Game-a-thon, I (among others) had the enviable and/or unenviable task of reviewing all the submissions to pick and choose what we wanted in the marathon. This was a first for me; I'd submitted runs to many marathons, but never been on the other side of the table.

At its core it's a simple task: watch over submission videos and decide if a given run would do well in the marathon or not, then review these feedback snippets and decide what gets in. Of course, the execution therein is a bit more complex: you have a limit to how much content you can fit into the marathon because most marathons have a duration decided before run selection, not a duration decided by run selection. As such, there's going to be a percentage of run-time you have to decline, and that percentage may be more than you would really want to decline.

That was the case with BBG. Submissions came in heavy at the end of our submission period and we ended up needing to decline roughly 50% of the submitted run-time. This is a pretty stark contrast from 2017, where we accepted 97%. As a result, the selection committee was forced to be pretty picky and things that would have not been looked at harshly in a marathon 5 or 10% over duration became really big deals. We declined quite a few good, solid runs unfortunately.

I think part of this is a lot of runners simply don't understand how a run selection committee thinks and does their job. I know I didn't until I actually had the task thrust onto myself. I hope to touch on some of the things I saw here and reveal some of my thought process to hopefully help other runners out.

I should note, BBG is nothing compared to GDQ, ESA, etc. We accepted half our submitted runs; events like GDQ accept less than 10%. We received about 100 hours of submissions, GDQ receives many times that. So take what I say here with a grain of salt and understanding that I'm talking about small marathons; I'm sure the process is entirely different for a GDQ.

Our Selection Process

The process for BBG was pretty simple: once submissions closed, we divvied up the runs and aimed to have three reviewers view and comment on each run. As a first pass, we merely added a few sentences of thoughts and flagged a run as surely in, surely out, or needing a second pass. This alone filled a good percentage of our schedule and I think this is where the "little things" really hurt some submissions.

To get through all the content we had, some reviewers watched runs on 2x speed, watched multiple runs at once, or relied on prior knowledge of a game to hop around the video and check run quality and commentary. So while I wish I could say every second of every video was watched three times over, that's not the case; there was just too much stuff to do that. I am confident, however, that every run was completely reviewed between the three reviewers for each.

We had to be fairly brutal in our vetting, with half the submissions needing to be declined. We actually did a second pass on any staff submitted runs with some enhanced vetting, because we wanted to be sure we could fit more submitters in, rather than just showcase ourselves.

After the first pass was done, the committee met and did three quick passes over the runs like so:

1. Do we want all the "3 Yes" runs? Does anyone object? Any objections on a run bumped it to "Maybe", then the rest were all accepted whole cloth.
2. Are we sure we want to decline all the "3 No" runs? Does anyone object? Any objections on a run bumped it to "Maybe", then the rest were all declined whole cloth.
3. We sorted the "Maybe" list by its score ("Yes" is 1, "No" is -1, "Maybe" is 0) and started discussions and voting at the top.

For each run we had a quick chat on its merits and usually the three reviewers and the marathon admins came to a quick consensus on "Yes", "No", or "Backup/Bonus run", and anyone could object to that consensus if they felt it necessary. At some point in this process our votes became consistently "No" across the board, at which point we did another pass of "Are we ready to decline everything below this line?"

This method actually got us within our marathon time window the first try; but if it hadn't, we would have done another round of discussion in a similar manner: going through either the Backup/Bonus list or the accepted list and deciding if we should demote or promote runs to fit the time window.

The total time of the meeting was about 90 minutes. There wasn't really a need to debate much because most of us were on the same page. The meeting was so short because most of the leg work was done over the past week on each reviewer's own time.

Once we had an accepted and backup list, our next step was to get availability from each runner. There's two ways to do this: get it during submissions, or get it from those who made the cut after. We went with the latter option for BBG2018, which brings the benefit of allowing runners to submit without worry that their availability will damage their chances of getting in. From here, we can make a best effort to fit everyone, but if it doesn't pan out for someone we can promote a backup run in their place.

The Golden Rule for Getting In

Every reviewer is different, but to get a "Yes" from most of us in the initial reviews, we needed to be 100% sure the game, runner, run, commentary, and interaction of all of the above would fit well within the marathon. If any one part was absent, at least one reviewer likely gave a "Maybe" grade and bumped that run from the fast-track addition to the schedule. From there that submission went to the thunderdome of "Maybe" runs and had less of a chance of getting in.

Most of what pushed us to reject runs can be summarized with one rule: your submission should be as close to what you will do and what you want to show off in the marathon run as possible. If you're going to commentate the run, commentate your submission. If you're doing a showcase or non-speedrun submission, do a dry run of it as your submission video. If you say in your submission "I can cut 10min off this", try to cut 10min off it before submissions close. Etc etc.

I understand, especially for a small marathon, that sometimes it's not worth the time and effort until you know for sure you're in; but if you really want to get in, this is your best chance to do so.

That's really it. I'm going to touch on some specific points for the rest of this, but it all boils down to that one golden rule.

The Little Details

1.) Don't just submit your PB video

Seriously. This is rule #1 for a reason. Most PB videos lack commentary and most runners, when they realize they're on PB pace, go serious time. So your showcase tends to be a large chunk of silence if you're one of those types. Do a run where you do commentary as if you were in the marathon; even if you don't come close to your PB, you'll have a showcase of what your run will look like.

Even better: if you ran the game in another marathon, submit that VOD. There's nothing better at showing off how a run looks in a marathan than a video of... how a run looks in a marathon!

It may even be a good idea, if you're really attached to getting a run in a marathon, to cut a submission video when you're actively running the game. It doesn't need to be marathon-specific; just do a run with commentary before you put the game down after getting your PB goal and keep it around just in case.

About 70% of BBG2018 submissions were just PB videos. We declined a LOT of borderline runs because the lack of knowing how commentary would go pushed it over the edge into decline.

2.) If your estimate and comments don't match your video, explain why

One of the more eyebrow-raising things in a submission is a 20min video with a 45min estimate. You need to explain that. Is the run heavy on RNG? If so, what do you plan to do if you come in 25min under estimate? Are you likely to screw something up? Can you improve it if so?

On the other side of the coin; if you estimate 20min and your video is 45min, that's even more disconcerting. Why is your showcase of what you can do horribly over estimate? If you're 100% sure you can optimize later, note that, but keep in mind this isn't likely to make your run look good on selection day. Known quantities are good, "I can fix it later" makes reviewers nervous.

3.) Your comments are the first thing a reviewer sees, and will color their perception

You have a tweet-length (usually) blurb in which you can try to sell the reviewers on your run, don't spend it talking down on yourself or saying things like "It's cool if you reject this". If someone talked smack about their own run in the submission comments, I tended to believe them.

The best use of this is to explain why the run is awesome and try to draw attention to the best parts of your submission video. If you say "The skip in level 3 is really awesome too", the reviewer will definitely be paying attention and will remember the cool thing you did when it comes time to vote on the run's acceptance.

That said, don't lie. It's better to say "I won't have time to derust this fully" than to be deceptive about that and put on a bad run.

4.) If your video quality is bad, we'll assume it'll be bad on marathon day (for online marathons)

If your submission video has jacked audio levels, we can fix that in pre-run setup. However if your mic sounds like you're streaming live from your toilet bowl, your bitrate is negative two, or you drop more frames than you stream, we're going to assume this is a chronic problem and will show up in the marathon run. Even pointing out "My net isn't usually this bad but..." is worrisome because, if that was temporary, why didn't you just record your submission another day or do local recording?

You don't need perfect crystal clear 4k video with a \$5,000 mic and mixer or anything, but if your submission isn't something you'd want to see in a marathon quality-wise, the reviewers probably don't either.

5.) You get no bonus points for submitting early

This is one I actually need to work on. I have this need to be the first person to submit to a marathon I'm really excited about. Don't though unless you have submission videos from prior events just ready to go. Take some time to stew on it, make sure your videos are good, you're submitting games you're ready to derust and commit to doing well in the marathon, etc.

No marathon is likely to ever close submissions early; you can submit on the last day and it's just as good as submitting on the first.

In future BBGs I'm actually going to lobby for randomizing the submission list before reviewers are turned loose on the vetting process; just to be sure early submitters don't get some kind of subconscious benefit.

6.) If you want a run in, submit it. Really. Don't worry about it

A few people contacted me, or came into general chat, after the submission period ended and talked about how they wished they could have submitted game X but didn't feel it'd get in and didn't want to waste our time. No really, if you think it fits, submit it.

It's our job to vet your submissions. It's your job to make a case for them; go for it. At worst we reject your submission out of hand for being wildly off marathon topic. It's fine. Marathons largely gauge their success by how many submissions they get. It makes us feel good to have a ton of content to choose from!

Also, you don't know what the marathon staff is planning. Maybe you acidentally fit the bill for a feature they were considering. Maybe you'll have them invent a feature for you.

Obviously don't go submitting Super Mario World to a kusoge marathon, but don't be shy if you think you got something that fits.

As noted, this all boils down to giving time and care to your submission video, and being unafraid to sell yourself and your run.

tags: personal, speedrunning, big_bad_gameathon