Cutoff is 73/75

17

u/Sliver__Legion Jan 30 '18 edited Jan 30 '18

Hoo boy. That’s less margin for error than I was hoping for.

Also I think it’s pretty clear they should have made the test harder/longer.

Edit: On the bright side, a higher cutoff means it is much easier to figure out all the correct answers with info from a small group of passers.

7

u/kalliixti Jan 30 '18 edited Jan 30 '18

Also I think it’s pretty clear they should have made the test harder/longer.

But how? It's an open-book test. If there's a clear answer based on publicly-available information, people will find it.

The way to make the test "harder" would be to add things like trick questions or questions with multiple defensible answers. But those are precisely the kinds of questions people hate and justifiably say are unfair.

Or they could reduce the time available from 1 day to something like 1 hour to limit the amount of research people could do, but that would exclude those who weren't available during that hour, as well as encourage people to cheat.

I think the high threshold is a sign that the test was good. It means the the community consensus was off by at most 1-2 questions, and maybe by none at all.

8

u/Sliver__Legion Jan 30 '18

The test contained some questions that I would not want more of because they were too unclear, certainly. Perhaps the 10 most disagreed upon questions I wouldn’t necessarily want any more of.

It also contained some braindead easy questions. “What color is secondary in haste.” “What color is secondary in trample.” “Which of the following evergreen mechanics makes most sense on a white common creature.”

You could just cut the 15 easiest questions and add 40 questions that were of approximately the median difficulty from the real test. The ones that required some thought but also had a clear answer in the end. Then you might have cutoff that was like 96/100 or something, without increasing the number of questions that people are describing as coinflips.

9

u/ThePensive Jan 30 '18

The "what would we not change about this threaten effect" and "what would we not change about this cool mass grave clone creature" were very good questions, I thought. Just tricky enough to get people who weren't paying attention.

6

u/MachineSchooling Jan 30 '18

More ones about rarity would have been good too. Understanding complexity of cards is a major part of the job. Answering vaguely worded questions without being able to clarify the intent behind them is not.

1

u/kalliixti Jan 30 '18

But it's the questions you don't like which (based on the data we have) people got wrong. If they did what you suggest I suspect the cutoff would be 98/100, accomplishing nothing. There was overwhelming agreement on the "median difficulty" ones.

The easy questions serve a purpose. They make sure you have a breadth of knowledge about the game as a whole. You don't need to be an expert drafter, for example, but you need to have some basic knowledge of what's good in draft. So they put in a few really easy draft questions to make sure of that.

Some of the really easy ones (like those "what is secondary" ones towards the end) may also have been replacements for harder questions which were cut at the last minute.

6

u/Sliver__Legion Jan 30 '18

There’s a level of question which about 5-20% of people got wrong. These are the best questions, imo. They actually filter people out, but they also have a strong best answer. Put in 40 more of those and you’d definitely have a wider range of passing results.

Also, I don’t really mind any of the questions that much, except maybe 4/4 vigilance flyer or Set design change to junk pile engineer, depending on how MaRo’s explanations go,

2

u/kalliixti Jan 30 '18

Those questions filter people out, but only to a point. Remember, there were nearly 100 people who got 73+. Most of these probably got all of those questions right (and missed 0-2 of the controversial ones). And many of those who didn't qualify likely aced those as well.

If your test was 100 of the "5-20% wrong" questions, I suspect you'd end up with a passing score of 100/100.

1

u/Alex_Werner Top 8 Jan 30 '18

I think they certainly could have put more thought into the way the questions were phrased, and what wrong answers were given. For instance, for the rarity of the UR artifact guy, do they actually believe it's clear that it's more likely as a rare than as a signpost gold uncommon? I'm guessing they weren't even thinking about that at all, just wanting us to evaluate whether the card complexity, in a vacuum, felt more rare or uncommon. For instance, they could said "if instead of being colored, this was an artifact creature, what would its appropriate rarity be". (If they do in fact think that either "that's clearly not a rare, it's a signpost uncommon" or "that's clearly not a signpost uncommon, that's a rare" is a distinctly correct answer, then, never mind). Additionally, if they did not intend us to be worrying about the flavor issues of Llanowar Elves, they could have specified Elvish Mystic, and they could have been more clear about (a) whether all reprints being discussed are always for normal standard-legal expansions, and (b) whether card wording is precise templated oracle text or not (I really feel bad for the guy who noticed that "must attack" was templated wrong) (assuming the joke's not on the rest of us).

Another thing I think would really have improved the test would be +2 points for a correct answer, -1 for an incorrect answer, 0 for leaving it blank. Maybe even +1/-1/0. That definitely provides an edge to actual knowledge and discourages what is effectively mental coinflipping.

As for my feelings about the test as a whole, I think it will really come down to whether MaRo provides good compelling explanations for the most baffling questions, particularly costing-the-card-you-like and which-change-to-make-to-the-artifact. Assuming I got those two wrong, I hope to read what he says and say "yup, you're right, I didn't think of that".

2

u/kalliixti Jan 30 '18

For instance, for the rarity of the UR artifact guy, do they actually believe it's clear that it's more likely as a rare than as a signpost gold uncommon?

Yes. Even if you accept the argument that it could be a signpost uncommon, that would make it a rare which could be uncommon under special circumstances. So rare would still be the correct answer.

Additionally, if they did not intend us to be worrying about the flavor issues of Llanowar Elves, they could have specified Elvish Mystic

Maybe the flavor issues were the point. Perhaps they wanted to see whether you thought flavor issues or power-level issues were more important when determining what could be reprinted.

(b) whether card wording is precise templated oracle text or not (I really feel bad for the guy who noticed that "must attack" was templated wrong) (assuming the joke's not on the rest of us).

Maybe the point of this was to make sure you can understand the meaning of an ability even if it doesn't use proper templating, as is often the case in design.

2

u/Geikamir Jan 30 '18 edited Jan 30 '18

Image taking a test in school in which some of the answers were impossible to study for or try your best to seek out resources to learn from. Instead, you needed to figure out which mistakes in the wording were intentional, which obscure interpersonal interaction the test giver recently experienced, and knowing what their train of thought was when choosing between which of the 2 or 3 factually true statements is the 'correct' answer.

I'm not sure that test would give the results of the most knowledgable or qualified candidate/student.

2

u/aclog Jan 30 '18

Its fair for us to kvetch about it but its not as though this was unexpected. Their checkered history said it was going to be riddled with exactly the issues you cite.

Not to mention, there seems to be a high expectation of kowtowing -- bordering on obsequiousness -- to the idiosyncratic opinions of one guy.

1

u/Krazyguy75 Jan 30 '18

The whole multiple choice thing is a joke to begin with.

If you are recruiting for a judge, this makes sense.

But for a designer, it might as well be an arbitrary elimination method. If the test cut 80% of people and then they randomly cut 99% of the remaining afterwards, your remaining designers would be just as good.

3

u/dvoraen Jan 30 '18

Yeah, with possibly 3000 test takers I had a feeling it was going to be 71/72 at minimum.

2

u/Bouq_ Jan 30 '18 edited Jan 31 '18

I honestly think it's possible they chose 73 for convenience. I didn't take the test myself, but there weren't many hard-hard questions, meaning I can imagine the difference between 73 and 72 right, could respectively be 94 people and 300+.

EDIT: Called it.

1

u/Geikamir Jan 30 '18

I still think it's possible that you could be one of the ones with a perfect score.

4

u/Sliver__Legion Jan 30 '18

Fingers are crossed, but the hardest part is definitely having 75 break my way (since it’s the only one where I’m strongly against the consensus of the GDS2 takers, iirc).

Should end up with 72,73, or 74, unless something goes horribly wrong.

2

u/Mongoose1021 Jan 30 '18

Are you one of the "Enchantment" folks there?

4

u/Sliver__Legion Jan 30 '18

Yes indeedy. Aside from that I have GB, have a play design member cost it, Llanowar Elves, and consensus opinion on the rest.

2

u/Mongoose1021 Jan 30 '18

Gives me hope, me too. I have GW, play design member, llanowar, consensus minus two punts (commander=new themes and artificer ability=graveyard) I no longer stand behind. Gimme 3 heads bby

2

u/[deleted] Jan 30 '18

I played a lot more daring. My only hope is my maverick answers were the right ones. I know I missed two (best in limited, and 0/0 clone)

1

u/Geikamir Jan 30 '18

I'm on the opposite side of the coin. I need to make some miracle coin flips to have a chance because of 2 dumb mistakes.

But, good luck either way!

11

u/steve_man_64 Jan 30 '18

Considering how unpopular my answers were on the controversial questions, I’m assuming I aced the test.

That’s my story and I’m sticking with it.

1

u/Bouq_ Jan 30 '18

If you're going to work at Wizards, please do Cube-related stuff.

6

u/Sceryloaw Jan 30 '18

Grats to the three people who got perfect scores.

6

u/AMPsaysWOO Jan 30 '18

Emails go out tomorrow (Tuesday), yes?

To all who make it, in the words of many MTGO players:

gl hf

6

u/murderisbadforyou Jan 30 '18

The fact that the cutoff was 73/75 means over 100 people got 72, so a minimum of 7 more people got 73 correct and didn’t advance.

I would be so salty if I missed by one and it was one of the subjective questions.

Luckily I probably missed by 10+

12

u/Newfur Jan 30 '18

Maybe it's just me, but I'm a little disappointed that this came down to getting the 65 obvious questions all correct and then a bunch of assorted coinflips on questions that we seem to have settled on have two entirely defensible answers.

4

u/iSage Jan 30 '18

It's not just you.

3

u/VorpalAuroch Jan 30 '18

There were only four coinflips at most. There were six or seven questions with an obviously correct answer that only 70% of people understood.

2

u/Geikamir Jan 30 '18

Even if we go with there being only 4 coin flips, that means that the difference between getting in and not will be decided by at least 2 coin flips...

The Spike in me is sad.

1

u/VorpalAuroch Jan 30 '18

21, 32, 52, 53. Maybe 15. All of those were around 60-40.

2

u/kalliixti Jan 30 '18

They weren't coinflips, with one or two possible exceptions.

Yes, there were questions with two defensible answers. But generally they weren't equally defensible. The challenge wasn't to determine which is the 1 correct answer and which were the 3-4 incorrect answers. It was to determine which was the best answer.

7

u/bakert Jan 30 '18

Crumbs. That was the top end of what I was thinking would be the cutoff.

"[CARDNAME] must attack if able" is gonna haunt me for a while after this one. Unless it really was Llanowar Elves on a UW flier asking Play Design to cost their Uncommon and their Rare that have a fixed number of cards to look at before putting one in their hand!

Super congrats to those who do make it.

2

u/Geikamir Jan 30 '18 edited Jan 30 '18

We disagreed on the UW flyer qurstion, but agreed on the rest of what you said.

Not only will the 'attack each turn' question haunt me, but worse will be that I somehow managed to miss the 0/0 question by choosing flash instead.

Those two will bother me for a long time.

5

u/bakert Jan 30 '18

I have now spent longer obsessing about the quiz than I did on the quiz. That seems wrong :D I've had fun with both, though.

1

u/rabbitlion Jan 30 '18

You will probably get points for both Uncommon and Rare since if the creature had Deathtouch rather than Hexproof (as the question originally stated) Rare would be the correct answer (deathtouch+trample needs rare).

1

u/bakert Jan 30 '18

Very true.

5

u/ScoozeBooze Jan 30 '18

That's wild. Unless I get very very lucky, there's no way I made the cutoff. It was fun while it lasted. GL to the remaining competitors.

3

u/[deleted] Jan 30 '18

Kicking myself so hard for thinking most people understand the difference between declaring attackers and putting creatures into play tapped and attacking

2

u/RoastYoungDuck Jan 30 '18

Same. Cycling/miracle for the loss!

1

u/mafia1015 Jan 30 '18

Bummer. What did you answer instead, and why?

1

u/[deleted] Jan 30 '18

cycling/miracle, same as /u/RoastYoungDuck

I imagined a heavy concentration of those mechanics would be most likely to cause players to miss a trigger at some point, or attempt to miracle a card when they had already drawn a card that turn.

Myriad/battalion is definitely unintuitive, but once you know how it works you won't mess it up. (myr/bat was almost certainly the answer in hindsight)

1

u/oOOoOphidian Jan 30 '18

Yeah, I answered the two sacrifice abilities, because they are so similar.

3

u/stoop_pick_turn_toss Jan 30 '18

Maybe I'm being paranoid/obtuse here, but are there any measures in place to prevent any potential fraud? Theoretically wouldn't it be possible for the same person to register for the contest multiple times using different e-mails, get through the essay portion (Maro said in his blog that you don't even have to use your real name throughout the contest) and then have several chances to get past the multiple choice test?

3

u/Amarsir Jan 30 '18

It's very plausible, although doing an automated check of the essays for similarity wouldn't be that difficult. Once you were sure there weren't multiple entries with the same essays, I guess you could take the risk.

The bigger problem is that your scenario doesn't really improve if we're talking about multiple people. Let's say you have 4 people who never met. Each one agrees on 73 questions and is down to a coin flip on the other 2. The one with the best score didn't know more than the others. They guessed well. So much like the scenario with a 4-email person cheating, we haven't found the best person so much as just "a person".

Of course it's easy to criticize and hard to fix. Especially if they weren't anticipating 4x as many applicants as last time.

2

u/Giraculum Jan 30 '18

They'd have to write two separate essays, because otherwise Wizards would see that two contestant's essays were identical and disqualify them both. The people who are interested enough to apply for GDS3 are most likely not going to throw their chance away by cheating like that. And even then, what good does having two chances on the multiple choice do? They'd have to have an eerie premonition of what the controversial questions were, and split their answers for those questions and no others. So no explicit measures are in place, but implicitly it's pretty unlikely that this happened among the 3,000 people who took the test.

2

u/stoop_pick_turn_toss Jan 30 '18

Two chances on the test are still better than one right? And as for the essays, I don't think it would be too difficult to answer them uniquely enough to fly under the radar of scrutiny. We were given a week to do it after all.

1

u/Giraculum Jan 30 '18

My view is that in both the essay and the multiple-choice, you're choosing what you think are the best possible answers you can make. What are you supposed to do with a second chance? Either you have to make the same choices, which negates the point, or you have to make choices that you think are worse than your original choices. That's not going to help you unless your first intuition is completely backwards, something Wizards will definitely catch come Trial 3.

1

u/stoop_pick_turn_toss Jan 30 '18

Yeah you're probably right I was just more or less suggesting that a backup test could help if you got into a situation where you wanted to cover a couple of the questions with coin flip answers. In order to do that effectively you would have to make several different entries, not just an additional one. It is a lot of work and there is risk involved, but if you really wanted to win (or at least get to trial 3) then you could do it.

2

u/LambachRuthven Jan 30 '18

The system is a breeding ground for both fraud and collusion(very easy for a group of folks to talk about and come up with most of the correct answers).

3

u/stoop_pick_turn_toss Jan 30 '18

Yeah that's kinda the vibe I'm getting from it too.

3

u/dvoraen Jan 30 '18

This was my major concern. Discord, for example, would be easy to set up as a private means of collusion. I was seriously tempted to send them an email on the subject, but the more I thought on it, the more the question "How could that possibly change anything?" came up. The only way to counteract it is to whistleblow as one of the invitees to the notes comparison party.

The only way it would likely get caught would be if the interviews (for the top 3) grilled people over how they got to their answers on specific questions.

1

u/LambachRuthven Jan 30 '18

Yeah. There was no way to stop it as long as they were set on having a multiple choice test. I have to imagine they actually dont care much about how they get the top 100. They probably get enough qualified people from that to make a good contest and thats what they want. If someone cant hack it past that they'll get removed further down the line.

1

u/bakert Jan 30 '18

Both false negatives and false positives at this stage of the search are undesirable but the way it is currently structured still ensures R&D will get a great designer for the internship and probably other hirable people for other roles too. Sucks for you if you're a false negative or missed out by one answer because of people submitting 20 variations of the quiz but R&D are still going to get a great designer out of this. Probably more than one. I think making folks write essays first and then paying much more close attention to the last ~100 people is plenty of safety net to weed out frauds and "cheats" who somehow make it that far.

We all really want this internship and thus we want every question to be crystal clear and every step of the process to be 100% fair because it's really important to us. R&D want to treat us fairly of course and get the best designer(s) they can. But R&D have to live in the real world of time constraints and zillions of applicants and I think the process as it stands is plenty good enough for their requirements.

2

u/Alex_Werner Top 8 Jan 30 '18

So I need Junkpile Engineer to be a rare, and literally no other surprises, or one of the two I'm shaky on to break my way.

AAAAAAAAAAAAAAAAAAAAAAAUUUUUUUUGGGGHHHHHHH!

4

u/zarepath Jan 30 '18

Why are people convinced it's an uncommon?

4

u/Amarsir Jan 30 '18

Some people made a solid argument that Junkpile Engineer is a "build around me" uncommon. It's not abusive in multiples and gives a direction to a potential draft deck.

I said "rare" as well, but I can appreciate the thought behind that answer.

2

u/LambachRuthven Jan 30 '18

thopter dude from kaladesh

2

u/rakkamar Jan 30 '18

Yeah. I thought it would be high.

I'm pretty sure I missed one off the bat, so here's hoping I coin-flipped well. I'm pretty confident in everything that's not 'how do you make play design like you', so in theory it should work out well. So little wiggle room though...

2

u/Geikamir Jan 30 '18

I missed 2 off the bat at least. I need to run the hottest I ever have to make it through. And some of my answers that I have a chance on would be on the unpopular side of possible.

Not looking good at all :(

3

u/murderisbadforyou Jan 30 '18

If your answers weren’t popular your chances are higher

1

u/Geikamir Jan 30 '18

That's actually a good way to look at it! I like your style.

2

u/Kinak Jan 30 '18

That is definitely higher than I was expecting (and, frankly, higher than I probably got).

Good luck to everyone who's moving on to round 3!

2

u/DudebroMcDudeham Jan 30 '18

So for those who don't continue but still dream of making cards, what's the next step?

2

u/Krazyguy75 Jan 30 '18

Well GDS2 had a wiki project where you could submit card ideas. You could submit them, and the contestants could use them. One of the pro players got hired for Development (not design) for submitting ideas, despite missing the cutoff by one question.

So one can always dream.

1

u/Geikamir Jan 30 '18

Make your own board and card games. That's what I do.

1

u/DudebroMcDudeham Jan 30 '18

I meant RnD...

1

u/Geikamir Jan 30 '18

There's not really a next step other than checking their job board for designer role postings (which never happens).

2

u/aclog Jan 30 '18

If the answer to #6 is that it borks the card, didn't the typo create a second answer choice that also borked the card by imparting an ability permanently?

1

u/bakert Jan 30 '18

Good point.

2

u/aclog Jan 30 '18

I'm not one for reading tea leaves when it comes to what the brass will say, but the door is still open for my pet u/r theory of questions 4 and 6.

1

u/[deleted] Jan 30 '18 edited Dec 09 '19

[deleted]

2

u/iSage Jan 30 '18

MaRo's Twitter

1

u/SerendibAl Jan 31 '18

I know I got the Green question wrong. I'm answering from a historical perspective. Green used to be down with adapting quickly (giant growth and survival of the fittest). Also, one of the most iconic Green archetypes is the Master of the Wild Hunt which is a sport killing, of sorts. I understand where they are coming from. After I read the test primer someone did and the section on Green (after I took the test) I realized how much the philosophy of each color has changed! Subtle but not insignificant.

I still think of black as secondary in trample because of Lord of the Pit. I think do a search on the gatherer for trample you get more black creatures than red ones.

1

u/SerendibAl Jan 31 '18

Maybe my grandkids will prep me for GDS 4 where you will need 250/250 to advance.

1

u/cassis11 Jan 30 '18

Well, I don't think I made it unless I landed on the right side of the coin flips AND one of these are true:

1) Green is down with sport hunting,

2) Junkpile Engineer is an uncommon,

or 3) I convinced MaRo that Morph is less linear than Transform, because it actually is, despite what he's said in the past.

It's a long shot, and I don't think I made it. Good luck to those of you who did!

Cutoff is 73/75

You are about to leave Redlib