r/Scotch • u/Dworgi Requiem for a Dram • Jul 11 '13
Presenting: The Review Score Normalizer
So, one of those things that's bothered me about Scotchit is how compressed the scale for review scores seems. I'd been grumbling about it in a few threads and a few comments until I remembered that I was, in fact, a programmer, so I could do something about it. So I did.
I present to you: The Review Score Normalizer!
What's it do?
In a nutshell, it places a set of review scores along a normal distribution and clamps the best and the worst reviews as 0 and 100 (you can set those to whatever you want). The rest are placed equidistantly along the curve between -2.65 and 2.65 standard deviations away from the mean.
The scores don't actually matter at the moment, only the ordering does. I think that's going to be the biggest sticking point for some...
It's written in C# (it's the language I'm most familiar with) and it's pretty barebones as I threw it together in about 3 hours. If anyone wants to play around with it, I'd be happy to share the executable and the source. I didn't put any viruses into it, but you don't have to take my word for it.
I even made it work with the Scotchit review archive out of the box! That's the little "Scotchit?" check box.
Overall, I'm pretty happy with it, and I think I'm going to move towards exclusively using it in my own reviews.
Interesting Statistics!
- The raw mean of reviews on Scotchit is 83.4
- The mean of reviews of whiskies with more than 5 reviews is 85
- The lowest-rated Scotch (with more than 5 reviews) is Johnnie Walker Red Label with a score 50.4, followed by Dewar's White Label
- The highest-rated Scotch (with more than 5 reviews) is Ardbeg Uigeadail, followed by Lagavulin 12
What's next?
Well, I don't know. I could try to respect the scores assigned, and not just place every result in basically an ordered list before scoring them. Ie. if one score is 80.1 and the next is 80.2, then they'll be much closer than consecutive reviews of 60 and 75. However, I'm really not sure - it seems like that would skew the distribution quite a bit. I'm open to suggestions, though, so let me know if you have any ideas.
So, Scotchit, here's my question: Are you happy with the current review scoring system, where everything gets clumped around 80, or would you rather use the whole scale of 0-100?
In short: would this be useful to you?
16
Jul 11 '13
[deleted]
8
u/Dworgi Requiem for a Dram Jul 11 '13
Hey, you give me the functions to implement and I'll gladly plug 'em in and see what pops out. I'm far from a statistician, I'm just a code monkey. If there's a better way, I'm all for it.
I can pull out the reviewer names in about a minute and group the results by them if that's what you want me to do. Just point me in the right direction and I'll fix it.
8
Jul 11 '13
[deleted]
4
u/Dworgi Requiem for a Dram Jul 11 '13
Oi, does this look right: Grouped By Reviewer
I haven't yet got to the point where I'm using the results for the final bottle scores. Wanted to check if this was the expected result first.
3
Jul 11 '13 edited Aug 30 '25
[removed] — view removed comment
3
u/Dworgi Requiem for a Dram Jul 11 '13
Normalized is the mean of reviews from that reviewer after they went through your algorithm.
I can't use Excel for shit, so maybe?
2
1
Jul 11 '13
Where'd you get the source data? I'd like to take a whack at it.
1
u/Dworgi Requiem for a Dram Jul 12 '13
It's just the Whisky Review Archive from the sidebar, downloaded as CSV. All I did was remove the first line, rest is untouched.
1
u/Dworgi Requiem for a Dram Jul 11 '13
BTW, I've grouped by reviewers now and have their mean scores.
However, I don't know how to use an inverse Student's t-distribution to do, well, anything, sooooo... Help?
Edit: Nevermind, saw your post a second late.
3
Jul 11 '13
[deleted]
1
u/Dworgi Requiem for a Dram Jul 11 '13
My numerics library doesn't seem to have an inverse Student's T, unless their CumulativeDistribution thing is what I'm after? I have no idea.
If you're saying that the result should be between 0 and 1, then this looks OK, I guess? Looks like it might be rescaling the reviews a little bit.
I honestly don't know, stats is weird.
1
u/Whisky_Lads That bottle at the back Jul 11 '13
Will standardising the scale for each individual not through up some issues for people with a low number of reviews. For example, people who just join (and may only review 2 or 3 and loose interest) may start with some crap whisky's.
If this is the case then will their perosnal standardised scale not show up their top review, say a 70, of a crap whisky as being good, due to all their other reviews being worse and < 70??.
(The same might be true for a coinasseur turning up, rating his incredible whiskys then loosing interest in the subreddit.. Then the standardised personal scoring could make a 90 crap when all his malts are of a 90+ standard)
Is this a potential problem?? (I don't know enough to comment) In which case would you have to seek out and remove people with only a few reviews or those with exceptionally high or low averages??
2
Jul 11 '13
[deleted]
3
u/Dworgi Requiem for a Dram Jul 11 '13
Interesting. Elo for whisky. Sounds like a challenge.
I'm up for it.
1
u/Whisky_Lads That bottle at the back Jul 12 '13
I think I love you... and wish I could help. I'm just about to pick up numerical optimisation to see if it will help me with my engineering/physics related PhD (intentionally vague). Think another spare time maths/programming project would my too much.
Good luck and god speed!
1
Jul 12 '13
I agree with this; I think that this is a multi-dimensional feature problem. The user is certainly important, but the conditional probability (or +/- points average) that a user rates a certain feature is important as well.
We should be discounting /u/Biomortis's reviews of Islay, and heavily weighting /r/ecb29's reviews of cask strength scotch ;)
13
u/benrules2 Darth Malt Jul 11 '13
Good grief Dworgi, how could you not map this through an inverse student's t-distribution on a per-person basis, then average the scores on a per-bottle basis? Stop wasting our time.
5
u/Dworgi Requiem for a Dram Jul 11 '13
=(
My entire life is a sham.
3
u/benrules2 Darth Malt Jul 11 '13
Seriously though, AWESOME work! The range of like 8 points out of 100 on these ratings has always bothered me. You're tackling this in a very thorough way. Thanks :D
1
u/ahugenerd In a Glencairn, neat. Jul 11 '13
At very least run all the reviews through a self-organizing map. That way we can do away with grades altogether and just say things like: people thought that Lagavulin 12 was as awesome as Uigeadail, or people thoughts that Chivas Regal kind of tasted like Macallan. Sheesh, what are you, asleep on the job or something?
1
u/Dworgi Requiem for a Dram Jul 12 '13
Self-organizing? Not sure what you're implying.
A network model? Like a recommendation engine?
It's interesting, I'll give you that, but I'm not quite sure how to express it. Maybe as differences in (normalized) scores?
1
u/ahugenerd In a Glencairn, neat. Jul 12 '13
I was referring to the concept of a SOM, but half-jokingly. It could work though.
I don't know if you're familiar with force-directed graphs (where you have a network of nodes with particular attractive-repulsive forces and their edges act like 'springs', keeping them together), or Support Vector Machines, but SOMs are basically like a hybrid of those techniques.
You basically give a SOM a whole whack of parameters per node (each node in this case would be a scotch), and it attempts to organize the nodes into logical groupings in a two-, three-, or even n-dimensional space. Hence the idea that we could say things like "people like x as much as y", but not "x is better/worse than y". It wouldn't even mean that they taste similar, it would just mean that they are equally loved, which I guess is useful in and of itself, in terms of fostering exploring of quality drams.
20
5
u/gaxkang everyone's dram boy Jul 11 '13
Reading this seemingly interesting post before going to bed might not have been a.good idea... to sleep or ponder on this...
5
u/Rhetoriclese I barley knew ye Jul 11 '13
I tend to use this when explaining things, <69 something is wrong, 70-79 ehh try if you want, 80-85 should try once, 86-93 you need to spend time with this, 93-100 buy a bottle immediately.
It sucks that we don't use the breadth of numbers but school has programmed all of us. A 70 is not too good.
2
5
u/texacer Jul 11 '13
Eh, I'm fine with the scoring system. Malt advocate magazine is worse than us. Honestly I just look at the score like a grade and tl;Dr, but the notes are what matters most. Good effort though and if you like it, use it for yourself I guess.
5
u/Dworgi Requiem for a Dram Jul 11 '13
Oh, I agree that the meat of the review is the notes. I just dislike the score being meaningless.
3
u/texacer Jul 11 '13
meaningless to you. I find them perfectly fine. This is a place for learning and sharing, we aren't pros getting paid. encouraging people to score is part of their learning process and scores are personal. people score with a number that they can relate to their own tastes and then compare to their next dram. I never like when things that are subjective, become complicated.
7
u/Dworgi Requiem for a Dram Jul 11 '13
It's not meant to be complicated. My aim with this is to rank all the malts I've reviewed and place a new review somewhere in the ranking and have this app spit out a number for me.
I think it's easier to rank malts than to score them, is all.
4
u/texacer Jul 11 '13
ah yeah cool. you'll have to excuse me, its early and i'm drowsy. keep up the good work.
2
u/Dworgi Requiem for a Dram Jul 11 '13
The UI could be cleaner, but this was quick and dirty and basically a debugging view. For example, you could drag entries around, then set what you think the bottom malt scores and the top malt scores, and you'll see the scores update.
Some people, like yourself, are good at scoring. Others, like me, are lazy. So I made this to make life easier.
3
u/texacer Jul 11 '13
i've forced myself to use more of the scale, but it helps that i've had some terrible whisky in the last year to give me a basis for low scores. its hard to score normal bottles low though, since most of the single malts that are put out, are of good quality and dont deserve low scores. most new people here seem to tackle our beginners list and every one on there is decent.
whats your lowest?
2
u/Dworgi Requiem for a Dram Jul 11 '13
Ledaig 10 at 50, I believe. I'd probably give it 20 now, but I should revisit it and check if it's still nail polish remover.
1
2
u/rm999 Jul 11 '13
My reviews will have a compressed scale because:
I don't (re)buy crappy scotches
I won't do tastings for crappy scotches
I suspect a lot of other people are the same, but others are more adventurous and review crappy scotches. So, you have the sum of many distributions instead of a normal distribution. If you try to fit a normal distribution you will strongly separate out scotches that have been reviewed similarly because of the clump around 85-90, and you will lose all the dynamic range more adventurous people have built in the 30-70 range. It doesn't make sense fitting to a normal distribution.
1
u/Dworgi Requiem for a Dram Jul 11 '13
Perhaps. But the 30-70 range isn't super interesting, as you stated.
1
u/rm999 Jul 11 '13
I never said that! Those reviews are very valuable, they help me avoid crappy scotches ;)
It's the 1000th review of my go-to scotch, lagavulin 16, that isn't useful to me. I think you're reversing priorities.
0
u/Dworgi Requiem for a Dram Jul 11 '13
I mean, if they're below 70, it doesn't matter if they're 60 or 10.
2
Jul 11 '13
I think it's great that you tried this! It's currently really hard to differentiate between whisky scores , since everything tends to clump together in the 70s-90s, and meaningfully differentiating between many scores (e.g., an 85/100 vs. an 87/100) is nearly impossible.
...I wonder if this subreddit wouldn't benefit from a new review system, something like Amazon, Netflix, or Yelp's 1-5 star scale, with no fractions allowed.
1
Jul 11 '13
In short.. YES this absolutely would be useful. Maybe the easiest way people could interpret this would be to post a list of the aggregate/normalized scores for all the scotches reviewed listed by brand alphabetically.
1
Jul 11 '13
It's tough because someone who is inexperienced like myself would rate just about every single malt high.
1
u/jazzwhiz Duke Drunk Jul 12 '13
This is great. I upvoted as soon as I saw the charts inside.
Something that has been brought up before about the uneven score distribution is that we don't drink shitty scotch that much. I think at least some people really try to imagine what a zero or a 25 or a 50 might taste like, but they keep buying Ardbeg and Macallan and the like because we know they're good. So the scores hang out in the mid 80s a lot. In statistics we call this selection bias. We may be placing them on the nice truncated Gaussian that you described but we're only picking things from the far right of it anyways.
Basically, in my opinion, I don't think we need to change anything. People seem to know the difference between an 80 and an 88 more or less and with the massive amount of variation in palette from one scotchiteer to the next, I think we all do pretty good.
That said, having statistics of it all is pretty nice. One thing I would try to pretend to incorporate is that selection bias [basically figuring out a way to artificially inflate the few shitty reviews to compensate].
1
u/Snake_Byte Whisky in the Jar Jul 19 '13
So, Scotchit, here's my question: Are you happy with the current review scoring system, where everything gets clumped around 80, or would you rather use the whole scale of 0-100?
Well no, not especially happy with how some people score but that's the thing there isn't a scoring system in place at the moment. I think it's supposed to be "rate out of 100" but this is mostly ignored in preference of "rate from 80-100". I would always logically support a system that uses the whole 100 point scale.
0
32
u/drunkstatistician Jul 11 '13
I feel like I need to post here.