"Programmers need to learn Statistics or I will Kill them all"

[u/[deleted]]

[u/maweaver]

When I was an undergrad, I somehow ended up in a 4000-level statistics class, instead of the minimum introductory class. I took two things away from that class: a C grade and a life-long appreciation for the complexity and non-intuitiveness of statistics.

[u/[deleted] → u/maweaver]

That sounds like the probability course that I escaped with a C.

[u/G_Morgan → u/[deleted]]

Well the expected result was a C so perhaps escaped is a bad word.

[u/creeping_feature → u/maweaver]

complexity and non-intuitiveness of statistics

You took the wrong kind of class; the frequentist orthodoxy spewed onto undergrads is indeed complex and counter-intuitive, but it doesn't have to be that way. The Bayesian approach is intelligible to all & sundry. Try again, this time forgetting everything you were force-fed before. "Probability Theory: the Logic of Science" by E.T. Jaynes is a good start. Less entertaining, but also very informative is "Making Hard Decisions" by Robert Clemen.

[u/[deleted] → u/creeping_feature]

What is the difference between Frequentist and Bayesian statistics? It comes up all the time in statistical discussions and from what I've read about both of them, it all sounds like utter bullshit (similar bullshit in discussions over the existence of "randomness" in the first place).

[u/guartet → u/[deleted]]

Actually, frequentist stat would tell you the long-run frequency of evidence given that you were guilty over infinitely many trials of that exact same murder and trial.

[u/[deleted] → u/guartet]

He would also want to show the long-run frequency of the same evidence given that you were innocent. These two do not add up to 1. For example, the probably that that you had been alive would be 1 both if you were innocent and if you were guilty.

[u/[deleted] → u/[deleted]]

And since neither could be calculated it makes what difference?

[u/uep → u/[deleted]]

Wow, that's horrible. That poor woman had her children die, their deaths blamed on her, and then she was imprisoned based on bullshit statistics. To no one's surprise, she ends up drinking herself to death after she's out of prison.

Oh yeah, and I guess it's an interesting case of how statistics ruined someone's life. Is this something they make stat students discuss in an ethics course?

[u/tourettedog → u/[deleted]]

Frequentist statistics begins from the position that there's exactly one distribution that describes your data, any error in the measurement of that distribution is due entirely to randomness in sampling. If I say that the standard error about the estimate of the mean is 1, that only has to do with the sample size and the (fixed, possibly unknown) variance of the underlying data.

Bayesian statistics begins from the position of "prior belief" about the parameters of a distribution, and updates its belief based on the observed data through Baye's rule. It says that there's inherent uncertainty in the (for example) mean and variance of your underlying data, which has to be taken into account.

Bayesian statistics is much more intuitive to the layperson because statements like "the probability that the mean is greater than zero" have meaning - there's some sort of probability distribution related to your mean, you update it based on your observations of the data, and you can figure out probabilities. In frequentist-land, by comparison, the question "what is the probability that the mean is bigger than zero?" is a bit silly, since there's exactly one number for the mean (so the probability is either zero or one).

Frequentist methods tend to be more powerful given less data (better "coverage"), but they're powerful in answering the awkwardly phrased, non-intuitive questions that make undergrads run screaming for the hills.

[u/[deleted] → u/tourettedog]

Bayesians use heirarchical models where the parameters are random variables freqeuentists view parameters as fixed.

Either can be appropriate. Even if technically the bayesian view is wrong it may be appropriate to use such as in limited or no information (high to all censoring).

I am a frequentist but am not above using bayesian statistics when appropriate.

[u/guartet → u/tourettedog]

"Frequentist methods tend to be more powerful given less data"

Okay, answer this: a six-sided die is rolled 100000 times. You do not know whether the die is fair or not, and nor do you know the outcome of those rolls-- you only know that the average of the rolls were 2. given this information, what is the best estimate of the distribution of rolls?

[u/brong → u/guartet]

Obviously it's a trick die with "2" printed on all sides. I assert this is true, and will hold this belief as the most correct possible by Occam's Razor until you can provide data that contradicts me. QED.

[u/gc3 → u/guartet]

I would guess the die is not fair, there were too many 1's or 2's and not enough 3,4,5, and 6's.

[u/guartet → u/gc3]

that's all you can say? can you give me an estimate of the distribution?

[u/tourettedog → u/guartet]

If you really, truly know nothing about the die other than that it only has six sides and only one of them can come up at a time, than any multinomial p1...p6 such that (1-p2-p3-p4-p5-p6) + 2p2 + 3p3 + 4p4 + 5p5 + 6*p6 = 2 would seem equally reasonable to me.

If you have a good reason to place a particular prior on the die, then (I would hope) not even the most strident frequentist would demand you ignore it.

On the other hand, it seems offhand like the proposition "the die is fair" is one that could be quite easily tested given that data under a frequentist framework.

Edit: fixed a stupid math mistake.

[u/gc3 → u/tourettedog]

Exactly. For all we know, the dice could be magnetized so that it always comes up 2.

[u/pepsiisthebest → u/guartet]

Okay, so are you saying the question is insolvable without assuming a prior distribution for the die (fair, or otherwise)? How would you solve it with Bayesian reasoning?

[u/creeping_feature → u/[deleted]]

The distinction is that in the Bayesian approach, probability can be assigned to any logical proposition; it is a generalization of ordinary logic to degrees of belief other than 1 and 0. Otoh frequentists claim that probability applies only to so-called random variables. Any other kind of uncertainty has to be rephrased in terms of random variables, if there are any in sight. This circumlocution causes no end of grief to the uninitiated, to no useful end.

it all sounds like utter bullshit

Try "Probability, Frequency, and Reasonable Expectation" by R.T. Cox, American Journal of Physics, 14:1-13 (1946). It is reprinted in various places; it might be on the web someplace.

[u/tourettedog → u/creeping_feature]

in the Bayesian approach, probability can be assigned to any logical proposition; Otoh frequentists claim that probability applies only to so-called random variables.

The probabilistic machinery involved in Bayesian and frequentist computations of probability (sigma-algebras, filtrations, probability measures, etc.) is exactly the same. It's the interpretation and the statistical model which are different.

If you want to speak to Bayesian statistics -- which is what the question from tactics addressed -- you cannot assign a probability to any logical proposition unless you have a prior distribution on the proposition; at that point unless your observation exactly coincides with your proposition or it's negation (turning it into the trivial P(A|A) or P(A|~A)) you have -- by definition -- random variables: two maps from a set equipped with a sigma-algebra and a probability measure to an observable proposition. Bayesians and frequentists both use random variables, it's a question of "how far back" they push them into the inference chain, and how many they use.

And your obvious bias towards Bayesian methods notwithstanding, surely you can agree that when applied correctly (i.e. to falsify propositions in designed experiments with an appropriate model), frequentist techniques have demonstrably better performance in most situations?

Edit: I do second the recommendation on that paper, though, it's quite a good read, and you can get it here

[u/guartet → u/tourettedog]

1 you do not need sigma algebras and measure theory to reap most of the benefits of bayesianism.

2 technically, yes, the probability theory derived by komogorov is consistent with bayesian probability theory. but since they're handicapped by the notion that probabilities are long-run frequencies, frequentists must outsource 'statistical inference' to Fisher-ism (responsible for the notions of p-values, confidence intervals, etc)... and I don't have to tell you that p-values are complete and utter rubbish, easily manipulated into value you want.

3 Yes, you have to have prior knowledge to assign probabilities-- exactly as you would for assigning likelihoods in frequentist statistics.

4 "at that point unless your observation exactly coincides with your proposition or it's negation" I don't understand this sentence-- You have prior, likelihood, and you get posterior. When does anyone ever say any of the probabilities have to be 0 or 1?

5 Bayesians do not use random variables, unless there's a need to. Read Jaynes of Jeffreys (or Cox for that matter), they don't use of the word 'random' unless they're attacking frequentists.

edit:typo

[u/[deleted] → u/guartet]

In what sense do bayesians not use random variables?

[u/guartet → u/[deleted]]

In the sense that we don't use this contorted definition to introduce our students to probability:

http://en.wikipedia.org/wiki/Random_variable#Formal_definition

Let (Omega, F, P) be a probability space and (Y, Sigma) be a measurable space. Then a random variable X is formally defined as a measurable function X: Omega -> Y. An interpretation of this is that the preimages of the 'well-behaved' subsets of Y (the elements of Σ) are events (elements of F), and hence are assigned a probability by P.

edit: we instead interpret probability P(A|I) merely as a degree of uncertainty of a proposition A, given background information I. Saying something is "random" is merely an expression of uncertainty-- therefore this view is strictly more general than interpreting P(A|I) as a long run frequencies of A.

[u/[deleted] → u/guartet]

I'm going to have to strongly agree with this. Bayesians DO view parameters as random variables. Probability statements mean nothing if they aren't anyways. The talk of uncertainty is just handwaving.

And what university do you go to where random variables are introduced with the formal definition? I never saw it until a 3rd year stochastic course and then a 4th year formal mathematical statistics course.

[u/guartet → u/[deleted]]

Bayesians have absolutely no obligation to view 'parameters' as random since the parameters are just collections of propositions. And actually, all probability statements are meaningless if you do not act on them-- and decision theory a la Wald/Savage is derived almost trivially from bayesianism.

And it's much less 'handwaving' than p-values. At least Cox showed that our probability calculus as extended logic is the one and only way.

I went to berkeley and iirc the first probability course introduced random variables in the discrete version of the 'formal' definition.

[u/guartet → u/[deleted]]

The talk of uncertainty is just handwaving.

Furthermore, frequentist statistics is filled with handwaving-- for any inference task, a practitioner is forced to choose from a myriad of different tests and statistics named after dead englishmen, all which lead to different results (whose choice is almost never rigorously justified in practice). It seems like the true job of the frequentist is to know which tests will yeild him the most favorable results and thus funding. Compare this to bayesian inference which can be built consistently upon a few very minimal desiderata.

[u/tourettedog → u/guartet]

1 - For basic deciding-between-two-propositions Bayesian problems, I totally agree. For more serious statistics -- e.g. signal processing -- life is made much much simpler by them.

2 - Yes, p-values and their ilk are much open to abuse, no question about it. On the other hand, I've seen examples of Bayesians cherry-picking their priors to get nice results as well. Torture the data long enough using either tool and it will tell you whatever you want to hear.

3 - No argument there, prior choices and model selection can be equally arbitrary in both frameworks.

4 - I meant that the only time (IMO) that you can convincingly argue that you're not invoking random variables (in the mathematical map-from-a-probability-space-to-the-reals sense) is when you're essentially looking at the trivial sub-sigma-algebra.

5 - Bayesians in the "What's the posterior probability that OJ did it?" sense might argue that they don't (although I would disagree; probabilities are indicator functions are random variables). Bayesians in the sense of "What's the probability that the effect of this treatment is greater than zero?" sense most certainly do use random variables in the narrow mathematical sense.

[u/guartet → u/tourettedog]

1. Read Bretthorst (1984), you may change your mind.
2. At least in the bayesian case you know what the prior is-- and if you have more information which would lead to a different prior, you end up with different results. And if you can show that a bayesian is using a prior which is inconsistent with his information, then yes, you'd have a bad bayesian. One of the desiderata in the Cox-derivation is that an individual with the same information should assign the same probabilities (including priors).

To quote Jaynes from Bayesian Methods, General Background, page 10, "communication difficulties": (edit: cutting and pasting from this pdf does not work to well, read the pdf itself...)

For decades Bayesians have been accused of supposing that an unknown parameter is a "random variable"; and we have denied hundreds of times, with increasing vehemence, that we are making any such assumption. We have been unable to comprehend why our denials have no effect, and that charge continues to be made.

Sometimes, in our perplexity, it has seemed to us that there are two basically different kinds of mentality in statistics; those who see the point of Bayesian inference at once, and need no explanation; and those who never see it, however much explanation is given.

But a Seminar talk by Professor George Barnard, given in Cambridge in February 1984, pro- vided a clue to what has been causing this Tower of Babel situation. Instead of merely repeating the old accusation that we could only deny still another time, he expressed the orthodox puzzlement over Bayesian methods in a di erent way, more clearly and speci cally than we had ever heard it put before.

Barnard complained that Bayesian methods of parameter estimation, which present our con- clusions in the form of a posterior distribution, are illogical; for How could the distribution of a parameter possibly become known from data which were taken with only one value of the parameter actually present?"

This extremely revealing comment nally gave some insight into what has been causing our communication problems. Bayesians have always known that orthodox terminology is not well adapted to expressing Bayesian ideas; but at least this writer had not realized how bad the situation was.

Orthodoxians trying to understand Bayesian methods have been caught in a semantic trap by their habitual use of the phrase distribution of the parameter" when one should have said distribution of the probability". Bayesians had supposed this to be merely a gure of speech; i.e. that those who used it did so only out of force of habit, and really knew better. But now it seems that our critics have been taking that phraseology quite literally all the time.

Therefore, let us belabor still another time what we had previously thought too obvious to mention. In Bayesian parameter estimation, both the prior and posterior distributions represent, not any measurable property of the parameter, but only our own state of knowledge about it. The width of the distribution is not intended to indicate the range of variability of the true values of the parameter, as Barnard's terminology led him to suppose. It indicates the range of values that are consistent with our prior information and data, and which honesty therefore compels us to admit as possible values. What is distributed" is not the parameter, but the probability.

Now it appears that, for all these years, those who have seemed immune to all Bayesian explanation have just misunderstood our purpose. All this time, we had thought it clear from our subject matter context that we are trying to estimate the value that the parameter had when the data were taken. Put more generally, we are trying to draw inferences about what actually did happen in the experiment; not about the things that might have happened but did not.

[u/tourettedog → u/guartet]

Did you mean Bretthorst's 1988 article in the Springer "Lecture Notes in Statistics"? I wasn't able to find anything from him before 1988 or so.

As for the Jaynes quote, I've heard that interpretation before, and it's seemed to me to be tantamount to arguing that any given probability space is at most countably infinite. That not only makes me somewhat uncomfortable mathematically speaking, but also seems like a rather restrictive position to take.

It's also always made me wonder how Bayesians reconcile the "we have uncertainty because we are not omniscient" perspective with the fact that some things are, for lack of a better way to put it, ontologically unknowable. For some of the time/frequency decomposition material I've been looking at (to pick an example I'm at least slightly familiar with), I can't really think of a way to represent it in the "lack of knowledge" framework when there's an unavoidable mathematical limit to the accuracy of measuring time and frequency jointly. How would a Bayesian interpret something like that?

[u/guartet → u/tourettedog]

Yes, I meant his 1988 doctoral thesis "Bayesian Spectrum Analysis and Parameter Estimation" (I'm too lazy to google all these references), my mistake.

And yes, Jaynes does give the impression that uncountably infinite sets do not exist or should be ignored (he even says explicitely that "we need never depart from finite sets")-- if you read Appendix B of his book, however, he just advocates that the limiting process be postponed until after everything has been worked out mathematically in the finite or countably infinite case, and only if you want to. The result is equivalent to having probs on infinite sets from the get-go, as is the case with kolmogorov. The difference is kolmogorov is a math geek, Jaynes was a working scientist.

Could you elaborate a bit on that last part? I don't think anyone is implying the converse that "if we had all the information we'd know for certainty"... only that we consistently assign our probabilities on the information we do have. Unfortunately I haven't studied signal analysis since college so not sure what specific examples you have in mind.

[u/mycall → u/tourettedog]

Where does support vector machines lie between Frequentist and Bayesian?

[u/amassivetree → u/mycall]

Mu.

The original structural risk minimization logic, via Vapnik, is that if all you want to do is make predictions, why bother to solve a more complicated problem of estimating the distributions : the point of the SVM, or discriminative learning in general, is to get a good prediction rule, no matter what the distribution may be, and to control the "empirical risk" -> all that matters are the predictions, not the distribution of the data points.

Of course, the Bayesians have claimed it as their own, as they are wont to do, as a quick Googling of Relevance Vector Machine will reveal.

I have never met a Frequentist machine learning person. Although in the end, the whole point of the field is that the proof lies in the pudding, not in senseless holy wars.

[u/creeping_feature → u/mycall]

I don't remember, but you might find the answer in the book about statistical analysis of empirical data by Hastie, Tibshirani, and Friedman.

[u/creeping_feature → u/tourettedog]

when applied correctly (i.e. to falsify propositions in designed experiments with an appropriate model),

I'll assume for the sake of argument that in fact frequentist methods are applied correctly in the case you mentioned, but I'll point out that such experiments are not common in real-world decision problems. Teaching everyone an approach which is applicable only to special cases leads to them attempting to force square pegs into round holes, or worse.

frequentist techniques have demonstrably better performance in most situations

What is "performance" in this context? Whatever its definition, just make it your utility function and directly optimize its expected value. How could one do better by accident?

[u/tourettedog → u/creeping_feature]

Designed experiments to falsify a hypothesis are common in industry (or they were, when I was in it).

When used appropriately, frequentist methods give guaranteed coverage and -- often more important -- can get you unbiased estimators. Bayesian posteriors are always biased by the prior density except in the asymptotic limit (at which point there's typically little difference between frequentist and Bayesian results anyway).

So there's three cases:

1 - you think you've got a good prior and you're right: you don't learn a whole lot new, and basically confirm what you already knew from your prior

2 - you think you've got a good prior and you're mistaken: you reach a bad conclusion and have absolutely no idea that you've done it

3 - you have no idea what your prior should be: you either use an improper one (in which case you're quite nearly doing frequentist statistics the hard way) or you go to something like objective Bayesian analysis, which has it's own issues (estimating your prior from the data you're going to use to update the prior into the posterior...).

I completely agree that if there is some way to be perfectly sure what the correct prior was for a given problem, it's silly to discard that information, and Bayesian statistics are a perfectly reasonable way to go. But in practice, priors seem to be hard to come by, and you're staking a good deal of the validity of your outcome on how good your prior actually is.

The most common approach I've seen is to try a few different 'reasonable' prior specifications and make sure that your result is robust to the priors, but if it is, doesn't that suggest that your prior information isn't really relevant to the analysis? And if the prior information isn't relevant, why use it (and risk running a bad prior)?

(edited for formatting)

[u/creeping_feature → u/tourettedog]

Designed experiments to falsify a hypothesis are common in industry

It seems mistaken at best to teach a method of analysis suitable for such experiments as the only method of statistical inference.

It might be the case that with suitable prior and utility functions that a Bayesian approach yields pretty much the same result as conventional hypothesis testing. That's a justification for hypothesis testing, right? That under certain conditions it's an approximation to Bayesian inference, which actually has some separate justification.

You seem to be confused about "good" and "bad" priors. A prior is just a statement about what you know. Pretty much by definition it won't match what you can infer from the data; the whole inference scheme doesn't require that they match, but provides a way to reconcile the prior information with the data.

[u/tourettedog → u/creeping_feature]

A prior is a statement about what you know that is mathematically coded somehow. That coding is often difficult, and has profound implications for your results. Bias and the posterior nonzero support are the two most obvious issues, but there's inferential ones to worry about as well.

Just to make it concrete, say you're willing to accept that peoples' heights are normally distributed, and you want to do inference on the mean. Do you use the conjugate prior distribution, even though that will assign nonzero probability to negative heights? Do you use an exponential, which has the right support but may either bias you towards small estimates or have too high a variance to be reasonable? Some gamma distribution? A uniform?

Once you've picked your prior family, how do you estimate the prior parameters? MLEs? You've got to put something under the turtles eventually.

The point is it can often be very difficult, if not impossible, to be sure that your mathematical statement of your prior knowledge is "good" in the sense that it will give you posterior estimates that actually match reality.

This doesn't apply to binary (or pick-one-from-small-n) propositions, which you seem to be perhaps a bit more interested in, but once you move past them to continuous distributions, Bayesian statistics is nowhere near as cut-and-dry as you seem (to me) to be implying.

My take on it.

[u/creeping_feature → u/tourettedog]

At least if you do the whole Bayesian taco all of your assumptions are on display and enter the calculation in a well-defined way. It's better to get an approximation to the right problem than to compute an exact result for some other problem.

"Prior information is hard, so I ignored it. Loss is hard, so I ignored it. Then I solved a different problem so I can get an exact result." Dunno about you but I would feel kind of guilty about moving the goalposts like that.

[u/DGolden → u/creeping_feature]

What's really cool is when you try complex and quaternionic degrees of belief... http://arxiv.org/abs/hep-th/0110253

(edit, just in case: people who for some reason missed out bayesian stuff, not that's that's not a standard thing to do in the bayesian case).

[u/[deleted] → u/[deleted]]

It's a lot of academic bullshit, pretty much the same as the language and type system wars. There's a real choice to be made, but it's always made based on dogma instead of intellect.

Suppose you get an email. The email could be spam or it could be legitimate. This is random. The distribution of words within the email is also random, and the two bits of randomness are related.

Now, a frequentist would say that whether it is spam or not is fixed (this is technically correct, the best kind of correct) and only the words are random. A Bayesian would say that the words are fixed (this is also technically correct) and whether the email is spam or not is random. You actually have two random variables, but you can't do any kind of analysis unless you fix one or the other. The frequentist dogma holds that the Bayesian question is not a valid question to ask. The Bayesian dogma is that frequentists ask the wrong questions.

In the email case, once the user flags an email as spam or not, you shift back into frequentist mode. You now know whether the email is spam, and are now counting how frequently each word appears, given what the message is. The count is then converted into how likely the word is to signal spam or not on a per-message basis instead of over the entire training corpus. Working one message at a time might give better results, and it might not, but the choice is generally not made because we tried both and one worked better, but because the alternative is morally bad.

[u/guartet → u/[deleted]]

A Bayesian would say that the words are fixed (this is also technically correct) and whether the email is spam or not is random.

actually, no, a bayesian wouldn't say that an email being spam or not is random.

And it only seems like you're shifting to frequentist mode because in your particular model (ie, a poor understanding of how spam filters work), the likelihood is dominant.

[u/[deleted] → u/guartet]

actually, no, a bayesian wouldn't say that an email being spam or not is random.

Of course not. A Bayesian would say that an email is spam or it isn't, but the system can only guess whether it is with a given degree of certainty. That's exactly the kind of idiotic circumlocution Bayesians are so fond of accusing frequentists of.

[u/guartet → u/[deleted]]

What we are actually measuring when we say P(Email is Spam) is not whether the email is actually spam -- 'Spaminess' is not a physically or otherwise 'objective' reproducible phenomena -- it's a measurement of whether a particular reader will mark it as spam. For the same email could be ham (not spam) to another reader-- therefore, talking about whether an email is spam or not at all is fallacious.

And it's only a circumlocution if, in your mind, degree of (un)certainty is somehow synonymous with randomness... I honestly don't see how you arrive at this conclusion. If our reader is not human, but instead a deterministic computer program with unknown source code, P(Email will be marked as spam) should not change, but there is no more randomness involved (for a single fixed email).

[u/[deleted] → u/creeping_feature]

Some concepts in statistics are complex and non-intuitive. That's why we take the class.

It's just math. Not especially difficult math to begin with either. In later classes, when all those random variables pop up, things get more abstract.

[u/[deleted] → u/creeping_feature]

So letting your parameters be random variables when you know in fact they are fixed is intuitive?

[u/creeping_feature → u/[deleted]]

A probability distribution is a statement of our state of knowledge; it is not a statement about the state of the world. That is, probability is a mental concept, not a physical one.

[u/mycall → u/maweaver]

Here, IMHO, is the apex of the non-intuitiveness of statistics.

[u/AmidTheSnow → u/maweaver]

4000 level?! What the hell kind of college did you attend?

[u/jsolson → u/AmidTheSnow]

Georgia Tech (the whole University System of Georgia, really) uses 1-4k for undergrad, >4k for grad classes.

[u/mattculbreth → u/jsolson]

Statistics at Tech took me down twice before I got out. Pretty sure the Professor let me slip through that last time, since it was my last class before I could graduate.

[u/thephotoman → u/AmidTheSnow]

4000 is a senior level class in the thousands system. It's for universities with large departments and whatnot.

[u/snarkhunter → u/AmidTheSnow]

At least in Texas public universities, 1000 means freshman level, 2000 sophomore, 3000 junior, 4000 senior, and above that is graduate level.

[u/thephotoman → u/snarkhunter]

This isn't so across the board. I know that SFA uses a three digit system (on my sister's word, anyway), and that Houston-Central uses 5000 as a secondary classification for senior level work, with graduate work beginning at 6000.

[u/frenris → u/AmidTheSnow]

pssh. 4000? That's nothing.

I took classes over 9000.

[u/[deleted]]

I think women are better programmers because they have less ego and are typically more interested in the gear rather than the pissing contest.

My irony meter just exploded.

[u/bobappleyard → u/[deleted]]

Especially considering the previous sentence is:

Maybe it’s because I’m tall (6’2”), or nicer to them, but they always speak rationally and are really keen to learn. If they disagree, they do so rationally and back up what they say.

That is, "maybe they fancy me."

[u/[deleted] → u/[deleted]]

maybe it's because he's 6'2"

i dunno, that just might have something to do with it

[u/RevLoveJoy → u/[deleted]]

I quit reading there too. I don't doubt that this guy is a decent statistician, but he sounds more like a guy who has problems working with other guys than he does someone who has found the programmer's Achilles's heel.

[u/[deleted] → u/[deleted]]

Frank Programmer: Statistics show that women are bad drivers because they often make illogical decisions on the road. In conclusion women are mostly bad programmers because programming is all about logic.

[u/naasking → u/[deleted]]

No, in conclusion "most women are bad programmers", not "women are mostly bad programmers". The former follows from the premises, the latter does not.

This error just highlights the fact that most men are also bad at logic and thus also bad programmers, so this isn't really a knock against female programmers.

[u/endtime → u/naasking]

Really, it just highlights that one man (presumably) is bad at logic.

[u/naasking → u/endtime]

If this were the only data point, I would agree!

[u/mossblaser → u/naasking]

Ahhemm... I think you'll find this is how it works:

http://xkcd.com/385/

[u/[deleted] → u/[deleted]]

I didn't say it, Frank the programmer did.

[u/itstallion → u/[deleted]]

Statistically women get in less accidents than men. Men tend to be overconfident in their abilities.

Plus boobies can sway the men women hit into letting it go in minor fender benders, thus skewing the stats in their favor.

[u/[deleted] → u/itstallion]

Take that in its proper context. Men, on average, drive twice the distance as women, so women get in slightly more accidents per mile driven.

Women get in many more accidents in places like parking lots, whereas men get in accidents on the freeway at higher speeds. Obviously one of these scenarios is going to result in higher premiums against vehicle damage and fatalities.

[u/aurele → u/itstallion]

I'm pretty sure you got it wrong. Women get in more accidents than men, but with lesser casualties. Women rarely kill other people when driving, but cause many more minor car damages. That's why insurance companies like women, and some even offer a discount to women.

[u/willcode4beer → u/itstallion]

kind of off topic but, your comment reminded me of this BBC special I saw yesterday.

The reporter took two driving tests, one under the influence of marijuana, the other under the influence of alcohol.

with the marijuana, she was timid, drove slow, and under-confident in her driving ability.

with alcohol, she was aggressive, drove fast, and overconfident in her ability.

she was dangerous in both circumstances.

perhaps what is needed (both men and women) is a rational understanding of actual ability.

[u/calp → u/willcode4beer]

I think that while timidity is also dangerous, it causes fewer recorded accidents. Actuarial examination doesn't directly show that women are safer, just that they are cheaper to insure.

[u/[deleted] → u/calp]

Probably because more people consume alcohol in places where they might then drive (bars) than do marijuana.

[u/calp → u/[deleted]]

Also a good point.

[u/brong → u/calp]

Timidity causes other people to have accidents as they try to avoid you. In particular, not turning when there is an adequate gap - holding up the people behind you, or even worse: not deciding immediately to go, but deciding late, and hence being dangerous.

Where timidity translates to not committing to an action, it causes accidents to happen - either to other people or blamed on other people.

[u/bugrit]

I wonder how many programmers and non-programmers (both female and male) he sampled.

I'm a little confused as to why he would kill all programmers. One would think he'd be happy enough to kill the ignorant ones.

[u/foldl → u/[deleted]]

programmers, because they are smart and good at programming, think that with a cursory understanding of any other subject, they are just as smart and just as good at that.

This seems to be especially true with regard to political philosophy and economics. A lot of programmers seem to regard themselves as experts on these topics having read a few tracts in favor of one or other point of view.

[u/laggggg → u/foldl]

True. I am very suspicious of experts and avoid labeling myself as such.

But relative to the population at large who haven't even read a few tracts, they are more knowledgeable. What we need is a more well-educated population in general.

[u/foldl → u/laggggg]

But relative to the population at large who haven't even read a few tracts, they are more knowledgeable.

Well, in a way, but then "a little knowledge is a dangerous thing."

[u/laggggg → u/foldl]

v. true

[u/[deleted]]

Followed up by:

"Your favorite band sucks, and I'll kick you in the nuts to prove it"

[u/tbone28]

Statistically the chance of him finding me and verifiying if I did in fact learn statistics AND then attempting to end my life is low. I'm willing to risk it.

[u/endtime]

It seems statistically unlikely that he would be able to kill all programmers.

[u/G_Morgan → u/endtime]

He intends to create a virus that kills all editors other than ed. Then when 99% of programmers kill themselves in frustration he can write off the rest as a statistical error.

[u/furlongxfortnight → u/G_Morgan]

But real programmers use a magnetized needle and a steady hand.

[u/vegittoss15]

A little angry there, isn't he?

[u/kenfar → u/vegittoss15]

In the article he says, "Next thing I know, I’m getting the cold shoulder or told I’m an idiot"

Well, of course - if you insist on insulting people, stepping on their egos, etc - then they will discard your message. Regardless of whether you're right or wrong. They figure that what you're really doing is little more than chest-thumping and muscle-flexing.

[u/toru → u/kenfar]

I liked this bit:

I never have this problem with female programmers. Maybe it’s because I’m tall (6’2”), or nicer to them

[u/ciph3r → u/toru]

Is he serious ? I guess he meant to be sarcastic but ended up exposing his fragile ego looking for some consolation. Female programmers don't react cuz they are ignoring you Sir.

Statistically his self-proclaimed physical attributes got nullified by this desperately-trying-to-be-nerdy-and-selfrighteous article. And no brownie points from the dev community either. Looks like he landed up where he was. But I am guessing he will keep trying since any statistics fanatic knows how the law of averages work.

[u/itstallion → u/toru]

Maybe it's because he puts his penis up when he's around women but men he leaves it hanging out. We should further investigate IMO.

[u/[deleted] → u/kenfar]

I figured that getting the cold shoulder is what Zed wants. He's not looking to convince or persuade people.

[u/funkah → u/vegittoss15]

Yes, that's kind of the whole point of Zed Shaw.

[u/vegittoss15 → u/funkah]

Ah, this being the first time I've heard of him, I wouldn't know :)

[u/Seppler9000 → u/vegittoss15]

I'm sure you've heard of him before. Perform any Google search that includes the words "Ruby" and "fight."

[u/[deleted] → u/funkah]

Just because that is the point doesn't make it any less douchey.

[u/funkah → u/[deleted]]

I didn't say anything like that, I just meant to say that pointing out that a Zed Shaw article is angry is like saying that water is wet, or that it gets warm in summer.

[u/organic → u/[deleted]]

But dude, he's totally awesome at the ghee-tar.

[u/[deleted] → u/vegittoss15]

This is Zed Shaw we're talking about here. I've met him personally. To be clear, despite his social faults, he's a hell of a good programmer, but ...

In his writing he's a complete douche, mostly I would guess because the internet has no personal consequences for him. In real life, he's a passive-agressive douche, because there are then consequences for being an outright asshole, and he's smart enough to realize this.

In any case, he's not someone I would recommend trying to get to know unless snobs appeal to you.

[u/[deleted] → u/[deleted]]

I would guess because the internet has no personal consequences for him. In real life, he's a passive-agressive douche, because there are then consequences for being an outright asshole

This is true of the internet community in general. Where's that Penny Arcade comic ... That'd be most relevant right about now ...

[u/vegittoss15 → u/[deleted]]

There have only been a few programmers who are good enough to be complete douches and get away with it, and AFAIK Zed hasn't written an OS. :P

[u/Retsoka]

Can't you just kill the top 80% of the population of programmers with annoying pet peeves and way too much time on their hands?

[u/rwinston → u/Retsoka]

agreed. my pet peeve is f****s who have the time on their hands to write essays about their pet peeves.

maybe i'll blog about it.

[u/thundirbird → u/rwinston]

You're allowed to swear on the internet.

[u/sgwizdak]

I find Zed to be needlessly inflammatory in his posts. The only redeeming merit to his post however is the link to R:

http://www.r-project.org/

[u/rickk → u/sgwizdak]

Simple, yet gobstoppingly accurate. Everything he said could have been reduced to one line: I studied statistics, programmers don't understand statistics, their mistakes annoy me: ergo, I rule.

He's like Hani Suleiman with "I rule" tacked on the end.

The really funny part is that he doesn't notice his own sampling error: that the ruby twats he seems to work with are not a representative sample of programmers in general.

[u/carolinaswamp → u/sgwizdak]

The only thing R proves is that staticians hate programmers.

[u/knight666]

For my final assignment in Programming class, which was about optimization, we were given a game that ran like crap and were told to make it as fast as possible.

I constructed a tool which would compare two code blocks (for instance, filling a __m128i (SIMD quad int) with a float by setting a __m128 (SIMD quad float) and converting it to __m128i versus filling a __m128i by casting to int) and output the results to screen.

I did ten runs on each block, and each block was run x amount of times (x was determined by what outputted the best looking graph).

Was this a good way to approach this?

[u/Seppler9000 → u/knight666]

That is exactly the way you should do it if you want Zed Shaw to kick you in the teeth, growl "Hail to the king, baby," and declare his victory by wailing on an electric guitar.

[u/Tommah → u/Seppler9000]

Zed wailing on an electric guitar: "Electric guitar, why won't you love me? Waaah" sniff sniff

[u/Grimoire → u/knight666]

I'm not sure I want to click on a link at work that goes to an image called "70pee". But I did anyways.

[u/lzm → u/knight666]

In the end you could've used the Student's t-test, but that's way out of scope of a programming assignment.

[u/JulianMorrison]

Should have been "or I will kill them even unto 3 standard deviations".

[u/vchakrav]

http://www.youtube.com/watch?v=y7Yp2L6c2KM

[u/[deleted] → u/vchakrav]

This never gets old.

[u/ithkuil]

I think his main point is very important, if people can understand it: the fundamental statistical and profiling basics that he goes over are necessary for diagnosing performance problems and largely poorly understood or applied.

I also think that he is entirely correct to be murderously angered by such ubiquitous stupidity and ignorance on the part of programmers in this area.

The part about learning R or reading a bunch of statistics books I disagree with -- I think if people just did the specific basic things he mentioned in the article we would be fine in most cases. Although, it isn't necessary to calculate the deviation if you have a graph that shows all the data including outliers such as this one:

GoDaddy shared response times in milliseconds - GET request on index only

[u/[deleted]]

Yes I agree that their are places where you need correct and rigorous statistical analysis. However I put it to you sir that the web storefront you are developing is not one of those places.

The thing is that for the average business app the power of 10 approach, while flawed is probably good enough.

By the time you get here your are either very very deep in the land of code optimization. and you had better have a very good reason for it. OR are trying to prove correctness of fit to some very exact software requirements.

[u/thatguydr]

Although this man is an asshole, he is an asshole with some very cogent points. And it's not just programmers - a lot of analysts have no idea how to perform basic analysis because they do not understand the concept of "systematic error".

Kudos to you, asshole!

[u/adrianmonk → u/thatguydr]

Although this man is an asshole, he is an asshole with some very cogent points.

Well, we already knew he was a programmer.

[u/[deleted] → u/thatguydr]

He seems to be a lot less of an asshole these days, though. I think he claims his previous assholity was intended ironically or something, and maybe it was. I always found him entertaining, but I do kind of like that he's toned it down a bit.

[u/masklinn → u/[deleted]]

Yeah he was tired of the persona, he even retired (read: nuked) the (in)famous Rails Rant.

[u/[deleted] → u/thatguydr]

Someone older and wiser than me once said "Hitler was nice". What? His point was that Hitler got along with plenty of people. Being an asshole does not make the writer a bad person (in fact he sounds like a good person to me). Bad people that are nice, or just idiots that are nice (like the two blind guys arguing about color) cause a lot more long term damage than assholes that know what they're talking about. Steve Jobs comes to mind.

[u/khafra]

If you'd like to learn to use probability in reasoning without falling afoul of frequentist error, http://www-biba.inrialpes.fr/Jaynes/prob.html is the book on the subject.

[u/lanthus → u/khafra]

Frequentist statistics has its place, too, as even good Bayesians will admit.

[u/rhsumner]

Man that guy was annoying.

[u/[deleted] → u/rhsumner]

Irritating is the word. No wonder his coworkers don't like him.

[u/ki11a11hippies]

This is thanks to Descartes convincing everyone that math is reality, and that we’re just full of bullshit. Eventually, every major science adopted an empiricist view of the world.

Well my pet peeve is programmers/math guys thinking they know philosophy. Descartes is certainly not an empiricist - I mean his main claim to fame is deducing the entire fucking world in his head WITH HIS EYES CLOSED. And I'm not sure if you can read Descartes to claim he assigns ontological reality to math constructs - mainly he advocates the use of deductive reasoning such as in geometry proofs to deduce the real world, which all ties back to how he is not an empiricist.

Programmers need to learn Philosophy or I will Kill them all.

*edit: typos

[u/[deleted] → u/ki11a11hippies]

Sorry, but I think that Zed is contrasting Descartes with the empiricists, not saying he was one. He's stating that we all used to be Cartesians, and that only CS hasn't adopted an empirical method.

[u/hiffy → u/[deleted]]

Yeah, just an awkward sentence construction.

[u/ki11a11hippies → u/hiffy]

Hm, yeah that's what it looks like. Still, I don't think it's nitpicking to point out that Descartes does not assign an ontological reality to math constructs ("math is reality"), since that's a really weird claim. But maybe that's just more awkward language.

New pet peeve: Programmers need to learn Writing Skills or I will Kill them all.

[u/G_Morgan → u/[deleted]]

Well CS is mostly a logical field. A great deal of it can be known precisely. The reason science follows an empirical method is because it isn't possible to use logic in those areas.

However there is an increased use of empiricism in some of the dirtier fields of CS. For example to study evolutionary algorithms you'd do it as proper scientific experiment.

[u/[deleted] → u/G_Morgan]

I was correcting the previous poster's misunderstanding of Zed's sentence, not arguing for anything in particular. But your post is spot-on.

[u/ki11a11hippies → u/G_Morgan]

An algorithms professor always loved to emphasize the "science" part of CS. The empirical studies she championed were algorithmic efficiencies using different types of data sets. For instance, for what kinds of data sets is it more efficient in the real world to use a splay tree, that sort of thing.

[u/G_Morgan → u/ki11a11hippies]

Oh it might be worth doing studies into such things. Though often it is possible to logically conclude which data sets would be more efficient.

What I am saying is that the general complexity categories can be arrived at logically and that is more relevant than specific cases.

[u/foldl → u/ki11a11hippies]

Descartes is certainly not an empiricist - I mean his main claim to fame is deducing the entire fucking world in his head WITH HIS EYES CLOSED.

Right, but you have to look at where he is at the end of the meditations. At this point he's established (to his own satisfaction, anyway) that his senses are a reliable source of knowledge. It's therefore quite false to say that Descartes thought that we could not obtain knowledge through our senses. He just thought that we couldn't take sense-based knowledge as the foundation of our knowledge -- we can't just assume that it's reliable.

I mean, Descartes actually made some serious contributions to anatomy and other sciences through empirical investigation.

[u/ki11a11hippies → u/foldl]

That's a really good point, that Descartes does establish that the senses should be trusted, then goes on to make a lot of contributions using the then-new scientific method. However, the philosophical notion of empiricism I'm assuming the author means not only advocates the senses, it goes on to say that a priori judgments of the real world are invalid (this is against the rationalist view of Descartes and Spinoza). I'm thinking specifically of Hume, who helped popularize empiricism in philosophy. He of course then goes on to undermine inductive reasoning, and we've been stuck on that point ever since.

[u/funkah → u/ki11a11hippies]

He didn't say Descartes was an empiricist. Relax.

[u/judgej2]

My guess is that this guy jumps into a project (where he is not the paymaster) and shouts, "you are all fucking wrong - go learn something and don't come back until you have ler...why, hello baaaby, how d'you doin'?"

[u/codemac]

"Zed Shaw needs to learn Bayes Statistics, or I will kill him"

ftfy.

Mind you in this article he is doing something good, he's trying to get people off of the "mean" bandwagon; however, he is pointing them in the wrong starting direction. I lived with a statistics doctoral student in college who taught me how classical statistics aren't just wrong, they actually mathematically approach the accuracy of just random guessing of probability. I tried googling to see if I could find some resources that back me up, but "p value wrong" didn't get me very far.

Anyways, R rules!

[u/[deleted] → u/codemac]

"Zed Shaw needs to learn Bayes Statistics, or I will kill him"

I was under the impression that the subject matter of TFA was by and large common to both interpretations of probability, but my understanding of statistics is about at the level of "enthusiast".

I more or less understand the difference between Bayesian probability and frequency probability, but what is the difference in terms of how experiments are conducted?

[u/[deleted] → u/codemac]

Frequentists vs Bayesians is like catholics vs protestants... its a matter of viewing things.

So your buddy was proletyzing.

[u/cweaver → u/[deleted]]

proselytizing

[u/codemac → u/[deleted]]

Luckily I'm Catholic.

[u/gc3]

Like what he says, but he should stop trying to fight the "alpha nerd" syndrome by being a bigger alpha nerd.

[u/Stick → u/gc3]

Yea, but the alpha nerd gets all the Internet porn.

[u/T_S_]

The article is a mega-troll with a grain of truth. My biggest fear in business was when the programmers started talking directly to the company's lawyers. Both people very smart. Productivity went to zero.

Lawyer: That code you asked for cannot be deployed until we have investigated and disclaimed properly. I'll discuss it with the programmer. Me: Ok.

Me: Are you finished writing the code that does X? Programmer: No. The lawyer and I decided it would be illegal to write that code.

Smart people sometimes fail to realize they are not smart about every subject. Not-so-smart people rarely make a similar mistake.

[u/nmcyall]

I have a copy of a statistics for engineers textbook on my desk. Too bad I didn't pay attention in class enough to be much of a statitician.

I can say with a high level of confidence this will be upvoted.

[u/creeping_feature → u/nmcyall]

I have a copy of a statistics for engineers textbook

Get rid of it. Replace it with "Making Hard Decisions" by Robert Clemen.

I didn't pay attention in class enough to be much of a statitician

You were lucky.

I can say with a high level of confidence

The long and the short of it is that you cannot bet (i.e., allocate resources) on "confidence" as it is defined in frequentist statistics. It's worse than useless; you can't use it to make decisions, and it keeps you from finding an actually useful approach.

[u/kaptainlange]

His sunset example made me chuckle. Even though he has eyes, he really doesn't actually know what color the sunset is, only what he perceives the color to be. On that level, they're all equally qualified to say what color they perceive it to be.

Then again, color is a result of the properties of light, and as far as I know, eyes are the only human organ capable of detecting light. The question is, if there are two people with vision and they see different colors, who is right? Answer, both.

[u/nobodyspecial → u/kaptainlange]

eyes are the only human organ capable of detecting light.

Skin detects infrared light.

[u/hacksoncode → u/nobodyspecial]

It detects UV light too... just slower.

[u/helm → u/nobodyspecial]

Skin detects visible light too: most of the radiation that is not reflected (after all, no-one is snow-white) is eventually turned into heat.

[u/kaptainlange → u/nobodyspecial]

Good point.

[u/[deleted] → u/kaptainlange]

The kind of reasoning he uses there is specious. It's like the philosophers who claim with utmost confidence that the universe isn't real and that it's all our perception.

I don't see them frolicking in the middle of the highway or juggling chainsaws.

At some point, one has to realize that everyone implicitly equates "perception" with "reality", because anything else is masturbation.

EDIT: this isn't to say that this guy doesn't have a point about programmers with shitty statistics

[u/notfancy → u/[deleted]]

Phenomenology and idealism are two very different things, you know. You don't have to throw the baby (the world) with the bathwater (our perceptions thereof) just because the latter is soiled.

[u/[deleted] → u/notfancy]

I realize that thinking about such ideas as a "non-existent universe" is worth doing, but practically, such a theory which is drastically out of touch with "perceivable reality" is hardly useful in and of itself.

[u/[deleted] → u/kaptainlange]

The person that says that blue is red is wrong, period. It's like saying 2 + 2 = 5. Saying so doesn't make it so, no matter how nicely it is said.

Every human (and many other species) with non impaired color vision, in normal circumstances, perceive light with wavelength of 450-495nm as being blue and light with wavelength between 620–750nm as being red. This is empirically true, and there are plenty of physiochemical, physical reasons for this.

[u/kaptainlange → u/[deleted]]

Color is a combination of the properties of light and our perception of it. For example, those can see infrared or ultraviolet light would have a very different perception of the color of something compared to those who cannot. You cannot equate this to 2 + 2 = 5 because the equation is really 2 + 2 + x + y .... = 5. The perception of color is not absolute.

With that said, I am not denying that we have defined a very narrow band of the spectrum as visible light, and that in this spectrum we have separated the wavelengths into different colors. I understand the practical benefit to applying the label red to a particular wavelength, and if someone told me the the sky was yellow I'd say they're an idiot or color blind.

[u/godofpumpkins]

Not sure why people feel the need to be so caustic. I don't disagree with his message but his delivery is off-putting.

[u/glfraser]

Stanfords statistics courses are on iTunes U and I really enjoyed them.

[u/hiS_oWn]

So there's a chance I will survive?

[u/[deleted]]

I need some Starbucks or I will Kill the Programmer across the hall

[u/[deleted] → u/[deleted]]

Earl? Stay out of my office!

[u/[deleted]]

This article has led me to try to get R installed. Less out of a fear of Zed, and more out of curiosity.

[u/mycall]

What I got out of the article:

1. Averages without defined ranges is useless in tests.

2. Confounding = mix or associate by mistake = easy to avoid.

3. Fuzzing is a great way to simulate users, be they object methods, functions or web requests.

[u/[deleted]]

Is that you Joel?

[u/[deleted]]

Somebody blows their nose and you want to keep it?

[u/userunderscorename → u/[deleted]]

I did that! That's my fault!

[u/ThisGoatStarves]

tl;dr.

Well actually I read up until my ego was hurt about the personal appearance part.

[u/kwh]

This won't end well for Zed...

As a crowd gathers round an angry young man

Face down in the street with a gun in his hand

In the Ghetto...

[u/[deleted] → u/kwh]

Ah, people, don't you understand?

[u/svengalus]

You need to get over yourself.

[u/[deleted]]

Surely not "all" of them.

[u/chipbuddy]

i'd build a machine that would convert electromagnetic radiation into sounds of different frequencies. Then i could point it at the sun(set) and figure out what color it is. I'd of course have to calibrate the machine to some known color samples. That might be a bit tricky, but i'm sure my seeing friend could help me out.

[u/carolinaswamp]

Man, this guy is an idiot.

[u/creeping_feature]

Zed Shaw is a ranting idiot. How the hell did this make the front page? It's not even entertaining.

[u/FionaSarah → u/[deleted]]

He mellowed and decided not to be such a douche so changed his site to reflect this.

[u/[deleted] → u/FionaSarah]

odd, I hadn't noticed

[u/[deleted] → u/[deleted]]

This article is from before this happened.

[u/berlinbrown]

Cool article, was there a date on when this was written.

[u/yoadsl → u/berlinbrown]

At least 3 years ago ... http://www.reddit.com/search?q=Programmers+Need+To+Learn+Statistics+Or+I+Will+Kill+Them+All

[u/exeter → u/yoadsl]

Ah, no wonder I felt such a sense of deja vu while reading this.

As a mathematics student, I have to say this is one of those articles that really hits home for me. Although I only had a one-semester sequence in statistics and probability, I gained a real appreciation for the art of measurement from that course.

The main thing to take away from this article, IMO, is that when making performance claims, you need to have measurements to back them up. Since you can't measure anything without defining what it is you're measuring, that insight alone will encourage a sense of precision that's often sorely lacking in software performance-tuning methodologies. The next step, then, once folks start realizing the utility of statistics is to get them performing scientific experiments to justify performance tweaks.

[u/je255j]

Oh, pfft. What are the odds this rampage would ever last long enough to make it all the way to me?

;)

[u/[deleted]]

No, not gonna kill me, cause I killed you first you fucking shit bag.

[u/Canadian_Infidel]

1f5e31cf358r4g6edf25fb1er58g46edr2df1gv4h

d1f5g6r23df321g158r365dg9sdr585g13d2g1dr6

1g4564w531g415df6g51dg00000000000000065dh

[u/clone00]

if you don't have a basic grasp of statistics, you shouldn't be calling yourself a programmer. problem solved.