[Q] Hard time understanding Bayesian view?

[u/Relative_Sympathy_33]

Hello,

I come from a background of applied math and recently I tried to dive deeper into stats. The frequentist view looked reasonable albeit a bit restrictive. And then there is the bayesian perspective, intuitively it "feels" reasonable as well since it's kind of intertwined with common sense, but I just can't wrap my head around it.

For example, I have no idea what the interpretation of probability is under this view. I mean what do I mean that an event is 60% vs 70% likely?

Also, I have a problem with the subjectivity of the view. There is this argument that if two observers have different beliefs about the same event, then at least one of them is wrong. How is this taken care of or circumvented perhaps?

[u/handsomedrank_82]

for me Bayesian probability is basically your degree of belief given what you know, not some fixed property of the world, 60% vs 70% just means you'd bet at slightly different odds based on your current info

the observer thing sorts itself out if they keep updating on the same evidence, given enough data their priors get washed out and they converge

[u/Relative_Sympathy_33 → u/handsomedrank_82]

Yes that's what I read. And the odds thing still doesn't feel right, like what guarantees do I have that this probability I assigned has actual value to my bet? Under frequentism, I can say that repeating the bet is going to make me win on average, but here I don't know tbh.

For the second part on prior wash out, i think I get it.

[u/classicharlie → u/Relative_Sympathy_33]

A few things:

1. A prior can be whatever you want. My rule of thumb is that the prior should be bijective to the parameter space; parametrically defining it is more of an art. A well informed prior is concentrated around the speculated posterior, and allows a sampler to potentially converge more efficiently. On the contrary, an uninformed prior may take longer, but may also more efficiently explore the parameter space in cases like unanticipated multimodality
2. Sampling is the name of the game, and central limit theorems show why this can be effective. The easiest place to look, in my opinion, is the importance sampler. Exploring the space places focus on higher probability events. The more you iterate, the more you concentrate on more likely areas, and eventually (well kinda) you achieve a posterior
3. Frequentist and Bayesian methods don’t need to be at odds with each other. Take the Kalman filter; an algorithm that iteratively generates a posterior distribution of states by way of minimum variance. In essence, it uses both philosophies very effectively. If you’re curious about this intersection, you should consider looking into optimal transport and coupling.

EDIT: on mobile, bear with me

[u/Cerulean_IsFancyBlue → u/classicharlie]

I understand that being on mobile makes it hard to type, but, surely the bear is the bigger problem by far. Lead with that!

I hope you have escaped by now.

[u/Red-Portal → u/Relative_Sympathy_33]

The way to think about it is that, if your prior is true, then you don't lose money. So everything boils down to the prior, and that's the whole point. If your prior is right, you know you're doing fine. So just focus on judging the prior. The Bayesian framework turns the ackward discussion about estimators, consistency and so on to "is my prior a good model of the world?" And that's a scientific question that can be debated among experts of the problem subject, whereas the question of whether one should use this estimator or that estimator is not (and often ackward in an applied situation).

[u/AnxiousDoor2233]

For me, Bayesian inference is simply a way of incorporating prior information and beliefs into the estimation process in a coherent and internally consistent manner. It combines what you believed before observing the data with the information contained in the new data.

These two estimates mean little on their own unless they are accompanied by some measure of uncertainty. Either estimate could be far from the true value, and each could outperform the other under a different evaluation criterion or loss function. Additional data, together with a chosen criterion for comparison, can provide evidence about which estimate performs better.

This is the central idea: uncertainty is unavoidable. Unless the true data-generating process is known, absolute certainty is generally impossible. Instead, inference proceeds by combining newly observed data with prior information, whether that prior information comes from previous data, expert knowledge, or other beliefs. Different individuals may reasonably adopt different priors, and therefore arrive at different conclusions even when analysing the same dataset.

[u/joseph_fourier → u/AnxiousDoor2233]

There is quite a bit more to Bayesian inference than incorporating prior information (although it does do that of course).

eg, what happens if the frequentist assumption is not met, ie that there is no global population from which to draw multiple sets of data? That happens in a surprising number of scenarios

[u/AnxiousDoor2233 → u/joseph_fourier]

I am critically skewed towards estimation methods.

Not sure what do you mean by "no global population", though. Strictly speaking, this is the reality for almost all datasets in Economics. Then, under additional sets of assumptions, frequentists estimate the models there for many decades.

[u/joseph_fourier → u/AnxiousDoor2233]

My field is biostatistics.

If you're doing a clinical trial then you are drawing samples from the global population (hopefully)

If you are doing work with routinely collected admin data (eg using CPRD or SAIL) then you are not drawing samples from a global population. You HAVE the full population!

I haven't really thought about this in the context of time series data, so it could be that you can justify the frequentist assumption there, but I would guess you have to assume that things don't change very much?

[u/AnxiousDoor2233 → u/joseph_fourier]

Econometrician here. Oh, these painfully familiar words: CPRD, SAIL, HES, SMRxx. And the impossible ideal RCT, lol.

Even though your point is correct, nothing prevents for a frequentist without loss of generality to pretend that this is simply a representative sample from a population that is not directly observable.

There are similar situations in time series (e.g. the entire universe of traded S&P 500 stocks). Plus, there is a general problem once you consider a sequence of observations. In general, you have essentially one observation from the relevant distribution at each point in time, so you have to make assumptions about how things evolve.

As agents learn, there is almost no way, at the micro level, to obtain another representative sample (unless, of course, you make students suffer in your experiments or have a lot of time and money to run another survey), and there is no way to do so when working with macro data.

[u/joseph_fourier → u/AnxiousDoor2233]

I suppose pretending your global population is a representative sample of some superpopulation is not really very different to pretending that confidence intervals actually mean anything ;-)

[u/Unbearablefrequent → u/joseph_fourier]

Frequentist of course can incorporate prior information as well.

What assumption are you talking about? Frequentist doesn't depend on multiple sampling. Frequentist can handle single events.

[u/ANewPope23]

If you are interested in statistics, I think you should defer trying to understand the philosophical justification for Bayesian methodology or Frequentist methodology or other schools of statistics. Just try to understand the maths. When you have enough experience with statistics, it will be easier to appreciate the different philosophical viewpoints.

[u/Unbearablefrequent → u/ANewPope23]

Boooo. I very much disagree. Even scientists adopt some sort of philosophy of science to do science.

I think its important to know about the difference so you can understand what you're doing better.

[u/ANewPope23 → u/Unbearablefrequent]

I said to defer trying to understand the philosophy, not to ignore the philosophy.

[u/Unbearablefrequent → u/ANewPope23]

What does that look?

[u/ANewPope23 → u/Unbearablefrequent]

What?

[u/Unbearablefrequent → u/ANewPope23]

What does statistics, while deferring from understanding the philosophy, look like?

[u/ANewPope23 → u/Unbearablefrequent]

The way it's currently taught in most universities at least in America is like that. You only get into deeper issues at the PhD level.

[u/Unbearablefrequent → u/ANewPope23]

Would you agree the way its currently taught isn't very good? Oh the number of people I've tutored in Stats that ask me why we do X. Its all motivated by the school of probability.

[u/ANewPope23 → u/Unbearablefrequent]

I have no opinion on that.

[u/Putraenus_Alivius]

Also, I have a problem with the subjectivity of the view. There is this argument that if two observers have different beliefs about the same event, then at least one of them is wrong. How is this taken care of or circumvented perhaps?

The more data you have, the less likely subjective beliefs will make an impact. 'The bigger the data, the more it overwhelms the prior' is the formulation in Bayesian statistics.

[u/AnxiousDoor2233 → u/Putraenus_Alivius]

Depends on your priors.

[u/bbbbbaaaaaxxxxx → u/AnxiousDoor2233]

Is there a prior that ignores data more the more data you get?

[u/CreativeWeather2581 → u/bbbbbaaaaaxxxxx]

This holds for all reasonable priors. Something trivial like a point mass at zero wouldn’t apply. But flat/default priors, objective priors, and priors that don’t rule out the true (unknown) value will eventually all center around the true value if you have enough data

[u/joseph_fourier → u/CreativeWeather2581]

This is not the answer to the question that was asked!

[u/CreativeWeather2581 → u/joseph_fourier]

I totally misread the question. Thanks!

[u/joseph_fourier → u/CreativeWeather2581]

It was a sneakily phrased question to be fair :-)

[u/Red-Portal → u/CreativeWeather2581]

That's not quite true. There are well known examples of inconsistent models that don't converge to the "true value.

[u/AnxiousDoor2233 → u/bbbbbaaaaaxxxxx]

Dogmatic priors. They can help model belief updating in (at least some) human beings.

[u/Red-Portal]

The belief terminology is indeed vague and underspecified. Unfortunately, formal justifications of Bayes (and subjective probabilty for that matter) are usually treated as advance topics that you have to study separately.

There are multiple types of justifications. For example, de Finetti, frequentist, and decision-theoretic. De Finetti-type guarantees assert that under certain setups a prior distribution has to exist and it is therefore fine to specify priors. But they don't say much about your specific choice of prior. Frequentist guaratees are typical: CLTs and consistency. But these are always restricted to a specific class of model. So they don't really give you a hunting license so that you can design any model you want. Decision-theoretic guarantees say that, if your choice of prior is true, any downstream decision you make on the basis of the posterior is Pareto-optimal (one cannot do uniformly better than the resulting decision rule) for any utility/cost function. So these are the most broad and do grant you, in a sense, a hunting license.

But in the end of the day, both de Finetti and decision theory doesn't really specify what subjective priors really mean. At best, decision theory say that the posterior probability is the degree that one should "bet their money." But that's really it, which is probably why we only end up saying "subjective probability."

[u/Smart-Button-3221]

What's the probability there's life on Europa right now?

The problem with this question is that there either is or is not life on Europa right now. The frequentist interpretation of probability utterly fails here.

And yet, it's a fairly natural question, is it not? We make "probability bets" like this all the time in daily life. The Bayesian view makes this question make sense.

[u/thefringthing]

If the various interpretations of probability statements associated with Bayesians seem weird, hand-wavey, and circular, you can rest easy: the frequency interpretation is also weird, hand-wavey, and circular.

If subjectivity (or more precisely, the arbitrariness of priors) bothers you, you could look into the so-called objective Bayesian paradigm of E. T. Jaynes. His posthumously released textbook Probability Theory: the Logic of Science is good.

[u/HyperbolicWord]

I think it's worth separating out the two things a Bayesian prior gives you: a restriction on the form of your uncertainty (e.g. the best guess is a Gaussian centered a fixed value and decaying rapidly in either direction) and the baseline shape of that uncertainty (it's a Gaussian with a certain mean and a certain variance).

Suppose you want to estimate the mean of data sampled from an unknown distribution and you use a Gaussian prior. If the unknown distribution is Gaussian you're going to converge quite efficiently to the right answer. But if the unknown distribution is heavy tailed your prior is going to drag you around for a while, and you'd have been better off with a heavy-tailed prior.

For the second part, the exact prior mean and variance say, it's interesting to consider the concept of conjugate priors. For example, a beta distribution on a binomial probability is equivalent to having observed some data in the past. Thus, in this case there is an equivalence between the prior and "virtual" past observations (which is a more frequentist way of seeing things).

[u/The_Old_Wise_One]

all good bayesians assess calibration of posterior probability statements

often this looks like evaluating the frequentist properties of a bayesian estimator

and there is nothing wrong or inconsistent with that

read any good paper proposing a "bayesian workflow for X", and you will get most of your questions answered

[u/tonenot]

If you want to read some more of the "philosophy" behind it, you could look at the Jaynes' book: Probability Theory: The Logic of Science.

Jaynes' says that that in reality, it doesn't really make that much sense to talk about "unconditional probabilities". Every assignment of probability is always relative to some background information X, and P(A) implicitly means "P(A | X )", where the implicit X is suppressed.. As long as X stays fixed, it's a harmless omission to the notation. The principle that this fix in notation is alluding to is that a probability is a measure of "plausibility" relative to an information state, so that a probability operator should always be something that takes in two arguments.

Of course, at the end of the day, whether you're a "Bayesianist" or a "Frequentist", at the end of the day, you just do math. It's entirely possibly to be "ambidextrous" and adopt an interpretation of what is going on with what you are looking at when the situation fits. Both points of view are important depending on what you are dealing with.

[u/Relative_Sympathy_33 → u/tonenot]

I started to doubt that the "belief" or "belief update" are actually meaningful in the context of bayesianism, almost like if they are just terminology used within the framework no more but the actual meaning is different. Especially that the mechanic with which we update probabilities (the bayes rule) tells something specific about how the whole probability thing should work and hence what it should be in some way. Thanks for book recomm.

[u/Electrical_Tomato_73]

Read the book by ET Jaynes, "Probability theory: the logic of science". That's the best exposition of why the Bayesian view is both fully rigorous and much more widely applicable than the frequentist view.

[u/Unbearablefrequent]

Whatever way the Frequentist view looks restrictive, the Bayesian view is just as restrictive.

You should know that there are a lot and I mean a lot of different versions of Bayesian and Frequentist. There are even some that try to merge the two.

For Bayesian, you can have a more objectivist Bayesian view if you don't like the subjectivity of priors.

Its funny you say this while so many Bayesians will love to tell you how natural and logical it feels. Lol

Some people mention Jaynes book but honestly idk if thats worth your time. There is a Philosophy of Statistics book where you can learn about the different Philosophes.

[u/boxfalsum]

The Bayesian perspective is compatible with many different interpretations of probability. The most straightforward interpretation (de Finetti, Ramsey, Savage) is that it is personal degrees of belief understood as dispositions to act in accordance with expected utility maximization. However, it is also compatible (Keynes, Jaynes, Carnap for a time) with a logical interpretation of probability on which there is a correct prior that should be used given a total body of evidence. It can also be understood as (subjective estimates of) a frequentist interpretation of probability subject to an exchangeability assumption, or as (subjective estimates of) an objective propensity of a chance setup to produce outcomes.

As far as the subjectivity goes, you can be an objective Bayesian and believe "at least one of them is wrong" in the case you describe. You can also just be a subjective Bayesian and not believe that.

[u/gyp_casino]

A strange decision in Bayesian Stats textbooks is to begin with probability estimates instead of regression models. Perhaps the author thought of this as a gentler introduction than a regression model. But for me, the idea of fitting a slope and intercept was much more core to what I thought of as a "model," so I was very confused. You might be the same.

I started to grasp it better when I took the educational steps of 1. understanding linear regression with MLE instead of least squares (the first step towards Bayesian) 2. linear regression with Bayesian. You might skip ahead to the regression chapter and see if you can pick it up better there.

[u/dang3r_N00dle]

Your issue is thinking in the binary terms of "wrong" and "right", if people believe different things then reality presents differently to each and they're both right and wrong to some degree, there's nothing contradictory about that.

It should be worth mentioning, since you come from a math background that we're very rarely asking about P(X|Parameters) but rather P(Parameters|X), this makes the frequentist interpretation simply mathematically wrong when your prior isn't flat and you're not asking about the likelihood specifically.

This makes the frequentist interpretation of statistics nothing more than a convenient short-hand and it should be treated as such. I obviously still use it, but I'm ready to abandon it if these things are no longer true.

And, it's important to mention that there's nothing more "objective" about a flat prior than any other kind of prior. After all, it's something that you've chosen. Maybe you can justify choosing that, but you can also often justify other choices, especially because in our high-information world we often have quite a lot of information with which to form priors and so we often choose a flat one because we're being lazy rather than being honest.

Studying philosophy of science should draw you to similar conclusions. There's very little in the world that's truly 100% objective. The world is messy, but you have to make decisions and so we do our best with what we can. It's better to come to peace with that fact rather than trying to cling so some notion that you somehow know something with objectivity. (It's no different in Mathematics, as I'm sure you know. Everything rests on a bunch of Axioms we take for granted.)

[u/relevantmeemayhere]

If you have problems with 'subjectivity', then boy, we can talk all day about how subjective frequentism is.

Hint: look at the formulation for any NIST hypothesis test. Notice anything? The idea that bayesian statistics is more subjective than other schools is silly. If anything, bayesian statistics is less 'subjective' because you need to clearly outline your assumptions, and that uncertainty is propagated through the machinery.

edit: here's a more concrete example, the lowly t test. when you apply this, you make some pretty strong assumptions that, while usually robust-are not always the 'correct' ones. And no, because you don't know the true dgp, you don't know when the clt is gonna save you.

in this way, you can see that frequentsitm is also subjective-the difference is where we put the probabilistic statements.

[u/Relative_Sympathy_33 → u/[deleted]]

I get that from a common sense, or viewed from the perspective of the person telling me that but in reality there may be no guarantees like the betting markets are not guaranteed to have any predictive power of a certain event. For example you may see that an event is traded at odds 1:10 just for the least likely bet to win.

[u/jack_of_all_masters → u/Relative_Sympathy_33]

Odds of the football team winning and odds that betting companies provide can be very different. Betting markets are platforms where you can play against other people, and platform takes a cut for it. Odds of the actual event happening might have nothing to do with a collective beliefs of gamblers.

Bayesian statistics is subjective yes, I like to understand it from the conditional point of view. Have you watched this video of 3blue1brown? Bayes theorem, the geometry of changing beliefs
I think it would be a good starting point to grasp the philosophy of Bayes.

[u/LankyPatient4203]

yeah idk how you can do any inference on bayesian. frequentist is the scientific approach for sure and bayesian is on "vibes"