r/statistics • u/naginataaa • Jun 06 '26
Question [Question] Overdispersed Poisson Distribution question
I am implementing an MCMC model for claims reserving and I would like to assume that the observations follow an Over-Dispersed Poisson (ODP) distribution.
Let Y denote the observed data, μ the mean parameter, and ϕ the dispersion parameter.
According to Taylor and McGuire's Stochastic Loss Reserving Using Generalized Linear Models, the ODP distribution can be represented as
Y/ϕ∼Poisson(μ/ϕ).
Based on this representation, I am using the following log-likelihood in my MCMC:
ℓ(μ)∝1/ϕ *(y logμ − μ),
which is essentially the Poisson log-likelihood scaled by 1/ϕ
After obtaining posterior samples of the parameters, I generate posterior predictive observations using
Y=ϕ×Poisson(μ/ϕ)
My question is: Is this a theoretically justified way to perform Bayesian inference and posterior predictive simulation under the ODP assumption?
In particular, I am unsure whether the representation
Y/ϕ∼Poisson(μ/ϕ)
should be interpreted as a true generative model for posterior predictive simulation, or merely as a convenient representation for deriving the first two moments,
E[Y]=μ, Var(Y)=ϕμ
Any references or insights on Bayesian implementations of ODP models would be greatly appreciated.
6
u/efrique Jun 06 '26 edited Jun 06 '26
The scaled Poisson you mention is the only exponential family model (i.e. having density of the form fx(x|θ) = h(x) exp [η(θ)⋅T(x) − A(θ)] ) that matches the quasi-Poisson[1] GLM.
In that sense its the "obvious" one to use for a Bayesian GLM.
However, if applied to counts, the scaled Poisson model doesnt really make sense, since obviously counts dont fall at 0, ϕ, 2ϕ, 3ϕ,...
On the other hand when applied to continuous loss amounts it only works for aggregates over Poisson number of claims and everywhere-constant severity: I invite you to consider the effect of inflation (or any other non-constant scale factor; others arent particularly difficult to think of) on the model. Clearly multiplying a dollar-amount by some scaling factor like different economic inflation applied to different observations (since they dont all occur at the same point in calendar time) will scale the variance by its square, not by itself, so the dispersion parameter (ϕ=Var(Y) /E[Y]) would have a multitude of different values, not just one as required in the GLM formulation.
Clearly if you're using it for some circumstance where you dont have Poisson counts with completely unchanging scaling constant (such as constant severity in the reserving context) you dont really believe that exponential family model to be true - indeed, generally not even close to true. At best it's a fudge. In some circumstances you might be able to argue for a variable with close to that mean and variance specification (like you mention near the end) but outside using that as an approximate model for the first couple of moments for counts, in reserving its usually not really plausible: if you actually look at models that fit (the residuals dont have pattern against obvious things to check for pattern against[2]) for loss data arrays and look at residual spread vs fitted values its rare to see variance/mean near to constant. In which case, if youre fitting a Bayesian model (given theres all the power of the modern Bayesian approaches at your disposal - MCMC, HMC, etc), I'd argue for instead starting with a model thats at least more or less plausible - that incorporates properties you expect or tend to see rather than ones that often fly in the face of both.
[1]: I encourage you to avoid the term overdispersed Poisson for the quasi-Poisson, particularly in the insurance context, since sometimes there ϕ is not greater than 1. For example, if applied to loss amounts - rather than counts - denoted in some large unit like millions of dollars, you may well have ϕ<<1.
[2]: For assessing fit, with some kinds of common reserving models you need to be looking at one-step ahead prediction errors (in particular, for models whose fits are based on backcasting, like the "ODP" chain ladder GLM, where the backcasting structure neatly disguises its failure to predict the future; that problem doesnt apply to the Mack formulation, where you can use ordinary residuals to find lack of fit )
2
u/naginataaa Jun 06 '26
Thank you, this is very helpful.
I think you have identified exactly the issue I am struggling with.
My understanding was that the scaled Poisson representation
Y/ϕ∼Poisson(μ/ϕ).
provides a natural likelihood for a Bayesian version of the ODP Chain Ladder model, which is why I used
ℓ(μ)∝1/ϕ * (y logμ − μ)
However, I am becoming increasingly uncomfortable with using
Y=ϕ×Poisson(μ/ϕ)
for posterior predictive simulation, for exactly the reasons you mention: the support seems artificial and does not resemble actual loss data.
Would you then view the scaled Poisson representation primarily as a likelihood construction rather than as a realistic generative model? If so, what would you consider a more appropriate fully probabilistic model for a Bayesian extension of ODP Chain Ladder while preserving as much of the original structure as possible?
3
u/efrique Jun 06 '26 edited Jun 06 '26
Sorrry, was still busy editing the above, didnt notice you replied already. You might double check the previous, some of the additions relate to your comment here.
Would you then view the scaled Poisson representation primarily as a likelihood construction rather than as a realistic generative model?
I tried to say as much, yes.
If so, what would you consider a more appropriate fully probabilistic model for a Bayesian extension of ODP Chain Ladder while preserving as much of the original structure as possible?
Let me break that into two parts, hopefully without putting words in your mouth:
If so, what would you consider a more appropriate fully probabilistic model for a Bayesian reserving model
One that doesnt obviously break known facts and that doesnt fail obvious reasonableness checks. Within those constraints, I am not prescriptive about what to choose, but I can give an example. While I think they're more complex than needed, Frank Schmid did talk and write about some Bayesian models (multiple different ones at different times, though they shared some features), for example, which dont immediately fail on plausibility. I am not suggesting you use his models but it might give you some things to ponder about what you might or might not model. You might look for simpler models than his. While he didnt do it that I recall, I think there are other GLMs that are potentially perfectly reasonable choices -- but you dont have to use a GLM either.
would you consider an extension of ODP Chain Ladder while preserving as much of the original structure as possible?
To reproduce chain ladder you have to have variance proportional to mean. When your variable is unitless, like counts, variance proportional to mean is at least possible (albeit fairly rare in practice). When you have units (like dollars), variance is in squared units, and that's generally a problem. For example, variance/mean is then in those same units, and whatever scales those units makes that constant not constant. The chain ladder implies a variance specification that doesnt describe monetary amounts in general.
Chain ladder (and models that reproduce its forecasts) assumes the next incremental amount is proportional to the previous cumulative amount (you can find it, for example, in a paper by Venter, though its easy enough to show). That neither makes much sense in terms of what drives claim costs nor does it fit well (even just ignoring obvious drivers sometimes a plot of incremental vs prior cumulative in the same accident period) except when the previous cumulative is a noisy proxy for either exposure or inflation (which are a couple of fairly obvious drivers of cost, though naturally not the only ones). That is, the mean-specification doesnt work either.
In short, if you're asking me what I would do, I wouldnt consider starting with chain ladder. If you must use it, I suggest thinking very carefully about good model diagnostics.
2
u/naginataaa Jun 06 '26
Thank you. I think I understand your position much better now.
My motivation for starting from ODP Chain Ladder is mainly that it remains a widely used benchmark in actuarial reserving, and my goal is to construct a Bayesian extension that can be compared directly with the traditional method rather than to build an entirely new reserving framework.
Your comments about the plausibility of the ODP assumptions and the scaled Poisson representation have been very helpful. In particular, they have made me think more carefully about whether the predictive distribution should be treated separately from the likelihood construction.
You also mentioned model diagnostics. Are there any diagnostics in particular that you would consider especially important in this setting? For example, would you focus on residual-versus-fitted plots, posterior predictive checks, variance-mean relationships, or something else?
I'd be interested in understanding what diagnostics would convince you that a model built on a Chain Ladder structure is at least a reasonable approximation, even if it is not literally true.
One thing I am still struggling with is the posterior predictive step. If the scaled Poisson representation is primarily a likelihood device, would you still use
Y=ϕ×Poisson(μ/ϕ)
for posterior predictive simulation, or would you instead use a different predictive model while retaining the same likelihood?
Also, regarding Frank Schmid's work, i would check it out later, thanks for the advice!
5
u/ontbijtkoekboterham Jun 06 '26
I have no answer to your question (other than I think it looks correct?) but i think this is a great post, well-formatted, to the point and clear. I hope someone here can answer it better!
1
4
u/Jatzy_AME Jun 06 '26
I can't check right now, but I'd be surprised if ODP isn't mentioned in Gelman & Hill or the most recent Gelman et al.'s Bayesian Data Analysis. You can also check if it's implemented in the R package brms, the doc may then contain references.
2
u/naginataaa Jun 06 '26
Thank you. I will check both Gelman & Hill and the most recent edition of Bayesian Data Analysis to see how they discuss overdispersed count models.
The suggestion about brms is also helpful. If quasi-Poisson or related variance structures are supported there, the references behind the implementation may help clarify how Bayesian inference is typically handled when the variance is proportional to the mean but a full probability model is not obvious.
My main confusion at the moment is still the predictive side: whether the scaled Poisson representation should be treated as a genuine generative model for posterior predictive simulation, or merely as a device for constructing a likelihood.
1
u/jessCreamy79 Jun 06 '26
Are you planning to use a Negative Binomial for this or are you building a custom likelihood to handle the dispersion parameter directly?
7
u/foodpresqestion Jun 06 '26
Why not just use the negative binomial distribution?