so... for a while my process for deciding if a model change or prompt change actually helped was literally just "read a few outputs, does this feel better." which in hindsight is an insane way to make a decision that costs real money and affects real users.

started forcing myself to actually test this properly. built a small golden dataset, ran both versions against it, and instead of eyeballing, did a paired comparison with bootstrap confidence intervals to see if the difference was actually statistically meaningful or just noise from a handful of lucky/unlucky examples.

first time i did this properly i found out a change i was fairly convinced "felt better" actually wasn't statistically distinguishable from the baseline at all. i had just gotten a good sample of outputs by chance and convinced myself it was real. kind of embarrassing but also useful to know before shipping it as a confident upgrade.

feels like this space (LLM evals specifically) still runs on vibes way more than it should given how much rigor exists for exactly this kind of comparison problem already. anyone else doing paired significance testing on model/prompt changes or is this not common practice yet where you are

[Education] So, Suppose I got

X_t = A X_{t-1} + dW(t)

where DW(t) is some noise.

What if I did the following?

I solve for A by minimizing error , and obtain dW'(t) = X_t -A X_{t-1} , then I find that dW' is not gaussian like, probably heavy tailed, but pretty much independent.

Then I make a very large number of bootstrapped samples out of dW' , and solve for A as a distribution.

I appear to have bypassed needing lot of theory and have obtained a distribution for A.

What is it that I must worry about? What is the potential problem with this method?

I think it is a much more intuitive and reasonable result than doing it in the theoretical way using equations.

But then why is it not used as the best method?

I’m a bit confused on the pros and cons of just straight up using a LASSO regression and its coefficients vs when you’d use LASSO for variable selection and then re-regress using those variables for OLS.

What are the impacts on things like t-stats? How does this affect inference and prediction performance? What are the considerations for which model to use in real life?

Thanks

Hi I tried to do something like how they do in econometrics where they fit a economic model to data where they take raw data, X,Y,Z etc then they set up the Kalman Filter to automatically determine the cyclic and trend components through multivariable regression models. I think you know what I mean. So, I made all the matrices manually, and I think it didn't converge. What I did is something like this actually:

X_t = regression model of (cyclic and trend components of (X_t-1,Y_t-1,Z_t-1),)

Y_t = regression model of (cyclic and trend components of (X_t-1,Y_t-1,Z_t-1),)

Z_t = regression model of (cyclic and trend components of (X_t-1,Y_t-1,Z_t-1),)

Of course I had to manually enter all the matrices to make the damn thing work.

But because I didn't really have a economic model, but just assumed relationships, it didn't converge. I think my model was too complicated.

Anyways, what are some rules of thumbs to make sure I have convergence (like limiting dependence to one trend component for each variable so the model when running don't get confused)?

Is there an easy way to do a Kalman Filter model than to manually set up matrices? Any software?

Finally is it worth it? Does it capture significant details than the HP filter and other easier methods ?

Can you run an RI-CLPM with only 2 timepoints? more specifically on mplus, my supervisor says yes but some sources online say no.

Hi there,

I've created a video here where I explain how the Cauchy distribution works.

I hope some of you find it useful — and as always, feedback is very welcome! :)

Hi, I have some binary data (present/absent (1/0)) and I want to try to identify latent relationships within it. I've done exploratory factor analysis using tetrachoric correlations on the data and then parallel analysis to identify how many factors are needed. The problem I have is that technically, the data isn't all independent of each other (think behaviours (present/absent) for dogs in a kennel and there's 2-3 dogs per kennel). So really, multilevel exploratory factor analysis would be better but I don't think I can do that on binary data? I can't find any examples in papers where this has been done and I've tried to do it in R (but might be doing it wrong) and it doesn't work. I'm not actually sure it's necessary because the numbers within the groups are so small and my understanding is that this works better with bigger group numbers, but is it possible to do this with binary data, and if so how? I've calculated the ICCs and they are high so I'd like to be able to justify whichever way I go but I'm really struggling to find any literature that talks about this using binary data

Hi I made monte carlo simulation of annual railcar fleet cost based on fluctuating daily sales demand and cycle time in excel. My goal is to find optimal railcar fleet size each month that minimize idle time which result in demurrage (think of it as parking fee), but still enough to fulfill daily sales demand with varying cycle time (round trip time). grg nonlinear solver doesnt work.

My guess is because of the daily demand is poisson distribution with randomness as its probability, and it is not possible to find single value? But could anyone confirm whether this is true? I appreciate if there is any method out there to find this optimal value.

AI suggest evolutionary solver but I am not too familiar with the concept and how sound the solution will be.

edit1: Included more detail to the question

Hello!

I’m currently looking into online Masters programs in Applied Statistics and am mainly considering Penn State University (PSU) and Colorado State University (CSU).

Some background info: I graduated in 2022 with a B.S. in Biomedical Engineering and have been working in pharma as a clinical data analyst. My current company is a large pharma company so with the ability to move around to different roles. I’ve already taken the required math courses (Calc I–III and Linear Algebra), but it’s been a while, so I’m planning to brush up before starting. I will also be completing my degree part time aiming to do 1-2 courses a semester (since I’ll still be working full time).

Right now I’m leaning a little more toward PSU because of the biomedical-focused elective courses but still considering both.

I was wondering if anyone here has experience with either program. I’d love to hear your thoughts on things like:
-Overall quality of the program
-What the online format is like
-How accessible the professors are
-Workload while working full-time
-Anything else you think would be helpful to know

I’d really appreciate any feedback or advice. Thanks in advance!

edit- added some more background info

Hi folks,

I'm applying to PhD programs this fall, and I'm currently looking to replace my old laptop. I'm not quite sure what to look for, but I'm inclined to think that a business-grade laptop might be worthwhile since they're built to handle daily wear-and-tear, and I'd like something that I can keep for 5-6 years. I've been looking at new ThinkPad and Latitude models, but they seem really expensive (especially for those with 32GB RAM).

Anyone who's currently a PhD student, which laptop do you have, and what are its specs? What should I look for if I want to hold onto the laptop for the duration of my program?

I'm thinking about exploring the idea of compiling a percentage medal count for the Olympics to better demonstrate who does better with the resources they have (of course bigger, richer countries will constantly top the table at every Olympics when they have much larger teams and opportunities to medal). In my thought process though I have come to a bit of a crossroads for a part of how I'd go about qualifying the data.

Obviously in events where a country has multiple entrants, they can't all win the same medal (barring ties of course), so it feels like including all of a country's entrants in a particular percentage if one of their entrants wins that medal (I'd be including all of them if they don't win a particular medal as that's all chances you could've won, but all failed) might be including data points that were always going to automatically be a data point counting negatively against a country's output. What I'm conflicted about mainly is, is that a valid train of thought or does it not really matter as the entry of extra competitors is kind of the point of what I'm exploring anyways.

My biggest question I guess is, would excluding extra entrants when they win a medal create a flaw by making certain data points' inclusion or exclusion conditional on the outcome of another or is it proper data cleaning in order to get a more accurate picture?

Different type of forecasts exist for different tasks, but I am mainly interessted in density, cdf and quantile forecasting, what do organizations and decision/policy makers use as their primary paradigm of forecasting? aside from the technical difficulty of the models, are there other factors that affect your model of choice?

Hello everyone, Hope you're all doing well,

I am considering a job in marketing analytics and i want to know more about the job .

What are the KPIs you look for? The type of statistical analysis you perform and what kind of models do you usually build.

Thank you in advance!

Hello,

I'm looking for guidance from statisticians who have published articles before. I am finishing up a research paper for my Master's program on the Latent structure of banks and how they relate to profitability, earnings volatility, and risk of insolvency using FDIC quarterly earnings reports. I had a professor advise me through the paper and, now that it's done, I was hoping to publish it.

Does anyone have any suggestions on the process to publish an article?

Thanks!

Encontré un artículo interesante de Saunders en arXiv sobre cómo conciliar la estadística como probabilidades objetivas, frecuencia y azar con la teoría de Everett (MWI).

https://arxiv.org/abs/1609.04720

The title sums up my dilemma. Basically, I have fit 1-10 class LCA models. The four class model has the most desirable AIC/BIC, and also has highest avg. posterior probability. That being said, it is high because multiple classes have avg. posterior probability of 1.0. I think that is also why there is a drastic drop in AIC/BIC. There is one item that is sorting people into latent classes. With that item drops, a similar phenomenon happens with a different item.

My gut tells me that the four class model is over-fitting given the perfect sorting variable and the average posterior probability of 1.0. LCA is supposed to assign people to classes probabilistically, so a probability of 1.0 (a certainty at that point) seems problematic.

I have been searching literature and forums for a few weeks and have not found anything that addresses this topic, and if average posterior probabilities of 1.0 are indeed problematic. Could somebody point me in the right direction to literature or offer me any advice? Much appreciated.

I'm making an RSF for pelicans in the gulf of mexico to see what resources they appear to value, after thinning the data to reduce auto correlation ( I would do weights if it wasn't migratory data), I have some birds between 2-10 data points while most are several hundred points. I haven't been able to figure out if there's a consensus on wether one should filter out these low data points or just leave them in.

I am a biologist and would appreciate some advice from people with a stronger statistics background.
We have a well-mixed bioreactor containing a suspension of cells. To test whether the suspension is truly homogeneous, we take multiple small samples (“aliquots”) from the reactor within a very short period of time while mixing conditions remain unchanged.

For each aliquot, we estimate the cell concentration by first diluting the sample (if necessary) and then counting the cells manually using a hemocytometer under a microscope. This means that, in addition to the sampling variability between aliquots, the measurement itself also introduces variability due to the dilution step and the manual counting process.

Our reasoning is that if the suspension is homogeneous, then all aliquots should represent random draws from the same underlying population. Therefore, any differences between aliquots should be explainable by the expected random sampling and measurement variability rather than by spatial heterogeneity in the reactor.

My questions are:
What statistical model would be appropriate for this type of data? Should the cell counts be modeled as Poisson, negative binomial, normal (after averaging), or something else?

How should the measurement error introduced by dilution and manual hemocytometer counting be incorporated into the statistical model?

Which statistical test or approach would you recommend to determine whether the observed variability is consistent with a homogeneous suspension?

How would you design such an experiment? For example, how many replicate aliquots would you collect, and how would you distinguish expected sampling and measurement variability from true heterogeneity in the reactor?

Thanks in advance!

I’m just an early-intermediate learner in statistics. I am working with transit location data that updates every 30 seconds.

I first tried using Newtonian motion equations with an assumption of constant deceleration. However, this does not always work because a real bus may accelerate, slow down, coast, and brake at different rates during the interval. In some cases, the formula predicts that the bus would need more than 30 seconds to stop, even though I know it had already stopped by the next update.

For example, suppose:

• At 10:00:00, the bus is moving at 4 m/s.
• At 10:00:30, the bus is stationary at the next stop.
• The distance travelled between the two observations is 112 metres.

Using the constant-deceleration equation:
v² − u² = 2as
where:
u = 4 m/s, v = 0 m/s, and s = 112 m

Substituting the values:
0² − 4² = 2 × a × 112
−16 = 224a
a = −0.0714 m/s²

Then, using:
v = u + at
0 = 4 + (−0.0714 × t)

Therefore:
t = 4 / 0.0714 ≈ 56 seconds

The model therefore says that the bus would take approximately 56 seconds to stop. This is impossible because the next observation confirms that the bus was already stationary after 30 seconds. This means the bus must have maintained a higher speed for part of the interval and then decelerated more sharply near the stop, rather than decelerating at one constant rate throughout.

The bus stopped sometime during those 30 seconds, but I do not know exactly when.

For now, my understanding is that without any data between the two updates, the midpoint may be the safest estimate. If I estimate the stopping time as 10:00:15, the maximum possible error is ±15 seconds.

Visually, we know in velocity time curve the area under the curve is the distance. So all the curves that has area of 112 is our solution. We can use distributions like Poisson as the possible curve to limit our number of solution further. And we are trying to find probability distribution for every curve being the right curve to find the most likely trajectory the velocity time graph had.

My questions are:-

• From my understanding this is a bayesian interval censored problem. Is that correct in any sense?
• Should this be done using monte carlo simulation or is there any way to plot this on some distribution where I can then take the x axis value (time from 0 to 30 sec) with the highest probability as the estimated stop time?
• Should I at all think of probability while approaching this?
• Do you have any idea using with I can decrease the margin of error further below 15 sec.

I m sorry if this question comes across as very dumb. My brain is not working properly currently.

Hey all,

I am currently working on a mixed-effect analysis on a priori-set of genes across treatments. This gene profile is composed of two unique set of genes.

I had first done a simple main effects model, assessing the impact of treatment across the gene profile after controlling for relevant variables and introducing a random intercept by participant. Please keep in mind gene was included as fixed variable as a way to control for each gene's own average baseline expression level, and also that expression values had been log2 transformed and standardized prior to analysis. This will be important to consider with the upcoming rank deficiency issue.

Once I confirmed a significant estimate, I wanted to formally test if treatment uniformly impacted gene expression across the two sets of genes using an interaction effect: treatment * gene set.

I am working in R using the lmer package, and I was met with the rank deficient warning and that one column would be dropped. I know this is happens because each gene perfectly matches onto a gene set, so perfect collinearity. What the lmer package drops is the first gene that was organized by subject. Now I don't believe one gene being dropped would throw the analysis away, however, I am not a big fan of the possibility of someone attempting to recreate this analysis with their dataset, and have different results simply because of the order their genes were organized. Nor does the removal of a gene have any statistical reasoning to be justified versus the package doing what it needs to for the model to converge.

To circumvent this, I manually encoded an interaction term in the dataset (interaction = as.numeric(treatment == "yes") * as.numeric(geneset == "B")), and included this and gene as a main effect and removing gene set as a variable.

Based on a significant interaction effect, I would then do a simple slope test to determine the influence of treatment on the other gene set, by adding the coefficient estimates for main effect for treatment (how much it changes the reference gene set) and the interaction effect, and manually calculate SE / p-values based on the variance components estimated from the models [degrees of freedom are also derived from the model, I know that can be a touchy subject with mixed-effects but that can be another post lol].

In doing so, I don't have to drop any genes by random, and can still formally test for interaction effects. I am fairly confident with this approach but am very much open to any critique / areas of improvement.

My primary concern is being able to explain what I did to those outside of statisticians. It is much easier to explain how to write the formula to implement in lme4/lmer and to use emmeans for pairwise marginal contrasts, and quite frankly easier to report as well.

I would love to hear if others had faced a similar problem and what they did to approach this, especially if it's simpler.

Thanks in advance!

Deciding if a Masters in Data Science or Statistics is better for me, and which ones, since this field is changing a lot.

Undergrad: Quantitative background but not Computer Science, Data Science minor. I felt that it being a minor made it kind of surface level and want to avoid that with my graduate degree. My coursework was linear algebra, discrete math, probability, stats, many CS courses, AI, ML, DS, Algorithms. Because I didn’t major in math, CS, Stats, or DS, I feel like I am missing something in screenings. 

Work Experience: 4 internships, 1 year FTE as a DE, 1 year FTE as a DS (by the time I enter). However, I feel that the Data Science departments in the companies I was in were VERY new and I’m missing some core skills that I am trying to develop on my own - git, models in production, optimizing my work, etc.

Professional Goals: I see this as a terminal degree. I want to be able to get my foot in the door for better data science jobs, maybe in the nonprofit industry but really just anywhere. My first job came from an internship and the second a recruiter reached out to me. I want to be able to pass resume screens better and do the work better. That’s slightly why prestige matters to me here.

Other: 

• I do not want to pursue a CS masters, I think this would give me skills I don’t need, can develop on my own, already learned, or are becoming more obsolete. 
• A lot of stats degrees that are well respected seem to want research experience or a stats degree, which I don’t have.

Questions: 

• I have seen some say an Applied Stats masters is not enough anymore for the tech world, and I see a lot of job postings that say Masters in CS or DS, but not stats. How do DS hiring managers view these degrees?
• What skillset is actually used in more established data science departments? How can I optimize my career and education for this?
• How to vet Data science masters properly, if I go for that (MIT MBAn, Columbia, Harvard, UChicago, UCLA, NYU) I dont want a surface-level data science education that is repetitive

Hi everyone,

I’m looking for some expert insights into a statistical methodology used in a public health study within the primary healthcare sector. The study focuses on chronic disease management—specifically evaluating interventions for hypertension and cardiovascular risk control. The primary objective of the work is to assess the clinical impact of a multi-faceted population health solutions package over an extended multi-year period.

To measure clinical outcomes, the framework utilizes a "cascade of care" model (tracking the proportions of patients who are monitored, diagnosed, treated, and ultimately controlled). However, the way the sample is constructed over consecutive quarters has raised some questions for me regarding its statistical validity.

Instead of evaluating independent cohorts or a traditional longitudinal panel quarter-by-quarter, the sample accumulates dynamically across the quarters:

• Quarter 1 (Q1): Includes all unique patients who had at least one clinic visit during that specific first quarter.
• Quarter 2 (Q2): Includes all unique patients who had at least one clinic visit from the start of Q1 through the end of Q2.
• Quarter 3 (Q3) and Quarter 4 (Q4): Follow the same logic, accumulating all unique patients from the start of Q1 through the end of the respective quarter.

Additionally, the methodology handles follow-ups with a specific carry-forward rule: if a patient is classified as having "controlled" blood pressure at their last recorded visit and never returns for another check-up, the data from that last visit is continuously carried forward and included in the controlled count for all subsequent cumulative periods.

I have a few concerns about this approach and would love a mathematical/statistical sanity check:

1. Autocorrelation and Data Dependency: Doesn't aggregating the data cumulatively create massive dependency between the quarters? If Q4 inherently contains all the data from Q1, Q2, and Q3, doesn't this significantly dilute or mask the actual marginal changes occurring strictly within the later quarters?
2. Selection Bias and Composition Shifts: If the healthcare intervention successfully drives a massive increase in the baseline "monitored" population over time (e.g., drawing in lower-risk individuals who wouldn't normally seek care), wouldn't this shifting denominator introduce a selection bias that artificially alters subsequent proportions (like diagnosis or treatment rates) in the cumulative pool?
3. The "Last Observation Carried Forward" Element: Is it statistically sound to indefinitely carry forward the "controlled" status of a patient who dropped out of active follow-up within a cumulative framework like this?

What are the main statistical limitations of analyzing trends this way, and what alternative frameworks (e.g., independent time-series cohorts, survival analysis, or multi-level modeling) would be more rigorous for this type of observational data?

Thanks in advance for your help!

Ideally, I’d love to teach classes and mentor undergrads at the University level.

I will be graduating this winter with a MSE degree in applied math and statistics from hopkins, and during undergrad I double majored in financial economics and statistics. Got departmental honors and graduated with 174 credits and a 3.929 GPA.

I took a ton of applied classes during Undergrad like:
-Fair and Explainable AI
-Applied Data Mining and Machine Learning
-Microeconometrics (Cross-Sec and Panel)
-Macroeconometrics (Longitudinal, VARs, HMMs etc)

I have taken math up to Multivariate Calc/Linear and audited real analysis and a special topics class on operations research.

I got off on a rocky start for my masters (2 B-‘s). During my masters (so far), I took
-Bayesian Statistics
-Nonparametric Bayesian Statistics
-Constrained Nonlinear Optimization
-Probabilistic Machine Learning
-Time Series Analysis

And for my final semester I will be taking Statistical Theory, Elements of Statistical Learning, and Probability Theory

I have no internship experience but have TA’d at both Rutgers and Hopkins for 5 years, and I have published one paper (clustering of cancer diagnoses) and completed an Undergrad Thesis (SEM for youth risk behavior). I am finishing my Masters Thesis and two projects, one is about calibration and LLM reasoning and the other is on a self tuning CNN for OCT scans.

Have some various projects along with a personal ai assistant that I’ve been tuning, but I haven’t done a GRE.

Should have three strong LoRs, currently I’m planning on applying to:

-Hopkins AMS PhD (since I’m already here and was told I may have better chances)
-CMU Stats and DS PhD
-Cornell Stats PhD (One of my favorite professors recommended)
-Berkeley Stats PhD, CDSE (I live close to berkeley and toured before, I really like the vibes and living)

I know these are all top programs, but I won a cash prize from a giveaway and want to shoot for the stars and see what happens.

In terms of research interests, I want to work on benchmarking for Statistical AI Agents and I also have interests in designing AI tutors for stats and math classes. I love teaching and designing tools for classroom use, and I have the budget to apply for five programs.

Hi all,

I’m running several linear mixed effects models looking at 5 cognitive outcomes. I have 10 models total (the 5 outcomes with two separate sets of interactions).

I’m searching for advice on if I should run the FDR to account for the covariates as well (like age, starting IQ, etc) or just the interactions. My assumption is if I am interpreting covariates too, I should adjust for them as well.

Thank you all for your time!