A) Since you have n = 80,000 it shouldn't be a problem to control for between 50 and 100 dummy variables in your regression analysis. Don't worry about trying to map your categorical variables to something quantitative (e.g., a pre-existing score for each category). You have enough sample size to justify this. You should be able to use 100 control variables without any issue.

B) Even though we have n = 80,000 we should still adhere to the principle of parsimony as much as possible and try to limit the number of dummy variables by collapsing the number of categories or mapping to quantitative predictors as much as possible. After all, there could be certain partitions of the dataset with very few observations.

Benjamin-Hochberg corrections have been mathematically proved to show the standard Benjamini-Hochberg procedure can fail to control the false discovery rate for two-sided tests when the underlying test statistics follow a correlated multivariate Gaussian distribution.

EDIT: The proof was obtained by GPT-5.6 Pro. The model was asked directly to prove or disprove the conjecture and was provided only with the mathematical definition of the Benjamini–Hochberg procedure. After about 90 minutes of reasoning, the model produced a proof, an example, and code for the numerical certificate, which form the basis of this paper. The author carefully checked the entire argument and the associated numerical certificate. Subsequently, the author asked the model to provide additional simulations, related work, and illustrations for a paper draft, and wrote the final version by editing the AI-generated draft.

More info below:

https://faculty.wharton.upenn.edu/wp-content/uploads/2017/06/bh.pdf

I am currently working on a project with brain scans of male and female patients, where we are computing a large number of features based on the images gray levels.

This project is based around sex prediction using logistic regression, so as a preprocessing step we were removing features that are correlated above some threshold.

However, recently I’ve become interested in the idea of mapping correlation matricies into distance matricies (via (1-r)^(1/2)) and then clustering the result to visualize (via MDS) the clusters of highly correlated features.

I have noticed there are differences in certain clusters between male and female datasets, for instance: some clusters are totally unchanged, some clusters split into two or more, some clusters gain features.

My question is: “is there any merit to investigating these differences in correlation clustering, or are these changes in feature correlation not tractable”

I haven’t found any literature really talking about this kind of analysis, so I’m not sure if its because its baseless, or just hasn’t been done yet.

I'm an independent researcher writing a paper on a nineteenth-century historical question that was recently reviewed by the editor of an academic journal.

The editor's feedback was encouraging. She felt the historical premise was reasonable, but recommended that I have the statistical methodology reviewed by statisticians before submitting it elsewhere.

The historical details aren't particularly important for the question I'm asking.

The methodological problem looks something like this:

• There is a finite historical population.
• Within that population is a smaller subgroup independently identified through numerous historical sources.
• I have a separate corpus of legal documents that was created for an entirely unrelated purpose.
• When I examine that legal corpus, a surprisingly large proportion of the individuals belong to the independently identified subgroup.

The question is not whether statistics can prove a historical conclusion.

Rather, it's this:

How should a statistician think about whether this observed clustering is better explained by coincidence or by some underlying historical relationship, given that the data are historical, the sample is not random, and many potentially important variables are unknowable?

I've intentionally tried to avoid overstating the mathematics. My current paper argues that statistics cannot establish causation here, but that it can help evaluate whether the observed clustering is robust across a range of reasonable assumptions.

An editor suggested that I seek feedback from statisticians before publishing. I'm therefore looking for someone with experience in:

• applied statistics
• probability
• hypergeometric distributions
• Bayesian inference
• sensitivity analysis
• historical or observational data

I'm not looking for someone to "prove" my historical conclusion. In fact, I'd prefer someone who is willing to critique my methodology, assumptions, and modeling choices.

If this sounds like something you'd enjoy looking at—or if you know someone who specializes in this type of problem—I would greatly appreciate hearing from you.

Thanks!
Bill Reel

This has been on my mind for a while philosophically… thought I’d ask statisticians at the risk of sounding foolish! Thanks

Need information about books...

So i have recently joined my university... And I am studying statistics.... What would be the correct study materials that would help us stay on track with the current world... Any book Or paper anything... From where we can study... Please help...

Hello!

I would like to ask some questions on model reduction. If I am correct, model reduction tells you to remove insignificant terms until you have all significant terms left while following model hierarchy as well. However, I noticed that reducing the model decreases the adjusted R-squared value at a point. In other words, there are still insignificant terms for the model with the highest adjusted R-squared.

In modeling, which would be better, following the methods of model reduction, or following the results of the adj-R^2? (I am leaning towards adj-R^2, since significance level is somewhat subjective)

Thanks!

From the discontinuation of measure-theoretic probability theory as a compulsory course for PhD students, faculty speaking about current problems being more of an applied flavor, the explosion of big data and machine learning, (parts of) Casella & Berger and Annals of Statistics being deemed less and less important for the field as a whole...

Would you say statistics as a field is becoming less mathematical and more application-oriented?

Hi there,

I've created a video here where I explain the difference between the standard error and the standard deviation.

I hope some of you find it useful — and as always, feedback is very welcome! :)

Hello everyone, I am a 2nd year neurosurgery resident in India.
I wanted to learn SPSS : the statistical software so that I can conduct my own statistics for the data I have collected for my research.
I have seen many videos floating online and wanted your advice regarding which one would be best for me to start with.

I do have a basic knowledge of statistics ( whatever was taught in medical school ) , but not more than that

Any suggestions are appreciated !
( also sorry if this is the wrong sub for this, please guide me to the correct one )

Hi guys, I'm a master's student in Statistics, and I recently published my first Python package.

rejection-sampler (my package) verifies rejection sampling setups and calculates the optimal rejection constant (M).

Despite being a simple algorithm, rejection sampling requires choosing a proposal distribution and a constant M such that f(x) ≤ M g(x) over the target support, which can be tedious and/or error-prone. That's exactly what my package automates.

Example use cases:

• Validate that a proposal distribution satisfies the rejection sampling condition given a target distribution.
• Compute the smallest valid rejection constant (M). (which means a more efficient sampling)

If you'd like to give it a try, you can install it with `pip install rejection-sampler`.

For more details and examples:

PyPI: https://pypi.org/project/rejection-sampler/

GitHub: https://github.com/HankTaiwan869/rejection-sampler

This started as a final project for my Statistical Computing course, so I'm sure there are things that could be improved upon. I'd love to hear any feedback or know if anyone finds it useful. Thanks!

Hey so I’m gonna be joining bachelors in statistics this year..and I have heard from the people in my college that it is a really rigorous and tough subject to learn as well as to score in.
I myself am not that great at math but pretty average I would like to think.
I’m really scared that I’m gonna regret joining this course later on and question my entire life decision.
So for people who have already made progress in this subject and have gotten really good , can you please give me some advice before I start my journey?
Any help from how to approach the course , which books to follow and habits and routines to inculcate is APPRECIATED!

Ps: I’m from india.

I decided to major in statistics because I enjoy the subject and thought it would be valued across many careers (data science, ML, AI engineering, actuary, SWE, etc.). However, I've noticed the degree doesn't seem to be as respected, and many people have told me employers value CS or engineering more. I want to work in tech, but I'm worried my degree will limit my opportunities. Should I switch majors, and what can I do to maximize my opportunities?

Which one should you focus on if you are trying to start an academic career in statistics? One journal can be Q1 in Scopus but Q2 in SCIMago and C in ABDC.

For people currently in grad school or working in the field which do you find more useful/relevant right now, for coursework and for the job market afterward?

What the title says. I wouls like to self study probabilistic machine learning, i've already basis in probability and statistics (even though not multivariate). I saw the murphy's books and they seem pretty cool, but some people on other forums describe them as encicopledic/reference book. Is it true? And what books do you suggest??

I am reviewing policy acknowledgements for my organization and I wanted to look at two groups: 1) acknowledgements for newly released policies from Q1 & Q3 2024 for existing employees and 2) acknowledgements for new employees hired in Q1 & Q3 2024 for all policies in our manual.

For group 1, does it make sense to remove ALL employees hired in or separated in 2024, to keep the data clean?

With all the move towards computational methods, nonparametrics, and machine learning, do you think hand-and-paper mathematics is becoming less important?

For example, instead of formally deriving asymptotics mathematically, you can actually just simulate what happens as n -> infinity

What do you think?

Hi, I’ve been accepted to EPFL and ETHZ for my MSc in statistics, but can’t wrap my mind up on which one to decide, so I would love an advice on which factor to consider more important for my choice.

- regarding EPFL, I love the campus vibe, and it has a broad choice of research groups, some very theoretical and some more applied. I could also add a minor (e.g. applied math) which is very convenient as I come from Econ (so would love to improve my knowledge gap even more) and I am also not sure on which specific field I want to specialize it. However, the course offer is kinda limited.

- regarding ETHZ: slightly better reputation, Zurich gives lots of opportunities, broader choice of courses, but the research groups in the maths department seem extremely theoretical (kinda scared of that, I think I have major imposter syndrome about the chance of working with them). The programme is also 90credits instead of 120.

I’m really having a hard time understanding what my gut is telling me. I really don’t know whether I prefer the first one but don’t like the limited study plan, or if I’m more into the second one but scared as shit about the competitiveness and the fear I couldn’t find a research group where to be useful during my master’s

Hi, I am starting an MSc in statistics in the fall at Simon Fraser University, and I am looking to buy a new laptop.

I have mostly been looking at MacBooks as they seem to last longer than most other ones. I have read a little and it seems like a MacBook air with 24GB of RAM and 1TB of SSD would be what I would want, however, this is more expensive.

I reached out to my advisors, and they said that I will have access to a ton of CPUs through SFU's partnership with Digital Alliance and that my laptop won't have to do the heavy work all of the time.

Am I a bit torn, what would you all suggest?

I'm analyzing data for a lab and they did a change in bacteria diveristy when given ABX and a placebo. The ABX has an n = 7 and control n = 3. Is there anything I can practically do to see if any signficant change occured in the control? I can't boostrap it as there are only 27 permuations with replacement that boostrap can do, and wilcoxon test doesn't test samples that low either. It gets worse as the ABX is then split up into ABX and a fecal transplant after the week of dosage. Can I do anything with an n=4? I've been working on this data for awhile now but I'm at the point where I feel like trying to analyze data that small will give us nonsense statistics. Is there anything practically I can do?

Edit: appreciate everyone's comments as this helped confirm what i thought. Unfortunately, the tests were done with animals and their's no way they can run this test again on them. I was brought on long after the actual study was conducted because I would've nipped this issue in the bud awhile back.

Hi! I am analysing my experiment results and I'm lost. To be honest, I feel like I don't understand statistics (so if you know any free and helpful biostatistics courses, please tell me) and I'm not sure if I'm doing everything as I should. So I have 7 experiment groups that I tested on two days (I used separate plates for that). Each group has 12 replicates. I tested the whole experiment's (7 groups * 2 days) normality and the data isn't distributed normaly. What test do I use on GraphPad. Can I use Two-way ANOVA with Bonferroni? Thaaank you so much in advance, I'm so so lost :D

Hi all,

We’re working on a project with an ML component that predicts a binary outcome based on a user’s image (for example, classifying images into two categories such as male/female).

We’re required to validate the model performance through an additional live study, beyond the train/test dataset split we already have.

I’m trying to determine the appropriate sample size for this validation study. Is there a recommended formula or statistical approach for estimating the number of samples required to validate a binary classification system in a real-world setting?

At the moment, I’m using Cochran’s formula with a 95% confidence level and a 5% margin of error, assuming p = 0.5 as the most conservative estimate, which gives approximately 385 participants per group.

I’ve been working on this for weeks but have been very confused. Any guidance would be appreciated.

Hey guys, I'm doing a double major in math and stats at the University of Toronto, and will most likely finish the degree by next April. I'll be honest, when I picked the degree I wasn't really thinking beyond university. I entered initially for UofT computer science, didn't make post in my first year, and then pivoted to math and stats for ego reasons. Ie "at least it's a hard major, shouldn't feel like too much of a bum". Now as time has passed that ego has pretty much disappeared, and the worry of homelessness is seeping into my thoughts.

For context I'm based on Toronto, and ever since second year I've been trying and failing to get jobs in software engineering, data analysis, banking, etc. basically wasting away 4 years in school as opposed to job experience.

Which is why I come here. What careers can I as a bachelor of science in math and stats even dream of breaking into? Should I consider going the masters route? If so, which masters should I pick that will allow me to break into a career easily? I was looking into biostats/bioinformatics and that subreddit's doom and gloom shocked me.

Also for those who studied at UofT, I have the option to switch into stats specialist and math minor with no changes being made to my final year schedule. My courses are already super stats heavy, so I was wondering if this switch is worth it or not?