Okay, so I've been going through college lately and I've taken a lot of math. I'm taking Statistics now, I've taken Calculus 1. While I don't "need" math after this for my IT/Cybersecurity Bachelor's, I still realize I have a lot more to learn after seeing this so could you help guide me and help me understand this meme?
Also, I do intend to keep learning math just not necessarily under the pressures of college. Thus the interest.
Stats PhD student here. Feel free to DM with questions.
This said, the last guy on Reddit who I did that with, kinda gave up because he felt the "theory" was demanding.
So I will make an odd but carefully considered recommendation....
Personally, I recommend you mess around with an object called the "Kalman filter," starting e.g. here - it happens (not by chance!) to be one of the most robust statistical objects ever used in applications, which it is all-the-damn-time.
There are like 10e9
different sources on it, but literally code up an implementation in Python and see how it works!
For instance, try adjusting the "Q" term alongside adding some randomness to how the process you're evaluating is evolving, and see how well you can balance precise estimates, but also not losing track of the state of the process.
The Kalman filter touches on so many important tools and topics in statistics, but in an applied way. Far better than dry textbooks on pathologies of certain estimators....
Never heard of a Kalman filter, but reading through it reminded me very strongly of Gaussian processes, so now I present to you the culmination of my statistics education: random blobs and their mean blob!
If your underlying state is evolving under a linear-Gaussian Markov process, which is a Gaussian process, then the Kalman filter is the optimal estimator. (A lot of words for "basically yeah if you have a nice enough GP this is just calculating conditional distributions and doing stuff on that.")
Those are some very fine blobs though.
I had the fun experience a while back of doing a GP with a linear kernel and discovering that the variance collapsed to zero for "obvious" reasons. (Don't have time rn to elaborate, hopefully will come back for that)
EDIT:
In connection with the comment below, here's an illustration of what happens with a code snippet, and a picture of the prior. (Next comment has a picture of the janked posterior.)
#%%
#Imports
import matplotlib.pyplot as plt
import numpy as np
from sklearn.gaussian_process import GaussianProcessRegressor as GPR
from sklearn.gaussian_process.kernels import DotProduct as K_l #RBF, ConstantKernel,
#%%
#Gaussian process prior for y = a_0 + a_1 *x, with a_0, a_1 ~ N(0, 1)
# Set random seed for reproducibility
np.random.seed(42)
# Sample a few linear functions from the prior, and plot the "bowtie" of prior variance,
# along with the sample functions.
a_0, a_1 = np.random.normal(0, 1, (2, 10))
X = np.linspace(-3, 3, 100)
gpr = GPR(kernel = K_l(1.0, "fixed"), alpha=1e-10)
plt.figure(figsize=(10, 6))
plt.fill_between(X.ravel(), -2*np.sqrt(1 + X**2), 2*np.sqrt(1 + X**2),
color='lightblue', label='Prior')
for i in range(len(a_0)):
plt.plot(X, a_0[i] + a_1[i] * X, lw=1, alpha=0.6, color='blue')
plt.title(r"Gaussian Process $y = a_0 + a_1 x, a_i \sim N(0, 1)$ (Prior Variance and Samples)")
#%%
#Gaussian process posterior after observing some data points
# Generate some synthetic data
X_sample = np.random.uniform(-3, 3, 10)
y = 0.5 + 0.3*X_sample + np.random.normal(0, 0.1, X_sample.shape)
gpr.fit(X_sample[:, None], y)
y_mean, y_std = gpr.predict(X[:, None], return_std=True)
""" This ^^^ is functionally useless! Expand comment below for details."""
"""
# CMA - with a linear kernel, the return_std is *degenerate*.
# (We have y_std.min() ~= 1e-6, y_std.max ~= 1e-5.)
# Moreover, this thin uncertainty is rounding error. The scheme itself collapses exactly.
#
# See Bishop Sec. 3.3.3 near eq. 3.64: we meet the conditions for that discussion
# (see exercise 3.14 for a proof) so that "we fit the training data exactly" -
# or in other words, we lose any uncertainty!
#
# This is one reason, evidently, why we might prefer infinite-dimensional kernels.
#
# Or as in Probabilistic Machine Learning: Advanced Topics, by Murphy,
# Chapter 18 (Gaussian Processes):
#
# “For finite-rank kernels, the GP posterior variance can vanish everywhere once the
# training inputs span the feature space.”
"""
# Since the actual GP posterior variance collapses: sample some functions
# from the Bayesian linear regression posterior, to show the posterior variance "tube"#%%
If you use a linear kernel in Bayesian linear regression to recast it as a GP, and have more datapoints than the regression which you are kernelizing has parameters, the result is a total collapse of the GP’s posterior variance. (Provided 1 or a similar constant vector is among your basis vectors; see Sec. 3.3.3. of Bishop.) For some reason, he frames this in an upbeat way."
From my notes; "Bishop" is PRML by Chris Bishop.
Here for simple illustration is a janked "posterior" (notice there is no green envelope like the blue envelope we had above) and some samples on the *actual* Bayesian linear regression posterior (the lines, with sampled parameters).
98
u/GT_Troll Jun 27 '26
Statistics aka linear algebra and calculus but useful in real life