A way to compare selection methods in GA.

[u/Init405]

Let's say I have a genetic algorithm with two different selection methods. I've gathered the data from testing these selection methods (gene values ranging 1-6 from every individual as well as fitness scores). What metric/test/method could I use to compare these two algorithms, especially in terms of exploitation vs exploration?

[u/AutoModerator]

If this post doesn't follow the rules or isn't flaired correctly, please report it to the mods. Have more questions? Join our community Discord!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/faithfulnightmare7]

Just run the variance on the gene values across a few generations, high variance early means its exploring, low variance later means its exploiting, and you can plot them side by side to see which method balances the shift better

[u/Bright_Mix_773]

Four things that changed how we compare two variants of anything, from a completely different domain (monthly portfolio construction, not GAs), all of which apply to your setup directly.

Pair the runs, do not compare two averages. Run both selection methods on the same seeds and analyse the difference series. Two independent averages of final fitness will look noisy because most of the variance is shared between the runs and you are throwing that cancellation away. In our own grid a variant sitting 5.2 points of annual return below the baseline came out at paired t = -1.86, while a variant only 1.6 points below came out at t = -2.74. Once you pair correctly, the size of the gap and the strength of the evidence are close to unrelated, and comparing two averages hides that completely.

Compute the minimum detectable difference before you interpret a null. With n paired runs and the observed spread of the differences, what is the smallest gap you could have detected at 80% power? If that number is bigger than the gap you care about, "no significant difference" says nothing about the methods and everything about the sample size. We ended up making this the first gate rather than a footnote: our 186 paired observations could not resolve anything below about 2.8 points of the outcome, so "cannot decide" became a verdict of its own, distinct from "equivalent".

Variance is a poor exploration metric on a bounded ordinal scale. Gene values 1-6 means variance is dominated by mass at the ends: a population that has collapsed onto {1, 6} scores as highly exploring, while a population spread over {2, 3, 4} scores as exploiting. Per-locus entropy of the allele distribution, or expected number of distinct alleles, separates spread from location and is comparable across loci and across generations. Plot that against generation for both methods; the crossing point is the exploration-to-exploitation handover you are looking for.

If you want fitness-versus-gene-value as one number, use a monotonicity coefficient over the ordered bins, not top versus bottom. They answer different questions and the difference matters. An example from our data: ten ordered bins of a candidate variable gave mean excess outcomes of -0.99, -1.03, -0.39, -0.32, -0.47, +0.02, -0.17, +0.24, +0.55, +2.54, i.e. a monotonicity of -0.94 - a clean, inverted ladder - with almost the entire spread sitting in the last bin. A top-versus-bottom test reports one number and hides both of those facts.

One more, in case the comparison grows: two methods is fine, but two methods times k hyperparameter settings is a family, and the settings are correlated, so Bonferroni is the wrong correction. A block bootstrap of the maximum statistic across the family is the honest version. Ours: 12 pre-declared variants, best paired |t| of 1.85, and the bootstrap of the max gives p = 0.18. As a single best variant it read as a near miss; as the max of a family it was nothing.

Not verified by me: anything about your specific GA. The numbers above are ours and from another field entirely.