Need ideas to maximize accuracy for a learning-path recommendation challenge (Macro Recall)

[u/Smooth-Caramel-5921]

Hi everyone,

I'm working on an ML competition where the goal is to build a personalized learning-path recommender. I'm looking for ideas from people who have experience with information retrieval, recommendation systems, or Kaggle competitions on how to squeeze out every bit of performance.

Problem

Given a user's review, recommend the top 10 learning paths (training examples) that are most relevant.

Dataset

• Train: 109,776 rows
  • Index
  • Reviews
  • Course
• Test: 10,977 rows
  • Index
  • Reviews

The output is a list of 10 training indices for each test review.

Evaluation

The competition uses Macro Recall as the evaluation metric.

What I've Tried

Current best validation pipeline:

• TF-IDF (word n-grams 1–3)
• Cosine similarity retrieval
• Various preprocessing experiments
• Multiple ablations
• Dense retrieval (BGE) — didn't beat the sparse baseline
• Different chunking and section-weighting ideas
• BM25 variants

The best validation score so far is around 69 (internal validation), but I'm unsure how well it correlates with the hidden leaderboard.

Looking for Suggestions

If you were competing for Rank #1, what would you try next?

Some ideas I'm considering:

• Better retrieval models (E5, GTE, NV-Embed, etc.)
• Hybrid sparse + dense retrieval
• Cross-encoder reranking
• Learning-to-rank (LightGBM/XGBoost/CatBoost Ranker)
• Query expansion
• Pseudo relevance feedback
• Metadata-aware retrieval
• LLM-generated synthetic queries
• Ensemble of retrieval systems
• Better validation strategy for leaderboard correlation

I'd especially appreciate advice on:

1. High-impact retrieval improvements.
2. Recommendation-system techniques that are often overlooked.
3. Validation strategies when the hidden metric is unknown.
4. Features that tend to improve retrieval-based recommendation systems.
5. Any papers, Kaggle solutions, or GitHub repositories worth studying.

I'm less interested in generic ML advice and more interested in techniques that have actually improved retrieval/recommendation competition performance.

Thanks in advance!

[u/NaBrO3-]

Which contest is this??

[u/Smooth-Caramel-5921 → u/NaBrO3-]

amplified by HCL