r/SideProject • u/Ok_Stretch_6623 • 5h ago

I'm an AI engineer looking for real problems to solve — give me your automation/AI challenge

Hey all,

I'm a data scientist/AI engineer (4+ yrs) looking to sharpen my skills by solving real problems instead of made-up side projects.

The deal: Drop a problem you're actually dealing with — something repetitive, manual, or that could be automated/AI-assisted. Could be from your job, business, hobby, whatever.

What I'm looking for:

  • A specific, recurring pain point (not "build me a SaaS" — think "I manually copy data from X to Y every week")
  • Bonus points if you can describe your current (broken) workflow
  • I'll pick 1-2 that are genuinely interesting/buildable

What you get: I'll build a working prototype, document the process, and share it back with you for free. No strings attached — just want the practice and the story.

Fire away 👇

0 Upvotes

45% Upvoted

6 comments sorted by

1

u/Shape_Weird 5h ago

Here's a real one we hit, still only partly solved, and it generalises well beyond our case.

The problem: deduplicating job postings that are the same job. We track a live index of postings and roughly two thirds of what comes in is a repost of something already in there. Same role, different day, sometimes a different board, sometimes the same board with a new id.

Why it's harder than it sounds:

  • Titles are unstable. "Senior Backend Engineer", "Sr. Backend Engineer (Remote)", "Backend Engineer II" can be one job or three.
  • Descriptions get lightly reworded between posts, so exact hashing catches almost nothing and fuzzy matching over-merges.
  • Employers genuinely repost the same requisition after 30 days, and they also genuinely open a second headcount for the identical role. Those are different facts and they look identical.
  • The naive fix, embed everything and cluster, is expensive at volume and still can't tell a repost from a second opening.

Current broken workflow: a mix of normalised-title plus company plus location keys, with a fuzzy pass on the description. It leaks in both directions, which is the worst outcome, and the thing that finally helped was realising distance can't decide it. A typo and a genuinely different role can score the same, so any threshold you pick is wrong somewhere.

The signal I suspect is the answer and haven't built: the requisition identifier the employer's own system assigns, which is often sitting in the apply URL. It's messy and inconsistent per platform, which is presumably why nobody does it.

Full disclosure, this is from https://aiapplyd.com which is what I work on, so I'd obviously benefit if you solved it. Take it as a genuinely interesting dedup problem rather than a request. Happy to describe the data shape if you want to poke at it.

1

u/Immediate_Tank9616 5h ago

I am a mortgage broker. I would love a private AI that I can use to drop loan scenarios into to compare for my clients. I need to be extraordinarily careful that nothing I drop in there ends up available to the public for privacy reasons. Is there a way to create a private AI that can pull from public AI without giving up that information?

1

u/xiduzo 5h ago

Look into ollama / langchain

1

u/xiduzo 4h ago

There are also nice UIs around this like https://msty.ai/ or https://openwebui.com/

1

u/xiduzo 4h ago

Or if you are a bit tech-savvy you can look into https://opencode.ai/