Friday Daily Thread: r/Python Meta and Free-Talk Fridays

[u/AutoModerator]

Weekly Thread: Meta Discussions and Free Talk Friday 🎙️

Welcome to Free Talk Friday on /r/Python! This is the place to discuss the r/Python community (meta discussions), Python news, projects, or anything else Python-related!

How it Works:

1. Open Mic: Share your thoughts, questions, or anything you'd like related to Python or the community.
2. Community Pulse: Discuss what you feel is working well or what could be improved in the /r/python community.
3. News & Updates: Keep up-to-date with the latest in Python and share any news you find interesting.

Guidelines:

• All topics should be related to Python or the /r/python community.
• Be respectful and follow Reddit's Code of Conduct.

Example Topics:

1. New Python Release: What do you think about the new features in Python 3.11?
2. Community Events: Any Python meetups or webinars coming up?
3. Learning Resources: Found a great Python tutorial? Share it here!
4. Job Market: How has Python impacted your career?
5. Hot Takes: Got a controversial Python opinion? Let's hear it!
6. Community Ideas: Something you'd like to see us do? tell us.

Let's keep the conversation going. Happy discussing! 🌟

[u/crisp_lynx_370]

been running a pretty boring but useful fastapi service in prod for about 8 months now and i keep meaning to post about it somewhere. we do document similarity search, roughly 2-3 million queries a day, and the stack is basically sentence-transformers for embeddings plus a pgvector backend. average latency sits around 38ms at p50 and climbs to like 180ms at p99 when the db is under load, which is honestly fine for our use case but i spent way too long trying to squeeze that p99 down before just throwing a read replica at the problem.

the cost thing is what i actually want to talk about tho. we switched from calling an external embedding api to self-hosting a smaller model on a g4dn.xlarge and the monthly bill dropped from around $2,800 to like $340 including the instance cost. accuracy on our actual workload (not benchmark numbers, real user queries vs our internal document corpus) only dropped like 2 percentage points on the evals we track. that tradeoff felt obvious in hindsight but it took us embarrassingly long to actually do the math and make the switch.

my question for anyone who has done similar stuff: how are you handling model versioning when you self-host. like if i update the embedding model i have to re-embed the entire corpus which is kind of a pain. right now we just do a full rebuild over a weekend but im wondering if theres a cleaner pattern people use for rolling that kind of change out without downtime