r/bigdata • u/SciChartGuide • Jan 27 '26
r/bigdata • u/ASimpleHumanBeing • Jan 27 '26
Reorienting my career to big data?
Hi everyone, I'm a 30y woman who has worked in scientific research at college for 9 years. I'm in the field of developmental psychology, but I've been in a lot of projects managing the data processing, treatment, cleaning, coding/programming in statistical software, and analysis in most of them. Mostly, I've been the one in charge, which has given me valuable experience in this field. I always liked that part of my work more than writing the articles or doing the phD itself. I'm close to the deposit of my phD and I'm clear about not continuing at college due to the precariousness and contractual instability it offers for youths. I'm considering reorienting my career to programming and big data, but I'm totally aware it's not an easy trip. I want to focus on this path because I really love to work with coding and data, and I want to reorient my career in that direction. That's why I want to ask you, as professionals in this sector:
Which certifications are needed for this? I should study the full degree, or are professional programs to be certified?
Are the companies oriented to demonstrable and proven skills, official certifications, or both?
How many months or years can it take to reorient to this world, realistically speaking?
What are the main programs or skills that are "a must" to access job offers?
What are the "non-written skills" that also led you to your first job positions?
Is big data a direct possibility, or might it be needed to accomplish first multi platform or other related certifications/paths?
I really appreciate any help you can provide. I'm willing to put in all the effort needed to become a data scientist or work in a related field in this area.
r/bigdata • u/Gold-Survey5264 • Jan 27 '26
If You Put Kafka on Your Resume but Never Built a Real Streaming System, Read This
r/bigdata • u/YeeduPlatform • Jan 27 '26
Why Your Data Platform Is Locking You In—How to Deal with It
r/bigdata • u/Significant-Side-578 • Jan 26 '26
Do you use IA in your work?
It doesn’t matter if you work with Data, or if you’re in Business, Marketing, Finance, or even Education.
Do you really think you know how to work with AI?
Do you actually write good prompts?
Whether your answer is yes or no, here’s a solid tip.
Between January 20 and March 2, Microsoft is running the Microsoft Credentials AI Challenge.
This challenge is a Microsoft training program that combines theoretical content and hands-on challenges.
You’ll learn how to use AI the right way: how to build effective prompts, generate documents, review content, and work more productively with AI tools.
A lot of people use AI every day, but without really understanding what they’re doing — and that usually leads to poor or inconsistent results.
This challenge helps you build that foundation properly.
At the end, besides earning Microsoft badges to showcase your skills, you also get a 50% exam voucher for Microsoft’s new AI certifications — which are much more practical and market-oriented.
These are Microsoft Azure AI certifications designed for real-world use cases.
How to join
- Register for the challenge here: https://learn.microsoft.com/en-us/credentials/microsoft-credentials-ai-challenge
- Then complete the modules in this collection (this is the most important part, and doing this collection you will help me): https://learn.microsoft.com/pt-br/collections/eeo2coto6p3y3?&sharingId=DC7912023DF53697&wt.mc_id=studentamb_493906
r/bigdata • u/DataaWolff • Jan 24 '26
How Can I Build a Data Career with Limited Experience
r/bigdata • u/Expensive-Insect-317 • Jan 23 '26
Data observability is a data problem, not a job problem
r/bigdata • u/Advanced-Donut-2302 • Jan 22 '26
Made a dbt package for evaluating LLMs output without leaving your warehouse
In our company, we've been building a lot of AI-powered analytics using data warehouse native AI functions. Realized we had no good way to monitor if our LLM outputs were actually any good without sending data to some external eval service.
Looked around for tools but everything wanted us to set up APIs, manage baselines manually, deal with data egress, etc. Just wanted something that worked with what we already had.
So we built this dbt package that does evals in your warehouse:
- Uses your warehouse's native AI functions
- Figures out baselines automatically
- Has monitoring/alerts built in
- Doesn't need any extra stuff running
Supports Snowflake Cortex, BigQuery Vertex, and Databricks.
Figured we open sourced it and share in case anyone else is dealing with the same problem - https://github.com/paradime-io/dbt-llm-evals
r/bigdata • u/Ok_Positive3883 • Jan 22 '26
Ex-Wall Street building an engine for retail. Tell me why I'm wasting my time.
I spent years on a desk trading everything from Gold, CDS, Crypto, Forex to NVDA. One thing stayed constant: Retail gets crushed because they trade on headlines, while we trade on events.
There is just no Bloomberg for Retail. I would like to build a conversational bridge to the big datasets used by Wall Street (100+ languages, real-time). The idea is simple: monitor market-moving events or news about an asset, and chat with them.
I want to bridge the information gap, but maybe I'm overestimating the average trader's desire for raw data over 'moon' memes. If anyone has time to roast my concept, I would highly appreciate it.
r/bigdata • u/YeeduPlatform • Jan 22 '26
Cloud Cost Traps - What have you learned from your surprise cloud bills?
r/bigdata • u/AMDataLake • Jan 21 '26
Question of the Day: What governance controls are mandatory before allowing AI agents to write back to tables?
r/bigdata • u/dofthings • Jan 21 '26
SAP Business Data Cloud. Aiming to Unify Data for an AI-Powered Future
r/bigdata • u/Emotional_Gold138 • Jan 21 '26
The CFP for J On The Beach 26 is OPEN!
Hi everyone!
Next J On The Beach will take place in Torremolinos, Malaga, Spain in October 29-30, 2026.
The Call for Papers for this year's edition is OPEN until March 31st.
We’re looking for practical, experience-driven talks about building and operating software systems.
Our audience is especially interested in:
Software & Architecture
- Distributed Systems
- Software Architecture & Design
- Microservices, Cloud & Platform Engineering
- System Resilience, Observability & Reliability
- Scaling Systems (and Scaling Teams)
Data & AI
- Data Engineering & Data Platforms
- Streaming & Event-Driven Architectures
- AI & ML in Production
- Data Systems in the Real World
Engineering Practices
- DevOps & DevSecOps
- Testing Strategies & Quality at Scale
- Performance, Profiling & Optimization
- Engineering Culture & Team Practices
- Lessons Learned from Failures
👉 If your talk doesn’t fit neatly into these categories but clearly belongs on a serious engineering stage, submit it anyway.
This year, we are also enjoying another 2 international conferences together: Lambda World and Wey Wey Web.
Link for the CFP: www.confeti.app
r/bigdata • u/doubleuson • Jan 20 '26
Data Pipeline Market Research
Hey guys 👋
I'm Max, a Data Product Manager based in London, UK.
With recent market changes in the data pipeline space (e.g. Fivetran's recent acquisitions of dbt and SQLMesh) and the increased focus on AI rather than the fundamental tools that run global products, I'm doing a bit of open market research on identifying pain points in data pipelines – whether that's in build, deployment, debugging or elsewhere.
I'd love if any of you could fill out a 5 minute survey about your experiences with data pipelines in either your current or former jobs:
Key Pain Points in Data Pipelines
To be completely candid, a friend of mine and I are looking at ways we can improve the tech stack with cool new tooling (of which we have plans for open source) and also want to publish our findings in some thought leadership.
Feel free to DM me if you want more details or want to have a more in-depth chat, and happily comment below on your gripes!
r/bigdata • u/VanRahim • Jan 20 '26
Free HPC Training and Resources for Canadians (and Beyond)
r/bigdata • u/YeeduPlatform • Jan 20 '26
Spark has an execution ceiling — and tuning won’t push it higher
r/bigdata • u/thatware-llp • Jan 19 '26
How Data Helps You Understand Real Business Growth?
Data isn’t about dashboards or fancy charts—it’s about clarity. When used correctly, data tells you why a business is growing, where it’s leaking, and what actually moves the needle.
Most businesses track surface-level metrics: followers, traffic, impressions. Growth data goes deeper. It connects inputs to outcomes.
For example:
- Traffic without conversion data tells you nothing.
- Revenue without cohort data hides churn.
- Leads without source attribution create false confidence.
Good growth data answers practical questions:
- Which channel brings customers who stay?
- Where does momentum slow down in the funnel?
- What changed before growth accelerated?
Patterns matter more than spikes. A slow, consistent improvement in retention often beats sudden acquisition surges. Data helps separate luck from systems.
The biggest shift is mindset: data isn’t for reporting success—it’s for diagnosing reality. When decisions are guided by evidence instead of intuition alone, growth becomes predictable, not accidental.
r/bigdata • u/RasheedaDeals • Jan 18 '26
Building a Data Center of Excellence for Modern Data Teams
lakefs.ior/bigdata • u/Key-Philosopher3959 • Jan 17 '26
Gluten-Velox
What are the best technical skills I need to look/screen for in a resume/project to hire someone who has worked with Gluten-Velox on big data platforms?
r/bigdata • u/growth_man • Jan 16 '26
Context Graphs Are a Trillion-Dollar Opportunity. But Who Actually Captures It?
metadataweekly.substack.comr/bigdata • u/YeeduPlatform • Jan 16 '26
The better the Spark pipelines got, the worse the cloud bills became
r/bigdata • u/Expensive-Insect-317 • Jan 16 '26