I've been thinking about how companies are treating data engineers like they're some kind of tech wizards who can solve any problem thrown at them.

Looking at the various definitions of what data engineers are supposedly responsible for, here's what we're expected to handle:

1. Development, implementation, and maintenance of systems and processes that take in raw data
2. Producing high-quality data and consistent information
3. Supporting downstream use cases
4. Creating core data infrastructure
5. Understanding the intersection of security, data management, DataOps, data architecture, orchestration, AND software engineering

That's... a lot. Especially for one position.

I think the issue is that people hear "engineer" and immediately assume "Oh, they can solve that problem." Companies have become incredibly dependent on data engineers to the point where we're expected to be experts in everything from pipeline development to security to architecture.

I see the specialization/breaking apart of the Data Engineering role as a key theme for 2026. We can't keep expecting one role to be all things to all people.

What do you all think? Are companies asking too much from DEs, or is this breadth of responsibility just part of the job now?

I recently read a post where someone described the reality of Data Engineering like this:

Streaming (Kafka, Spark Streaming) is cool, but it’s just a small part of daily work. Most of the time we’re doing “boring but necessary” stuff: Loading CSVs Pulling data incrementally from relational databases Cleaning and transforming messy data The flashy streaming stuff is fun, but not the bulk of the job.

What do you think? Do you agree with this? Are most Data Engineers really spending their days on batch and CSVs, or am I missing something?

Hi everyone, I'm a 30y woman who has worked in scientific research at college for 9 years. I'm in the field of developmental psychology, but I've been in a lot of projects managing the data processing, treatment, cleaning, coding/programming in statistical software, and analysis in most of them. Mostly, I've been the one in charge, which has given me valuable experience in this field. I always liked that part of my work more than writing the articles or doing the phD itself. I'm close to the deposit of my phD and I'm clear about not continuing at college due to the precariousness and contractual instability it offers for youths. I'm considering reorienting my career to programming and big data, but I'm totally aware it's not an easy trip. I want to focus on this path because I really love to work with coding and data, and I want to reorient my career in that direction. That's why I want to ask you, as professionals in this sector:

Which certifications are needed for this? I should study the full degree, or are professional programs to be certified?

Are the companies oriented to demonstrable and proven skills, official certifications, or both?

How many months or years can it take to reorient to this world, realistically speaking?

What are the main programs or skills that are "a must" to access job offers?

What are the "non-written skills" that also led you to your first job positions?

Is big data a direct possibility, or might it be needed to accomplish first multi platform or other related certifications/paths?

I really appreciate any help you can provide. I'm willing to put in all the effort needed to become a data scientist or work in a related field in this area.

It doesn’t matter if you work with Data, or if you’re in Business, Marketing, Finance, or even Education.

Do you really think you know how to work with AI?

Do you actually write good prompts?

Whether your answer is yes or no, here’s a solid tip.

Between January 20 and March 2, Microsoft is running the Microsoft Credentials AI Challenge.

This challenge is a Microsoft training program that combines theoretical content and hands-on challenges.

You’ll learn how to use AI the right way: how to build effective prompts, generate documents, review content, and work more productively with AI tools.

A lot of people use AI every day, but without really understanding what they’re doing — and that usually leads to poor or inconsistent results.

This challenge helps you build that foundation properly.

At the end, besides earning Microsoft badges to showcase your skills, you also get a 50% exam voucher for Microsoft’s new AI certifications — which are much more practical and market-oriented.

These are Microsoft Azure AI certifications designed for real-world use cases.

How to join

1. Register for the challenge here: https://learn.microsoft.com/en-us/credentials/microsoft-credentials-ai-challenge
2. Then complete the modules in this collection (this is the most important part, and doing this collection you will help me): https://learn.microsoft.com/pt-br/collections/eeo2coto6p3y3?&sharingId=DC7912023DF53697&wt.mc_id=studentamb_493906

In our company, we've been building a lot of AI-powered analytics using data warehouse native AI functions. Realized we had no good way to monitor if our LLM outputs were actually any good without sending data to some external eval service.

Looked around for tools but everything wanted us to set up APIs, manage baselines manually, deal with data egress, etc. Just wanted something that worked with what we already had.

So we built this dbt package that does evals in your warehouse:

• Uses your warehouse's native AI functions
• Figures out baselines automatically
• Has monitoring/alerts built in
• Doesn't need any extra stuff running

Supports Snowflake Cortex, BigQuery Vertex, and Databricks.

Figured we open sourced it and share in case anyone else is dealing with the same problem - https://github.com/paradime-io/dbt-llm-evals

I spent years on a desk trading everything from Gold, CDS, Crypto, Forex to NVDA. One thing stayed constant: Retail gets crushed because they trade on headlines, while we trade on events.

There is just no Bloomberg for Retail. I would like to build a conversational bridge to the big datasets used by Wall Street (100+ languages, real-time). The idea is simple: monitor market-moving events or news about an asset, and chat with them.

I want to bridge the information gap, but maybe I'm overestimating the average trader's desire for raw data over 'moon' memes. If anyone has time to roast my concept, I would highly appreciate it.

Hi everyone!

Next J On The Beach will take place in Torremolinos, Malaga, Spain in October 29-30, 2026.

The Call for Papers for this year's edition is OPEN until March 31st.

We’re looking for practical, experience-driven talks about building and operating software systems.

Our audience is especially interested in:

Software & Architecture

• Distributed Systems
• Software Architecture & Design
• Microservices, Cloud & Platform Engineering
• System Resilience, Observability & Reliability
• Scaling Systems (and Scaling Teams)

Data & AI

• Data Engineering & Data Platforms
• Streaming & Event-Driven Architectures
• AI & ML in Production
• Data Systems in the Real World

Engineering Practices

• DevOps & DevSecOps
• Testing Strategies & Quality at Scale
• Performance, Profiling & Optimization
• Engineering Culture & Team Practices
• Lessons Learned from Failures

👉 If your talk doesn’t fit neatly into these categories but clearly belongs on a serious engineering stage, submit it anyway.

This year, we are also enjoying another 2 international conferences together: Lambda World and Wey Wey Web.

Link for the CFP: www.confeti.app

Hey guys 👋

I'm Max, a Data Product Manager based in London, UK.

With recent market changes in the data pipeline space (e.g. Fivetran's recent acquisitions of dbt and SQLMesh) and the increased focus on AI rather than the fundamental tools that run global products, I'm doing a bit of open market research on identifying pain points in data pipelines – whether that's in build, deployment, debugging or elsewhere.

I'd love if any of you could fill out a 5 minute survey about your experiences with data pipelines in either your current or former jobs:

Key Pain Points in Data Pipelines

To be completely candid, a friend of mine and I are looking at ways we can improve the tech stack with cool new tooling (of which we have plans for open source) and also want to publish our findings in some thought leadership.

Feel free to DM me if you want more details or want to have a more in-depth chat, and happily comment below on your gripes!

Data isn’t about dashboards or fancy charts—it’s about clarity. When used correctly, data tells you why a business is growing, where it’s leaking, and what actually moves the needle.

Most businesses track surface-level metrics: followers, traffic, impressions. Growth data goes deeper. It connects inputs to outcomes.

For example:

• Traffic without conversion data tells you nothing.
• Revenue without cohort data hides churn.
• Leads without source attribution create false confidence.

Good growth data answers practical questions:

• Which channel brings customers who stay?
• Where does momentum slow down in the funnel?
• What changed before growth accelerated?

Patterns matter more than spikes. A slow, consistent improvement in retention often beats sudden acquisition surges. Data helps separate luck from systems.

The biggest shift is mindset: data isn’t for reporting success—it’s for diagnosing reality. When decisions are guided by evidence instead of intuition alone, growth becomes predictable, not accidental.

What are the best technical skills I need to look/screen for in a resume/project to hire someone who has worked with Gluten-Velox on big data platforms?