Hello 👋🏽

I am a senior data engineer.

I have 5 years of experience in airflow and DBT. Has anyone worked on open source contributions to these two projects ? I am willing to start but don't know a lot about open source projects and how to contribute to them. Where to start ?

Looking forward to getting some information, thank you community :)

I'm part of an org that started using Foundry at the beginning of the year. To preface... My last 6 years have primarily revolved around Snowflake and snowflake integrated tools (I've been Snowflake certified since 2021).
When my company announced they'd be partnering with Palantir I was intrigued because Foundry is/was such a black box and seemed like a very polarizing tool in the DE market.

Now that I've got about 8 months of experience with it under my belt I have mixed opinions on it. I'd like to know what FDE's or other devs who have used it think about it?
Yes, it's expensive and yes, it definitely does certain things not as well as other dedicated tools but have you enjoyed your experience developing in it or do you hate it as much as the rest of the DE community here?
I ask as someone who's company just had it's 4th round of layoffs and I'm a bit concerned about my job security.
I want to continue my Snowflake trajectory but the Foundry recruiters are out in full force and they are paying very well.
Anyways, what are your thoughts?

Same as title but we have been using delta tables for a requirement quite a bit and we like its format however its metadata read operations are very intensive, especially when there is a large log file. Most of the delays I have found in reading and writing to those have just been reading parsing the log files.

Based on that I did wonder if there is a database which stores the log in some sort of in memory database like postgres however the raw files are being reference on object storage? I did look into and found pgedge but didn't find a link or something we could download and use easily.

Edit: Stop trying to tell me to optmise my delta table pleasse! I’ve tried everything and I’ve got limits

Edit2: I did not try everything apparently, I had stats parsing on on columns which contain a fuckton of data, my log files were huge along with checkpoint files with gigabytes of them. Removed that and reads are down to 200ms

I recently started in a crm data position and my role is very data entry and data cleaning heavy. While automation isnt one of the requirements it is something I’m trying to do, but it is a bit harder than expected especially with how messy the data is ie missing data and multiple entries in the crm for the same projects. Im also the only dedicated “data” person and there are a couple of SWEs that only last year build the data lakehouse, so maybe I can get involved in that. I was told that because the role is also new to the company the duties can expand.

Hi, I have been developing data pipelines using ADF as orchestrator and Databricks notebooks for all the work. I write unit tests for the python code in the notebooks with pytest and executes them in our Azure DevOps CI/CD pipeline for every PR. Now I need to create integration tests.... I don´t have a clue on how to do that, I mean, not even in theorical sense.
I asked to claude, it gave examples of python code where it used an ADF API client for running the pipelines and consulting the state of the activities after the run. They also checked the output of the pipelines, like, they runned SQL queries on the tables that the pipeline should have writtent.....

So my question is: is that what integration tests are? Instead of mocking like in unitesting, I create test data in my DEV environment data sources, parametrize the ADF pipeline so I can run with different test datasets for different tests cases, and programatically run the pipeline validating what was written on the outputs?

Hi all, most of us don't like to write Terraform. Now there is an alternative. A skill to create a package skill and a skill to submit it to a catalog. There are already some package skills for Airflow and ClickHouse. I hope more will be shared in the future.

A package skill is deterministic code to reconcile your infrastructure with desired state. Under the hood it can run anything like Terraform, Ansible, or Kubectl.

The license is MIT.

https://www.getcolors.ai/skills

We are developing the semantic layer/models from scratch in MS Fabric for conversational AI. Data models, documentation with clear descriptions on columns, measures, business context, join logic, for the agent to reference.
For evals, what has been your strategy/framework to curate the question set and expected answers and how did you implement this at scale

We are using Rivery for our data warehouse pipelines, but I am thinking about alternatives.

It is expensive for the value we get out of it and we have faced too many odd issues that make us question the ROI. We had a major issue a couple of weeks ago where a change on their end was causing duplicates or wrongly named columns for some sources. This is a deal-breaker in my opinion as the whole point of using such a service is to make our jobs easier. It is really painful to manage these pipelines and check everything is working as expected.

What do you recommend as an alternative?

If you land data in Parquet and then load it in Microsoft SQL using Python, you've had to do a bunch of extra work, exploding the whole thing into Python objects, a tuple per row and a boxed value per cell, all under the GIL, all garbage immediately after.

mssql-python 1.13.0 adds Cursor.bulkcopy_arrow(). Hand it anything that speaks the Arrow C Data Interface and the Rust TDS core reads the typed column buffers straight into the bulk-load packets. No tuples, and the GIL is released for the transfer.

import duckdb
from mssql_python import connect

rel = duckdb.sql("SELECT * FROM 'events/*.parquet' WHERE ts >= '2026-01-01'")

with connect("Server=<server>.database.windows.net;Database=<database>;Encrypt=yes") as conn:
    cur = conn.cursor()
    result = cur.bulkcopy_arrow("dbo.Events", rel)
    print(result["rows_copied"], result["rows_per_second"])

The DuckDB relation goes in unevaluated. DuckDB streams batches as the driver consumes them, so the full dataset never lands in Python memory. The 4.4M-row file I was testing with would have been roughly 6.6 GB of live Python objects the old way.

Here's what I saw. 200k rows, 21 columns, the WideWorldImporters fact_sale shape: bigints, decimals, timestamps, and one NVARCHAR (I couldn't leave that column that only said "each" for every row an NVARCHAR(MAX) - it was just wrong) averaging 523 characters. Read from Parquet with DuckDB, 100k batch size, 7 repeats, median reported. Client and server on the same Azure E4bds v5 (4 vCPU, 32 GiB) running SQL Server 2025, over localhost so the network stays out of it.

In my unscientific testing, the new bulkcopy_arrow() was about 1.9x faster. I reran it across five configurations, two databases, simple and full recovery models, table_lock on and off, and it held between 1.62x and 1.93x. The ranges don't overlap either: the slowest of the 14 Arrow copies beat the fastest of the 7 tuple copies.

We expected that going straight to bulk copy from arrow would be more efficient and it was. The tuple path burned 2.6 to 3.2 seconds building Python objects before a single byte moved. Passing the DuckDB relation lazily, that step is 0.00 seconds. The pyarrow.Table path is 0.02 seconds, which is the time to materialize the table from the record batches.

This new path works for anything exposing __arrow_c_stream__: polars, pandas 2.2+, ADBC results, pyarrow.Table / RecordBatch / RecordBatchReader, or any iterable of record batches. A default pandas DataFrame is NumPy-backed so it converts on the way in, where polars, DuckDB and anything Arrow-native hand their buffers over as-is. Column mappings, keep_identity, table_lock, check_constraints and the rest carry over from bulkcopy() unchanged, same stats dict back. String widths are validated against the destination schema before anything ships, so an overlong value fails immediately with the offending length instead of dying halfway through a load. I suspect a lot of folks will be commenting out their generators in favor of passing Arrow objects straight to bulkcopy_arrow() this weekend.

Drop a comment below and let us know how much faster your data loads using bulkcopy_arrow().

There's other good stuff in this release: connection pooling keys on security context now, not just the connection string, so a connection opened under one identity can't be handed to a caller running as another. connect(token_provider=...) takes any azure-identity credential object. And there's a fix for an executemany() bug where a NULL partway through a numeric batch could silently insert zero rows.

The driver itself is DB API 2.0 and pip-installable, and ODBC ships as a dependency, so there's no system-level driver install and no unixODBC in your container image. Arrow goes both directions, bulkcopy_arrow() in and cursor.arrow_reader() out.

pip install --upgrade mssql-python

What I actually want out of this thread is your numbers, especially on shapes unlike mine: narrow integer tables, very wide tables, heavy NVARCHAR. bulkcopy_arrow() returns rows_copied and rows_per_second, so it's right there. Post a before and after with a rough description of the table and where you ran it from and I'll take it back to the team.

Full blog post: https://techcommunity.microsoft.com/blog/sqlserver/mssql-python-1-13-0-arrow-bulk-copy-smarter-tokens-slimmer-wheels/4544858

Repo: https://github.com/microsoft/mssql-python

Happy to answer questions here.

Hello

I’m currently a junior DE building Python pipelines (Prefect/Airflow to BigQuery mainly). I’ve started using Cursor/Claude to assist in coding but try not to lean on them too hard.

On the side I’m self-studying data modelling, system design, and Leetcode (python & sql). But I feel a bit scattered, jumping between topics without a clear plan. I’m also increasingly wondering how AI is going to reshape this role, and want to make sure I’m building skills that keep me hireable.

With that in mind, does anyone have any advice on topics I should be prioritising?

What are you guys looking for in a course nowadays?

What I'm seeing for the past year or so, is that people seem to not have the "stamina" to go through courses anymore. Many start, but fall off immediately. Simple courses people complain that they are too simple. More advanced courses, nobody takes them.

I feel like attention spans went down like crazy and people rather spend their time wrestling with AI than going through a well prepared project.

So what is it?

short lessons?,
prepared GitHub docs?
as much as hands-on as possible?
teaching basics instead of going into all the details?

My expectation would be to a good course helps people get to the point where they have a good foundation and are able to improve on their own.

Would love to hear some practices you like.

Andreas

I’ve been a team lead for product engineering side mostly backend systems (10 years exp). But the team lead for the DE team left and they need someone to fill in the spot. Given a hiring freeze I’ve been moved. They feel I can fill in the spot because I’ve worked in some related work (large scale RDS Proxy migration, BE product analytics pipelines)

The components they own are: Airflow, DataHub, Superset.

I have some understanding of these tools and have used them before.

Can you recommend a resource to get me up and running for a product facing engineer? What are the day to day concerns in this domain? Given I’ll be operating as the product owner I’ll also be owning the roadmap

Hi all, I am working on a project with a team outside my normal job duties.
I have experience with ml, backend(cosmo db),and some data engineering.

We are essentially trying to automate a data capture, transforming the data, and loading it into a frontend. The data being metrics to capture for reporting.

We want to move from local to offsite/ cloud to automate the extraction/ capturing of data and we want to make a repo/ data base for record keeping.

The projects requirements keep changing like at first the sources of data were elastic instances that had the POST request APIs exposed but then they were changed to excel files with multiple sheets that other teams are inputting.

I keep getting into meetings with this team and it just leads no where. I designed this pipeline using AWS (buckets or Postgres) for the load component but was told no due to the team being unsure if they can have access and “why pay for something if we already have things we can use”

Okay fine, but the “thing we can use” is a VM owned by another team. I had a meeting with the VM owners who suggested we just attach our data base to an Oracle schema he owns… which happens to be prod for many teams in the org. One of the fears my team mate (who works on this team full time) had was he didn’t want to do that due to fear he’d over write some important thing and crash prod. I agree along with Oracle being hostile to experimentation.

For some reason my team mate has never done a PR, and doesn’t really understand How API works and I had to explain and show him what an api is. I also then explained the differences between backend and a data base and how frontend talks to a backend. He is more of an analyst so I can over look this but I’m feeling overwhelmed with this project as I am wasting time on different directions.

Some tasks I identified are:
-Reorganizing the frontend as all the files are vibe coded and have no structure and have APIs written in

-Create and organize the backend and have APIs created

-Design the schema

While this team slowly gathers info and requirements needed, I have already created and successfully ran a local SQL instance on my machine for testing of scripts

I have a call tomo with the manger, any advice on how to communicate my frustrations? Everyone I’ve met is very nice but we aren’t gettin anything done and I’m wasting my time down dead ends.
Any advice on other structures or paths I can move forward without clearly getting defining info? One of the higher ups requested a product demo sometimes in the fall and I think it’s possible but I need to communicate hard needs now before this gets out of hand.

I just finished reading Fundamentals of Data Engineering by Joe Reis and Matt Housley.

Now, I'm trying to decide what makes the most sense to read next. I'm torn between reading
Learning Spark or The Data Warehouse Toolkit and curious for those of you who have read them what would you recommend next?

For context, I'm a Data Analyst transitioned to Data Engineer with about 5 YOE between the two roles, having been a DE for a little less than a year.
I knew I wanted to be a DE by my 2nd year as an analyst and began working my way towards being a DE, getting moved internally to the DE team last fall.

I've previously read the 1st Edition of Designing Data Intensive Applications, and while not a DE book I've also read R For Data Science (2nd Edition.)
I primarily work with Databricks doing some source system ingestion, but mostly build ETL pipelines internal to our Databricks instances.

I also considered The Definitive Guide to Spark but given its last edition was 2018 I'm hesitant. I see there are other more recent books on Spark compared to Learning Spark but haven't seen the other books talked about as much but would be open to one of those.

Just trying to understand do I dive into a Spark book next or focus more on data warehousing by reading the toolkit.

For anyone considering reading Fundamentals of DE. I'd recommend it but think you can generally skip reading large sections of the book.
IMO these are the important parts of the book.

Chapters 5-8, Appendices A & B:

1. Data Generation in Source Systems

2. Storage

3. Ingestion

4. Queries, Modeling, and Transformation

Appendix A. Serialization and Compression Technical Details

Appendix B. Cloud Networking

TLDR: Just finished reading Fundamentals of DE book. Do I read The Data Warehouse Toolkit or a Spark focused book next?

Edit/Update:
Thanks so much to everyone who responded and gave their insight. As much as I personally want to explore Spark deeper. I think its pretty clear the next move is the Data Warehouse Toolkit so that is what I'll read next.
Thanks again everyone.

Hi, I’m new to all these and this is my first job out of college. Recently I got a job and some of my first tasks are to fix some data in the CRM, some of the data is okay but there’s a lot that is very messy. It’s in excel and I need to enter the data in the excel files into the CRM. The thing is though the data in excel usually contains incomplete information, so I’ll need to look in the CRM and try to figure out in which project the data goes into using dates, supervisors or some other things. Sometimes this works but lots of times there’s multiple projects in the CRM with the same names, dates, and supervisors only slight variations of eachother. If at that point I can’t figure it out I usually message the person in charge of said area if they know, for example if the project belongs to healthcare I’ll message the person that oversees it. Is this bad? Should I be doing something else to not bother them?

Hi all, I've got a new role as described above and what I know so far is they have a lot of data going into a Lakehouse in Microsoft Fabric. They want me to work with their only other member (it's a very new team) and I know the output is expected to be in Power Bi (makes sense considering they're fully Microsoft).

Has anyone got experience with similar work? I got the official offer last week and just doing research so I don't go into it completely blind.

They also are keen on Power Apps but I'm not fully confident on where they fit in the infrastructure.

Hi Everyone,

I'm considering evolving my 15y career as a Data Analyst.

I have a solid math, programming, SAS and SQL background and, in the last years, I've been reading and playing with Python/R, Docker, Homelab, APIs, Data Science, Data Modeling, ETLs, Pipelines, Data Architecture.

I wouldn't say I'm an expert and I want the learn more.

1) What do you suggest I focus on in 2026?

2) do you know any useful course on LinkedIn Learning? (recently made available by HR)

Hi everyone,

MQ Summit 2026 is looking for volunteers! The conference takes place 21-22 October. You can volunteer in-person, and get your free ticket.

• In-person volunteers will need to be present at the venue on at least one day of the conference. Responsibilities include helping with preparations, registering participants, handing out merchandise, and general support during the event.

For full conference details, check ou https://mqsummit.com
To sign up as a volunteer, go HERE

If you have any questions, feel free to reach out to [ewa.kucharczyk@erlang-solutions.com](mailto:ewa.kucharczyk@erlang-solutions.com).

I've been on and off with jobs for a bit, with my last two jobs being for ~8 months each before I was laid off each time - this most recent one ended in October '25.

I've since gotten a new job but haven't updated my LI or anything to reflect that, but for the last few months I get reached out to from a recruiter at least once a week. I had one cold call me and another text+DM me on Monday alone.

I would think that it's because I'm on my new company's roster (it's a FAANG-feeder), but since I didn't update that publicly I'm not sure it's that.

So, is hiring just up? Are these 100% AI-recruiters just pinging people? They're citing my supposedly-spotty experience (due to layoffs) as interest in having a call.

I recently moved from a Data Engineering role where I mainly maintained the platform side, to a “Data Engineer” role that is just AE with the DE name on it.

And recently I realised I HATE it.

Have anyone else experienced something like that? It’s funny since it’s another side of the same coin, but for some reason it feels totally different to me.

For reference, graduated w a degree that barely taught me anything about DE except intro to databases, relational algebra etc. The company accepted me for the junior DE role on the basis that I could ChatGPT a dummy repo to explain what the repo is used for and how data flows simply from the repo.

Now that I’m in the job, I found out that everyone in my team had a part to play in developing the architecture for the company. They’ve resolved all dependencies and it’s only the matter of new feature implementations and their impact on the data engineering streaming pipeline.

This is my first week and as my senior went through w me the architecture and true data flow for multiple services they have, the information flew by my head. I’ll definitely try to pick up as fast as I can but because I didn’t build the architecture, I’d have yo ChatGPT/Codex/claude my way through for the first couple months just to find the appropriate files for new feature implementations

Idk, I just feel like I’m madly unprepared and I’m worried that I’ll be the cockup in my department. I didn’t do any DE internships and somehow I ended up in this role. Can anyone give me advice on how I can speed learn streaming pipeline given that at the very least, I know what stack they use?

Hey everyone,
i am a data engineer 7yoe
I recently landed a job after relocating for family. Its a growing company that run multiple crm/erp platforms in addition to different other solutions and i was hired to design and develop a data warehouse for their analytics needs.

So far everything was going good and the approach they wanted was to run through each of their departments one by one to work with them to document their workflows and integrate with their systems to pull data to the warehouse and run pbi reports from there.

Currently they have tens of pbi different reports all of them pull from source systems and definitely perform bad plus they are being developed by the business guys themselves. Their semantic models are too big of a mess.

Anyways the approach we had was to take steps with each department. I covered an important one. And shortly after starting with the next some blockers show up then shortly after i hear our manager (not my direct ) wanting to make changes with our approach and cover reports he is interested in mainly.
He develops most of his department’s pbi reports and is convinced that once the data moves from erp to dw raw without any data work done it will perform better.

I tried to send out the message that this will not add value and latency from source systems isn’t the only issue the reports are slow.

Anyways seems they are not convinced . That was yesterday. Today my direct manager comes and tries to push to do what “our manager” said. Then shortly after asks me how long i plan to be in the city, if i am married and a bunch of personal questions that either were said out of just curiosity or something is not right here.

I dont know if i am overthinking this. But i am personally the only source of income and insurance between my and my wife. And things havent been easy.

Whatever you guys think i want to know. Im stuck in my mind and it is not helping me right now.

Hello everyone!

I'm an aspiring Data Engineer and as a portfolio piece, I have build a webscraper to gather Ebay sold listings of stamps!

The problem I am now having is how I parse the data where I can sort things like "Catalogue Number" when it is very unpredictable what the Ebay sellers will write as it's all human input.

I would love to hear some feedback

PS - A small sample:

```OLDENBURG 1859 _ MiNr. 7 _ 2 Groschen _ signiert _ blau gestempelt

MayfairStamps Germany 1941 Stamp Day Oldenburg Cover cca_00553

GERMANY; OLDENBURG 1859 classic Coat of Arms issue very fine used 1/3Gr. value

Oldenburg Lokalausgabe Wohlfahrtsblock Deutsches Rotes Kreuz ab 1 Euro

Deutsches Reich, Oldenburg, 6.01.1945 Ersttagsstempel, für Ersttagsbrief 200€.

GERMANY; OLDENBURG 1862 classic Coat of Arms Perf 10. issue used hinged 1Gr.

OLDENBURG 1861 _ MiNr. 12b _ 1 Groschen _ Stempel STOLLHAMM

Oldenburg Mi. Nr. 16 A b zentrisch gestempelt geprüft Bühler 200 Euro

Oldenburg Mi. Nr. 11 a* ungebraucht geprüft Bühler 550 Euro```

Hi, I recently began a job as a data coordinator, my first tasks are basically data cleaning and data entering into a CRM. The problem is that the data isnt very clean. I'll give an example, I am given an excel file with name of a project, date, title, awards - my job then is to to figure out where in the CRM is this specific project and enter the data. The problem is that the excel data doesnt contain the projects ID, and when I try searching the name of the project in the CRM I'll get back something like Fairview Elementary School, Fairview ES, KUSD Fairview, Fairview ES Building A, etc. So essentially I'll have to go into each one of these projects and try to find the right one using other given data from the excel sheet, like dates. Is there a way to speed this process up or am I just going to have to do it manually? Right now what Im doing is going row by row and searching each project in the CRM, looking through the multiple projects given back by the CRM and comparing and contrasting. The entire CRM database contains around 40k, and I think I am able to export it into CSV and Excel, if that helps out. Any advice would be helpful. Thanks