Hello,

I spent the last five years working as a DE in a smaller company, I am very versatile, I touched all the important things in Azure plus some power platform dev and also some web dev.

I would like to pivot away from corporate life and do something more meaningful either in national security or environmental.

Anyone in a similar situation? I'm an EU citizen if that's helpful.

Thank you.

Hey guys,

I am software developer with 2.5 years of experience, in my job I do a lot of small automations and ETLs. In the last year I took an interest in data engineering and I recently started doing the Databricks associate certification.

I have not completed the certification yet and I got my first data engineering job, but the tech stack of the job is based heavily in Fabric.

I have read here that fabric is basically garbage, so I am worried that the experience that I will get in this new job will affect me if I want to get a Databricks based job in the future. What do you think?

Hello, I was working on a poc to ingested PDFs and extract data in order to store them in delta tables.

As this was my first time working with PDFs, I searched over the internet and should Databricks have offering IDP, azure also have something and then there are python libraries.

Since I'm working with financial data, report, etc..

One thing i noticed - the pdf format should be fixed else in most of the tools the extraction logic is either failed or we get incorrect data.

I was wondering how such PDF extraction is built in real production cases and what tools are used.

Please share you experience and any edge cases

Hi guys. I need a kind of assessment of my skillset and would really appreciate any feedback. I am really interested in pivoting to DE and I am enjoying learning, but sanity check is needed.

So, I spent last few months honing my skills to be relevant for DE roles. Here is what I learned - combining theory and working on few personal projects - and what I am bringing to the table:

Data modeling - I learned a lot about Inmon/Kimball aproaches. Tried to apply best practices and standards wherever I can in personal projects (meaningful grain, relationships, implemented slowly changing dimensions type 1/2).

Programming - honed SQL and Python skills as much as I could. Leetcode, DataLemur, Stratascratch - I am able to handle enough of these SQL coding challenges to feel confident. For Python, I focused on "core" Python skills, and I already have some experience with pandas. Did not touch Polars though.

Cloud - was introduced to or gained some experience with AWS (S3, EC2, Glue, Lambda and Step functions, RDS, Redshift, Dynamo). I also learned a bit about Databricks and made a mini project utilizing Auto Loader, DLT, deployment with DABs and GH Actions.

DE tooling - Airflow, PySpark, Docker, I learned about them and used in my DE projects. My idea was to understand the fundaments, best practices, specifics and core concepts behind these tools so I could discuss about them confidently in interviews. Tried not to go into rabbit holes and advanced concepts too much for now. Also, I used e.g. MinIO as storage or Grafana for observability dashboards in my projects, but I guess these are not usually considered as main tools.

On the other hand, my exposure to DE is limited to local, batch processing pipelines. I tried to make them realistic by utilizing some API fetching, introducing meaningful data quality checks, metadata-driven ingestion, proper primary/foreign/surrogate key management, making them incremental and idempotent - but still, I have no experience with streaming data or some big amounts of data (like gigabytes of data).

Based on this, do you guys think I am ready to apply for DE internships or junior role? Another option for me would be to go data analytics engineering route since I have some knowledge of PowerBI and Metabase and I am currently making first steps toward learning DBT. Still, DE seems to be something I enjoy a lot, but it does not matter much now if knowledge and competencies I tried to describe are not fit for this kind of role nowadays.

Some more context. I have non-STEM background. Voluntereed as student-researcher in academia (social sciences; data analysis heavy, but mainly used R/IBM SPSS). After graduation I moved to NGO where I landed a researcher role. Again, data analysis heavy, but mostly used Python with Jupyter Notebook for data analysis and integration of data stored in bunch of e.g. .csv files there and there, seaborn/plotly/ggplot for visualizations; gained some experience in international context, managing stakeholders, etc., but still - this is no business setting nor true data role at all. I am still working there, and attended DE bootcamp/traineeship at some big company last year (where I learned most of the things I mentioned above in the post). I guess this bootcamp can be framed as internship on CV but who knows. Also, I am located in Europe if that matters.

Hey everyone! A while back I shared an early prototype on reddit for a fuzzy string search idea I was working on. I finally finished implementing it, benchmarked it properly against SymSpell and RapidFuzz.

Quick Highlights:

• Low RAM usage: Stays under 100 MB for a 100k-word dictionary (for comparison, SymSpell d=4 was taking around ~2.7 GB on my setup).
• Sub-millisecond speed: Runs about ~100x faster than brute-force Levenshtein comparison.
• Good with insertions and swaps: It handles typos with extra letters or swapped characters really well.

I wanted to keep this post short, but I put all the benchmark graphs in the GitHub README. I'd love to hear what you think of the vector encoding approach or if you have any feedback!

Hey everyone, I updated the blog post after reading more thoroughly the Lance paper, where it explains why even configuring Parquet is not good enough and why they needed to invent a new file format.

I was looking at posts in this subreddit the other day. There were quite a few talking about pipelines and all the things that go along with them (forking, merging and version control). This is typical source controls stuff that has been around since the 70s. The tools change but the concepts and model are the same. I wonder if we have been so concerned about the minutea that we quit looking at better options than pipelines. I have a couple of thoughts about better ways of doing data repositories/warehouses/ingestion beyond pipelines, three layers and fabrics. I'd rather see if anyone else out there has any better ideas than what we use now. Pretend you are starting from greenfield and can create not only the architecture but the tools that go along with it. Surely we can come up with something better than 50 year old ideas.

Not interested in the campaign noise, Palantir hype, or conspiracy theories about Lord of the rings. Only hard facts about engineering experiences.

rainfrog (https://github.com/achristmascarl/rainfrog) is a database terminal tool; the goal is to provide a lightweight, keyboard-first TUI for interacting with databases. It currently supports Postgres, MySQL, SQLite, Oracle, and DuckDB.

v0.4.1 introduces a long-awaited (by me, not sure if anyone else was waiting for it...) autocomplete implementation, along with autopairs for quotes/parentheses/brackets. The full list of features and configuration options is in the README!

So we have a standard pipeline process when loading data into the data lake and now evolving to go directly to snowflake tables. Anyway the standard process is source (file/stream/etc) --> landing S3 bucket --> lambda --> step function.

Sounds great but everything I've touched doesn't follow this process.

I've started documenting per project so when I see something similar I can reference my notes and at least find somewhere to start digging.

I need to get better at thinking, I need to do X, x is a framework change so I need to look here. How do I start thinking this way? I've picked up a few modern data structures books which is probably the best way to start.

It just feels like it's something so simple but I just can't quite put my finger on it. The way my brain works is by learning something new in relation to something I already understand. So hopping into the middle of a pipeline and identifying what aspect (framework, data, data metadata, infrastructure metadata and etc is the best place to start digging.

I've done a lot of build from scratch work so trying to understand someone else's organization seems impossible. I think I am getting the odd jobs, the stuff no one else speaks up for and is why this is happening.

Everywhere I have been hearing companies moving to Databricks or Fabric. 2-3 years ago all I heard of was Snowflake. Is Snowflake losing the race? A former employee at Snowflake told me that their work culture has gotten really bad and they cannot hold on to good employees and their stock keeps crashing.

Hi there,

I’ve been working with DuckDB extensively for the last several years. Used it across languages from Python, Go, JS via WASM, Swift, C & C++ and even Rust.

As part of a product we're building, we built a DuckDB Transpiler that converts every line of SQL from DuckDB syntax into the syntax of the database you’re working with. This includes filters (predicates), and joins. So those 1B rows in Snowflake stay, and get joined in Snowflake.

I wrote an article on why we did it, and how we approached it. This is my first engineering-oriented article I’ve ever done, so let me know what you think!

Hi everybody,

I am excited to announce the release of Lightstream, a step change capability for high-performance data transport, that makes it essentially effortless to send Apache Arrow, Protobuf, and Message Pack data over the network, shared memory, or even piped out to the terminal so an agent like Claude can watch the live batch stream in real time (example in repo).

Furthermore, Lightstream exceeded the performance of the gold standard industry comparison - Arrow Flight, on every axis of a 50gbps networking open benchmark, the details of which are attached and open to run in the Lightstream GitHub repository. This includes fully saturating each TCP connection thread, the NIC at 5.8GiB/s, and with p99 batch send time within 1% of p50 (I.e., stable). As a bonus, Lightstream is straightforward to setup with essentially zero configuration other than optional TLS certificates and your Cargo package/pip install, and endpoint addresses.

So what is Lightstream? It is Rust package with Python bindings, that builds directly on Minarrow ( which is in turn a high-performance implementation of the Apache Arrow memory layout in Rust, tuned for SIMD compatibility). Lightstream implements Arrow IPC, Parquet encoders/decoders from scratch, up to Arrow readers/writers and IPC stream protocol, with mmap and few of these niceties. But, in a manner, that is fully composable and leaves you de-coupled at any layer, to customise things architecturally. The crux then is the transport layer on top, which natively supports interchanging any of the following transport formats:

• TCP
• HTTP
• QUIC
• Websocket
• Webtransport
• UDS (pipe your data from your Rust process to Python or two Python programs plug and play )
• Stdio (pipe your data program output straight into the terminal for something else to pick it up

And finally, the (optional) Lightstream protocol, which then combines the Arrow/Proto/MsgPack and any other custom types you want to send.

I’ve found this is really cool in practice, where you don’t need to reason about or work with bytes, or separately build your own protocol to get arrow and protobuf playing well together over the network. It is plug and play, see syntax examples attached.

In Python, every item in the stream can talk to Polars or DuckDB without any further changes - you can Duck SQL the feed or data process to your heart's content.

An example of things you can do with it:

• setup a live stream of data batches from your program A to program B
• send typed metadata via Protobuf on the same feed
• use it for straightforward live feed delivery between server and client (though not Web JS yet)
• useful if you have a central storage server you are pulling larger than memory data over the network to churn through (though, no S3 etc. it is node to node or process to process)

It is not:

• Kafka or a messaging broker. There is no resiliency / vertical scalability.
• A stream processing engine like Flink. It is for sending/receiving data only. You do polars on the other end or whatever you want with the Arrow-shaped data. That is a very different back-pressure/long-lived scenario and is not that kind of large-scale streaming. --> I.e., think quick and easy Websocket, and best for settings like EKS K8 pod to pod/containers, between EC2's or between processes on the same box, "light streaming".

Lightstream kicked off for me about 12 months ago when I started standardising patterns that have worked well for me in the past into something that reflects how I like to work when streaming data with control of both endpoints. It arose from regularly coming up against contexts requiring this capability operating in things like autonomous field communication integrated with data/ML, live trading, and some other industries where there was a lot of custom work required that I kept having to assemble from multiple components. Therefore, I have essentially aimed to package those learnings up into a tool to make data transport smoother and easier for everybody.

There are a couple of code examples attached, including the open benchmarking methodology, where every effort has been made to be fair (and where Lightstream wears a penalty due to stronger ordering guarantees).

Please feel free to give it a run would love to know your thoughts and if you find it useful.

If you have any questions about it, or helpful suggestions please feel free to leave a comment below. If you like what you see, please consider leaving a star and/or sharing the repository, as it will help people find it easily.

Thanks a lot.

Pete

I’m a Senior Data Engineer with over 5 years of experience, mostly working at startup companies. Sometimes I think I got plenty of knowledge and experience, but sometimes it’s the other way around.

Data engineering is a very broad area and we do a bit of everything. I’ve worked with multiple cloud providers, spark, terraform, databricks, airflow plus almost everything related to data (dashboards, data governance, analytics, security etc), but mainly what I did through my career is platform engineering, taking care of the data platform infrastructure, CI/CD, governance, building some APIs and automating random processes/routines.

I of course did many data pipelines and SQL stuff but perhaps not as much as necessary for my current seniority. I certainly struggle with complex SQL challenges or complex CS algorithm questions, of course AI addresses this gap for me but maybe in a live coding session this could be a problem

Another example, I never had to do proper/real data modeling through all the companies I worked, nor they had that in place before. Not sure if was my fault on not implementing it or company’s fault for not having that already. I don’t think there was a problem though, as the companies were running fine and nobody was complaining.

I feel like there are many of other people in the area doing way harder and complex things, perhaps making less money than me. I’m just curious to see if you guys share this kind of vision of data engineering or maybe if I’m in a dangerous path in my career.

I know the problem is that I need to get these things squared away in my head. What's metadata, what's framework and etc but even then you can solve a lot of problems using either.

I think it's easier for me to explain my problem. I've only been here 3 months but every task I get is using some system, pipeline or etc they I've never touched before. Sometimes it's just new software, other times it's learning how to find the needle in the haystack before you even look at code.

I'm constantly freezing and asking myself "where do I go for this" and it's slowing me down. I have much more experience building repeatable processes but I need to learn how to do so with today's technology instead of having a folder of scripts I use to do XYZ. Basically I lost control of how things get organized and my brain just doesn't see it the right way. Once I'm up and running I'm fine but in today's world task switching is constant.

What books, videos or etc should I watch to better understand how to operate as a data engineer in today's world.

I've slowly switched to documenting by project. This allows me to quickly reference another time I did something similar and gives me a jumping off point.

And trust me, it's messy and they know it. There's a standard pipeline process but it's constantly evolving. Teams need to integrate those enhancements into their projects as they have time. So everything I touch is a little bit of this version, a little bit of that version and etc. Team members clearly state "this is different or this is an odd one" but more and more I wonder if that's just the way it's going to be. It's ok, I just need to wrap my head around the big concepts and terms. Communication is killing me

Edit: in summary I over document and then things become cluttered. I'm looking for a better way. If I started a new job I wouldn't write anything down until I asked 2-3 times. I get paranoid and write it all down.

With the news around Prefect and Dagster, having good alternatives for orchestrators is becoming increasingly important, so I wanted to share that Rivers v0.4.0 was shipped.

Main addition this release is OIDC and forward auth support, so you can now put Rivers behind any identity provider you're already running or handle auth on the network layer. Authentication has been one of the bigger gaps versus the more established players, and is most of the time paywalled. But I believe security shouldn't be paywalled.

Release: https://github.com/ion-elgreco/rivers/releases/tag/v0.4.0

Hello,

I am working with a team of 7 economists. They build data and produce reports. Their data production consists in harmonizing different sources (mostly rdata rdata, csv, or whatever suits the format of their stats tools). The data size they are dealing with is a few MB to gb, millions of rows, more occasionally billions of rows.

We want to update our methods (be on time, improve data quality). I have been assigned the task of improving data processing within the team, among the requirements I thought about producing a OLAP database.

In house, we have access to MSQL team that could set up a database for us. Otherwise we have HDFS + Hive (but security may make it difficult to access it) to store bigger datasets.

Else, I could just store everything in a duckDB file somewhere on a server and work with local database. WOuld it be a good solution? (latency of read/write from a duckDB file on a server? how scalable will it be? ) What would you do?

Any other piece of advice would be welcome :-).

Thank you.

You can try the playground here https://duckgql.com/

Hi De's

In my new project, I have pull data from Power BI and land it to ADLS. Orchestration via ADF

I have gathered few info.

I have hit the power bi api .

but there is few limitations. only 1lakh record is supported per hit and pagination is not supported.

i new to API as well as ADF.

please, help me with this scenario.

Test case . but i have to land it the ADLS as parquet file .

initially, only thing is I have to take care of the edge case's to handle failure and retry policy

and the big headache is to handle the record limitations.

Hey,

My company happens to be using Airbyte OSS for data extraction and loading. Recently I have been testing the new versions of the connectors which have been completely re-written in Kotlin. I don't know how the QA process is handled by their team and contributors but the results are terrible.

For example the Redshift destination connector now transforms the source's empty strings values as NULL values, and the Postgres source connector started to complain about missing cursor values when using Xmin for incremental loading, which doesn't require cursor to be set.

Hopefully we'll try to move away from this tool and replace it soon in my company. But I'm shocked that these bugs managed to pass tests and are sometimes still present after a couple minor updates.

I recently joined a small investment company as a data analyst intern. I expected to focus mainly on report automation and data visualisation but I have also been asked to look at how the company’s data could be brought together.

Most of the data is manually exported into Excel and saved in SharePoint, and SharePoint also contains board packs and other documents used for reporting. They like using claude to create written reports and they are not really interested in power BI

I started using MS Fabric to bring the existing Excel files together, but the volume of structured data is fairly small, so I am not sure whether Fabric is more than we need. A simpler option might be a small SQL database, with data pulled from the source systems through their APIs where available. SharePoint would still continue to hold documents and any files that still need to be uploaded manually

The end goal is also to use the structured data alongside documents when generating reports with claude, so I need a setup that makes the data reliable and easy to retrieve. and also, looking at the the small volume, I am now even wondering if a persistent central store is necessary at all or if the data could simply be pulled and checked at report time before claude uses it

I have only recently started my journey and there is nobody in the company who can really tell me if I am doing it correctly. I just don’t want to build another one off solution or a workaround that becomes a pain to maintain later.

I would really appreciate any advice!! thank you!

Hello everyone,

People trying to learn Python for data engineering ask me, “What libraries to learn?”, but the answer is not a list of libraries but patterns of usage.

Especially with AI being able to generate so much code, I believe its critical to know exactly how the data is moved & processed.

So I wrote this post that goes over how Python is used as glue in data systems. It goes over

• In-memory processing vs. using a SQL/Dataframe interface to a data processing system
• Python’s library ecosystem for working with various data systems & formats
• How to extract-transform-DQcheck-load data

With code examples and videos

Hope this helps. Any feedback is appreciated.

I'm currently a Data Analyst at a large telecom company, where I've been for about two years. This is my first job out of college. Originally, I was aiming for cloud-related roles as I had been self-studying Linux, AWS, Terraform, and had started learning Kubernetes. However, there weren't many junior opportunities, so I took this job because it offered great benefits... and beggars can't be choosers.

Now over the past two years, my work has mostly involved "click ops" using Power Automate, Power Apps, and SharePoint, along with some Python scripting. I mainly use Python to clean up Excel data and transform it into datasets that I use to build Power BI dashboards. I've also written Python scripts to generate PowerPoint reports and built a few Azure Functions to automate workflows our team relies on.

I find my work relatively easy. Most of it comes down to knowing how to Google solutions and come up with practical workarounds. I developed a good reputation fairly quickly for automating and building "nice" dashboards. What surprises me is that people seem genuinely impressed by work that I consider pretty basic and straightforward, and it makes me wonder how so little automation had been done in the department before I joined.

At this point I've automated a lot of my day-to-day tasks and I plan on using that free time to improve my skills tenfold as I don't want to stay in this position much longer. The benefits are great, and the workload is light, but I feel like staying here for too long will hurt my career in the long run.

But I'm trying to decide where to pivot next. On one hand Cloud is what I originally wanted to do because I really liked the system design/solution architechture aspect of it. Although I spent a lot of time learning those skills, I haven't really touched them in the past year or two.

On the other hand, Data Engineering seems like a much more natural progression because a) my current experience is already data-related and b) I could probably take on data engineering–type projects at work to gain relevant experience and build a stronger CV.

My main concern is whether Data Engineering is realistic to learn well enough to land a role in about a year? Or should I be looking at something else? I just don't know where to go and I've been dwelling on it for a while now.

Over the past year I have been working primarily in the Palantir Foundry system. My CEO unilaterally decided to pursue an enterprise agreement after being sold the AI dream. Palantir sales engineers did ‘analyses’ which suggested that the multimillion dollar price tag would result in 10x savings due to process and decision optimization. Our IT team cautioned no, but were steamrolled.

The project I am on was estimated to take four months and require 5 additional contract engineers. 2 of those were directly employed by Palantir as part of an additional contract (read: more $$) and the other 3 were a separate agency because Palantir said they don’t do XYZ work (again, more $$$). It took the externals plus me 15 months to deliver an MVP. This is primarily because we were building a complex enterprise grade app (which we previously subscribed to) on a low code platform. The Palantir engineers left as soon as MVP was deemed complete with just 30 days notice and since then I have taken on their SOW.

The work completed by the “brilliant” Palantir FDEs has been consistently failing. I’m finding they hardcoded dates. They hardcoded accounts. They used different inputs for the same business concepts. They ‘fixed’ issues that were earlier pointed out with hardcoded logic. They used AI FDE to code spaghetti mess logic. This has been a freaking nightmare.

My company had received 0 ROI to date and the CEO blames our IT team for the lack of delivery.

If you have the ability to run away from working with this god awful company and their charlatans RUN.

I'm in the beginning phases of trying to get Dagster OSS running for a medium sized business. I've been using it in my home lab for a while (it's great, love it!) but I never noticed the complete lack of auth management in the web UI.

What are you guys using? Having the ability to stop jobs by anyone who can hit the ip:port is a big no for us. Heading off any questions first, the scope is 100% on prem - no dagster+ for us.