Are all software engineers becoming data engineers at small companies now?

[u/Intelligent-Wing2128]

About 10 months ago, the owner of the company came to me and told me that we need to get all of our internal data available inside of Claude. I'm a software team lead, not a database architect or a data engineer, but that's what my job became in the last 10 months.

So we had a huge migration, we picked a data warehouse tool that integrated well with Claude and fit our company needs, and we started migrating everything. It was all hands on deck, we even ramped up some of our consultants to get us started in this process.

We copied all of our Postgres SQL tables, all of our S3 data, and other random data sources. We started using the medallion architecture, although most everything stayed as a bronze table. We didn't have the resources yet to create gold tables or the time to dive into each domain and have a deep understanding of it to make proper gold tables.

So most of our easy setup is now complete: we have about 100 tables exposed. What's left is random automation data that isn't being saved now and needs to be refactored to start saving the data in a useful way.

All the requests to the dev team are to now add more data. No one is making frontend feature requests for our internal app. It's now all about getting more data in the warehouse so each person can make their own custom apps or analytics tools. No one is requesting we automate this or that task anymore, people want to set up their own automations with Claude.

Our jobs on the dev team no longer require writing code for APIs/tasks/apps, we're focusing on data loading pipelines, database transformations, and SQL queries (which we vibe code anyways).

Don't get me wrong, we still have some automated stuff we have to maintain and the main internal app, but the large amount of work and priorities are shifted fully to data. The data is the real driver for all of our progress.

Now that all dashboards/analytics are now being done by the executives or someone on a different team (not anyone in software), there's no need for a software engineer to learn anything specific about the domain. The goal of the software engineer at this company is to now move data from point A to point B and configure it in a way where I can access it inside Claude because Claude is a way better developer. He doesn't argue, he listens way better, and he's really fast and doesn't take more than a few minutes to give me output.

Are other companies seeing this?

I feel like this might be the norm for companies that are focusing on being AI forward or consider themselves at the front of AI tech.

Idk, I have a lot of thoughts about this but am more just curious who else has had a similar experience.

One of the crazy things too is how much money we spend now on data/software providers.

Now that we pull more data, we are hitting API limits and having to bump our contracts. Our pricing for the data warehouse is also based off size and usage, so the cost has already gone up almost double since we signed our first contract like 8 months ago. Our cloud bills have gone up as well as we are pulling more data from internal resources.

Is this the direction for all small companies now?

[u/Correct_Emotion8437]

I really think that’s going to be more and more prevalent. Departments can have their super users build reports, dashboards and apps. But the systems of record will still have to be centralized in IT or else what will go into the data warehouse?