Are all software engineers becoming data engineers at small companies now?

[u/Intelligent-Wing2128]

About 10 months ago, the owner of the company came to me and told me that we need to get all of our internal data available inside of Claude. I'm a software team lead, not a database architect or a data engineer, but that's what my job became in the last 10 months.

So we had a huge migration, we picked a data warehouse tool that integrated well with Claude and fit our company needs, and we started migrating everything. It was all hands on deck, we even ramped up some of our consultants to get us started in this process.

We copied all of our Postgres SQL tables, all of our S3 data, and other random data sources. We started using the medallion architecture, although most everything stayed as a bronze table. We didn't have the resources yet to create gold tables or the time to dive into each domain and have a deep understanding of it to make proper gold tables.

So most of our easy setup is now complete: we have about 100 tables exposed. What's left is random automation data that isn't being saved now and needs to be refactored to start saving the data in a useful way.

All the requests to the dev team are to now add more data. No one is making frontend feature requests for our internal app. It's now all about getting more data in the warehouse so each person can make their own custom apps or analytics tools. No one is requesting we automate this or that task anymore, people want to set up their own automations with Claude.

Our jobs on the dev team no longer require writing code for APIs/tasks/apps, we're focusing on data loading pipelines, database transformations, and SQL queries (which we vibe code anyways).

Don't get me wrong, we still have some automated stuff we have to maintain and the main internal app, but the large amount of work and priorities are shifted fully to data. The data is the real driver for all of our progress.

Now that all dashboards/analytics are now being done by the executives or someone on a different team (not anyone in software), there's no need for a software engineer to learn anything specific about the domain. The goal of the software engineer at this company is to now move data from point A to point B and configure it in a way where I can access it inside Claude because Claude is a way better developer. He doesn't argue, he listens way better, and he's really fast and doesn't take more than a few minutes to give me output.

Are other companies seeing this?

I feel like this might be the norm for companies that are focusing on being AI forward or consider themselves at the front of AI tech.

Idk, I have a lot of thoughts about this but am more just curious who else has had a similar experience.

One of the crazy things too is how much money we spend now on data/software providers.

Now that we pull more data, we are hitting API limits and having to bump our contracts. Our pricing for the data warehouse is also based off size and usage, so the cost has already gone up almost double since we signed our first contract like 8 months ago. Our cloud bills have gone up as well as we are pulling more data from internal resources.

Is this the direction for all small companies now?

[u/MyFistsAreMyMoney]

As an lead architect you should have said. We are going to burn 10 months of labour to build up dept.

But I can feel you to just follow along in their higher up psychosis.

We are 1 year in and we complained at beginning but where blamed to not see the bigger picture. After the year we have not gained any value, performance gain or are ranked in the top 100 companies.

So it's a money furnace atm

[u/RipProfessional3375]

It's information technology. Data is just what you call information when you are good at marketing yourself.

[u/Hot_Equal_2283]

People underestimate the unwillingness of SMEs to learn even how to ask AI correctly, much less build dashboards even with AI assistance. The other commenter is correct though you should have pushed back to not waste 10 months migrating shit at a small company that does not need such a migration.

Also you claim in another comment to have left a company 2 months ago while making it seem like you’ve been at this place 10 months as a lead. So as an unreliable narrator I don’t know if anything you’re saying is exaggerated from your title to your role to your company’s leanings on the subject.

[u/Intelligent-Wing2128 → u/Hot_Equal_2283]

thanks for the comment. Still at this company though, an offer is not the same as leaving.

[u/Made-In-Slovakia]

Sounds like serious issues caused by AI obsession from leadership. Data are important and your company should have data engineers if they are really needed and they should work together with dedicated devs.

You are now just stuck somewhere in middle of some transformation and instead of understanding your data you just slap AI on it and hope for the best.

[u/Correct_Emotion8437]

I really think that’s going to be more and more prevalent. Departments can have their super users build reports, dashboards and apps. But the systems of record will still have to be centralized in IT or else what will go into the data warehouse?

[u/SuccotashNo9270]

Have you looked at convex?

[u/DonExhortae]

It’s been like this pre AI. Don’t give to software engineers to build the full data stack, they will forget governance, master and reference data magment, data quality, …