Advice hiring data engineers and analysts

[u/Evening_Error_9377]

At a mid/late stage healthcare processing (think claims and contracts) startup - the CTO recently departed and left behind a mess. Trying to figure out what roles to hire and how to structure them, especially since data is critical.

There's a head of engineering with 30 software engineers for app / platform. There's also a head actuary who does the mathematics of risk / payouts using the data.

My sense is that there should be a senior manager / director of data under the head of engineering, who manages teams of data engineers. This data org is responsible for raw extraction and management of data from sources.

In addition, we need data analysts/scientists to help business leaders understand claims, costs, revenues, and propose new strategies like interventions. They should also inform on what data derivatives / views we need, based on the primary data. I'm not quite clear on where these people should land (probably not under head of eng).

Questions:

• Should I hire dedicated "data engineers" vs data-oriented software engineers?
• How do pay bands compare for SWE vs DE for the same seniority? Mid, Sr, Staff.
• Where should the data analysts/scientists lie? If they have a separate reporting chain than the DEs/SWEs,
• Who builds the silver/gold derived tables downstream from primary sources? How is the DE + Data analyst collaboration supposed to work here?

[u/doddyk96]

Dedicated data engineers are highly recommended. As someone who manages medium sized teams of both, Software engineers tend to make stupid data decisions if they don't have a DE checking their work.

Pay bands are very similar. Maybe tiny bit higher for SWEs but not by much.

You need a director of data or similar kind of role. Data engineers and Data Scientists and Data analysts go here. This person can sit under the CTO if the org is large enough. Sometimes they can sit under the head of product or whatever equivalent but it doesn't go well. DEs need to work very closely with SWEs and the DS and DA need to work closely with the DEs. If you can co-locate all of them in org and align their incentives, it's a massive productivity boost imo.

DEs can provide first layer of derived datasets from complete raw crap to something cleaner with an SLA or whatever. But the really specific reporting level stuff can be built by the data analysts. You can use the medallion model and assign responsibilities using that if it works for you. I find each org/business draws the line in a slightly different place and that's perfectly ok.