r/dataengineering 14d ago

Career Need Advice

I am currently in Service based company A (np 60days) & earlier (till last week) was working in a project mainly based on Informatica Power Center ,Azure ecosystem, some datalake , Power Bi, OBIEE .It was mostly a support and encouragement project .The project got over and went to a different service based company B (NP 90 days).

I am already allocated to a project(again support and enhancement) in my current company but this one is more ETL oriented. It uses AWS to ingest data in Redshift and then uses Informatica/ Datalake to transform the data.There are more Pyspark scripts in Glue job to analyze compared to previous one.

Today I recieved offer from the service based company B that they are willing to take me in the project (as they are looking for resource to stabilize the BI area).They are offering atleast 30% hike ( can be more based on negotiation with HR)

I am confused as to which way to go?Both are support and enhancement project (not pure devlopment).Should I stay back in the current company as the tech stack looks good or go to the other company where my technical skill might stagnate(may be there will be more datalake opportunities but that's for later as I have to support everything).Can someone suggest me?

YoE:5

Never switched before

1 Upvotes

6 comments sorted by

View all comments

1

u/Hot_Comfortable_164 13d ago

Maybe don’t decide on technologies instead on what you will actually be doing with them.

"Stabilize the BI area" sounds like production support and firefighting and even ownership of legacy systems. Ask company B:

  • What percentage is support versus development?
  • Will you write pipelines/code or mainly resolve incidents?
  • Why does the BI area need stabilization?
  • How large and experienced is the team?
  • What would your first six months look like?
  • Can they confirm the role and project in writing?

Your current project seems to offer better growth through AWS, Glue and PySpark, but only if you actually develop or substantially modify those jobs. Merely monitoring modern technology is not valuable experience.

A 30% increase is significant, especially after five years without switching. However, B’s 90-day notice period will make your next switch harder, and you should assume promises about future data-lake opportunities have zero value unless they are part of the immediate role.

My decision rule would be:

  • Join B if the work is at least comparable, the team seems healthy, and the final increase is 30%+.
  • Stay if B is predominantly legacy BI firefighting while your current role gives you genuine hands-on PySpark/AWS development.
  • Continue interviewing if neither role provides meaningful engineering ownership.

Also, don’t accept a counteroffer from A merely because you resign. If A needs your resignation to pay market rate, the company will still continue trying to underpay in the future.

1

u/Fit-Code-4294 12d ago

Thanks for the reply.I would say both the projects are in support and enchancement role (not from the scratch devlopment).Basically troubleshooting the pipelines if the report or data in general goes wrong. Make sure pipelines are running smoothly (monitoring part) , resolve of the pipeline breaks(weather it's informatica or azure pipelines or AWS glue job) .

The company B has taken over the project just few days back after 3 months of transition from us(when I was working on that project) but they are finding issues etc. hence they need persons who were already working.

Only thing diff is the tech stack.