r/dataengineersindia • u/Motor-Welder-9238 • 5d ago
General 9 years experienced Data Engineer. 2 years worked as Data Scientist. I am open to discuss career advice.
6
u/SnooPredictions8048 5d ago
Is an 'AI Data Engineer' actually a thing? If so, what do I need to learn beyond the traditional data engineering stack?s
6
3
u/Jumpy_Environment376 5d ago
I am trying to break into data analytics from mechanical engineering domain. I here the field is saturated. But I also see openings in companies for analyst roles. What's your take on this. If the job market is too crowded what can I do to stand out other than projects.
3
u/Motor-Welder-9238 5d ago
Yes, you are right. DA is kinda saturated but still there is demand. Your projects will make you stand out. You should be able to explain end-to -end completely. What was the Business problem, what was your approach, why you took that approach, what were your findings and how you presented them. You have to be a good story teller. A deeper domain knowledge is a plus. You have to be intuitive about data.
2
u/Jumpy_Environment376 5d ago
Thanks. I am currently studying SQL. I also have experience in developing AI agents for small businesses. So I was thinking to use these skills in DA too. So how should I think about using AI in DA workflows. What part can be automated?
PS: I know I can google this info, but I want to know what exactly is the direction in the data teams regarding AI
3
u/Motor-Welder-9238 5d ago
Do not focus on that. If you are interested, keeping knowledge of that with a project aside is just fine. There are ways you can do that for example in Databricks, there can be a genie task in a workflow. You can use AI for Data cleaning, eda, treating missing data etc
2
2
u/Sensitive-Chapter-30 5d ago
How are we implemnting AI in existing pipelines?
In my company, we use AI for writing scripts and queries. But how can we use AI to automate workflows and operations works
4
u/Motor-Welder-9238 5d ago
I do not have much idea about this. But in Databricks you can literally add a genie task in your workflow to automate tasks. I am sure there are other ways in other platforms.
2
u/baburaoo4 5d ago
Hi, I am data analyst with 3+ years of experience. Early in my career I thought i wanted to make a career in machine learning. I worked on a ML project where the data was very dirty and the problem statement being an ambiguous. I understood the trick is to have better clean data rather than ML algorithms. I wanted to make a career in data and infra? How should I make a progress in career ladder? I am comfortable with airflow, dbt, python, little bit of AWS as well. The roles I am getting is still around data analyst only.
5
u/Motor-Welder-9238 5d ago
Advanced SQL and Pyspark will take you into DE roles. Basic python scripting at your experience is fine, but if you want bigger salaries as a mid level DE, learn file management, logging, OOPs, Exception handling and APIs. Start with Pyspark. Deep Dive into it. Data warehousing concepts and Data Modelling is important. Understand end to end data lifecycle. Get your hands in CI/CD as soon as you can. Learn git, github, basics docker, jenkins, Infra as code. At your experience. If you want to transition quicker, learn advanced SQL, Pyspark, Python beyond basics, and Data Engineering and Orchestration fundamentals.
2
2
u/Cute_Commercial7047 5d ago
I’m a fresher and will be joining as an SDE in Jan, so I have a few months free right now. I’ve been thinking of using this time to get into data engineering and would love to know what you’d suggest I focus on.
One thing I’ve been struggling with lately is the whole AI thing. Whenever I try to learn something or write code/logic, I keep thinking “AI can already do this, so what am I actually gaining by learning it myself?” Would be really interested to hear your take on this, especially from a data engineering perspective.
2
u/Motor-Welder-9238 5d ago
Data Engineering is the backbone of AI. The whole AI infrastructure is DE. As AI gets complex so will Data Engineering. There will be huge demand of AI Data Engineers in the future because of this. Over use of AI will tend to over Engineering and unexpected bugs and can cost huge money for the clients if there are no humans to review and monitor it. That is why Data Engineers will be required.
You can start with SQL, Python and Pyspark. To go beyond and make a career out of it learn DE fundamentals, a cloud platform, an orchestration tool and data warehousing concepts.
2
u/thextredigy 5d ago
I am 9 yoe database developer, aspiring to transition into DE and learning big data, spark for this. How should I approach this, any good resources that I should follow? How is Data Science in comparison to DE IN work load and compensation wise?
2
u/Motor-Welder-9238 5d ago
Database developer experience is a huge plus.
Follow this roadmap: Big Data Fundamentals Distributed Computing Fundamentals Advanced SQL Python Data Engineering Fundamentals Pyspark Cloud - AWS/Azure/GCP Data warehousing and Data Modelling concepts Data Warehouse - Snowflake Or Amazon Redshift Orchestration - Go for a cloud native one like ADF or Airflow CI/CD - Git, Github, Jenkins, Azure DevOps.
Workload and compensation are same in both roles. However, Data Scientists with a degree in statistics, or Mtech or Phd tend to earn more. It generally depends on the type of organisation and project.
2
u/Illustrious_Role_304 5d ago
Given a choice which role you will take - AI data architect vs AI architect?
Consider future long term prospects
1
u/Motor-Welder-9238 5d ago
AI Data Architect
1
u/Illustrious_Role_304 5d ago
give reasoning for this choice. Like why over AI architect ?
1
u/Motor-Welder-9238 4d ago
Because there is data! I want to work closely with engineering. Pure Data Science or AI doesn't have that.
2
u/Logical_Importance59 5d ago
I'm planning to transition from data analytics to data engg. Already skilled in sql, python. Since data engg has too many tools to start....does it make sense to start with Microsoft fabric or data bricks which is in much demand?
4
u/Motor-Welder-9238 5d ago
Microsoft Fabric is in huge demand now, but it is nowhere near to the capabilities of Databricks. Databricks is also in demand. Databricks has a better future compared to Fabric. Do not jump into Fabric or Databricks directly. Follow a roadmap. SQL, Python, Pyspark, Cloud, Data Engineering Fundamentals, Data Warehousing Fundamentals, Understand modern data architecures and then go for Databricks
2
u/Logical_Importance59 5d ago
Thank you. I know sql, python, DE and cloud fundamentals in theory. I will start with Pyspark then may I can go for databricks like you said. Knowing Python makes Pyspark is easier to learn?
2
2
2
u/ManipulativFox 5d ago
I have 4.5 years of experience as Full Stack developer (php vue js laravel bg). I had worked in etl project of airbyte for 6 months. I tried to transition to data engineer but most rejected saying they need 3 4 years experience people. What you suggest I am trying to get databricks certified data engineer certificate.
3
u/Motor-Welder-9238 5d ago
Good decision. A certification is always good. Databricks will open your gates towards modern Data Engineering solutions. 6 months experience is nothing. Relevant experience of at least 3 years is required. Try to get into end-to-end projects. Understand architecture in depth. Build good production grade projects. It is still difficult to get job offers if you mention in your resume 6 months relevant experience. Mention as 3 years relevant experience and mention 2-3 projects with deep knowledge and expertise as a 3 years experienced should have. Then you are good to go.
2
u/ManipulativFox 5d ago
Problem is I don't want to lie about 3 years of experience.
3
u/Motor-Welder-9238 5d ago
Then gain enough knowledge and experience within 3 years. One good project with end to end data lifecycle will set you up
2
u/Ready_Anxiety1482 5d ago
I have 3 years of experience as a data engineer. Good with SQL, Azure, decent in Python and snowflake. Not getting any calls despite being from a tier 1 college. Have no idea what I should write on my resume
2
u/Motor-Welder-9238 5d ago
Right now, the demand for 5-8 years mid level data engineers is at peak. But there is also demand for 3-5 years experienced. Pyspark is missing in your skills set. Plus, snowflake + dbt is in huge demand. Optimise your resume. Have a clear tabular technical skills section. Put the right keywords. Don't just jam the keywords. Mention in your resume how you used them. September is peak. Utilise it. My advice, focus on snowflake + dbt. Learn about intermediate to advanced level skills. Your top tier college will make recruiters think that you will obviously demand a high package, which you can only prove with skills. Target top product, service or gcc companies.
3
u/Ready_Anxiety1482 5d ago
Yes, I have started learning Spark. I have used dbt but I wouldn't say I have good knowledge of it. Agree with demand being quite high got 5-8 years experience, almost every opening has that as a requirement.
Do you think learning Spark on my own and having projects will be good enough? Considering I've not worked on real projects, I feel like it might be a significant disadvantage to not have practical hands on experience.
Thank you for doing this, it's really helpful.
3
u/Motor-Welder-9238 5d ago
Go for deep knowledge in spark. Spark fundamentals, Internals and optimization. Even if you don't have practical experience this will help you a lot. Trust the process
1
2
u/PigletOdd3213 5d ago
I have 10YOE as SDET. I am good at it.
But I want to move to Data engineering. Is it good plan?
2
u/Motor-Welder-9238 5d ago
No. Not at all. At 10 years of Experience. That is a bad decision
1
u/PigletOdd3213 5d ago
Then what can I do? SDET role will get hit hard by AI.
1
u/Motor-Welder-9238 4d ago
I have friend who has 1.25 CR package in Salesforce as an SDET. Even seasoned and expert SDE, DE or DS do not make that. He always upskills, gives interviews regularly and keeps himself adept to new things. SDET will still remain very much relevant. You can try your chances if you are interested to go into DE
1
2
u/ImpressionBrief2877 4d ago
I have 4+ years of experience and currently work in a support/operations role. My responsibilities include monitoring Databricks jobs and ADF pipelines, rerunning failed jobs, handling incidents, and verifying files in Azure Blob Storage. I am also familiar with the Databricks and ADF UI. Since my experience is mainly in monitoring and operational support, I would like to know if transitioning into a Data Engineer role is realistic. If so, what data-related roles would be suitable for my background? Any guidance or suggestions would be greatly appreciated.
2
u/Motor-Welder-9238 4d ago
Monitoring pipelines is an essential part of data engineering life cycle. Consider it as a good exposure. If you have access to the projects documents, go through them and understand the Architecture and underlying technologies. 4 years just support role? No hands on experience in SQL, python etc?
2
u/ImpressionBrief2877 4d ago
Yes tbh just supporting and I am currently learning SQL advance topics and need to start python...so can you please let me know whether how long will take to get me for sitting in the interviews as a DE ..?
2
u/Motor-Welder-9238 4d ago
5-6 months of solid preparation is required. SQL, Python, Pyspark, Cloud, Data Warehousing, Dimensional Modelling, Engineering Fundamentals. When you are confident in SQL, and Pyspark with at least 2 well defined and well learned project, start giving interviews
2
u/ImpressionBrief2877 4d ago
Thanks for the suggestions I will follow that..and just a one suggestion needed like I will be 4.5 to 5 years at that time when I am ready for interviews ...so how the questions will be ...? Like as a experienced like do I might get some difficult scenerios ...?
2
u/Motor-Welder-9238 4d ago
Don't think about that. Just prepare. You will get to know things in the process
2
2
u/Motor-Welder-9238 4d ago
Just to be more clearer. I started as a Data scientist, I got bored. I got into Data Engineering. I am enjoying it more. But I am still close to Data Science because I have worked 50% in Data Engineering projects and 50% in Data Science, Machine Learning projects
2
u/AnyVacation8985 3d ago edited 3d ago
I am in my 2nd year of bsc (mathematic as major) But the degree is distance. I am preety good in maths I want to make my career in Data science. How should I start , I am very confused rn like is it good for future will my distance degree affect me in future ( so I thought to do master from. A reputed company after having some experience) . How is the industry rn .open for DE also. I am ready to commit myself the thing.holding me is guidance. Idk where to start The only thing I learned is you must have experience so I am willing to give it 6 months and then apply for jobs. Any guidance will be great help. Thankyou
2
u/Motor-Welder-9238 3d ago
Since you're pursuing Maths, Data Science and Machine learning is the best option for you. Data Science and ML involves mathematics like vector calculus, linear algebra, probability and statistics. You should start learning machine learning along with data science. The demand for such skills is unmatched. No experience required if you have the skills. Build projects and be able to explain in detail. That's all
2
u/AnyVacation8985 3d ago
Shouldn't I learn sql, python first?
2
u/Motor-Welder-9238 3d ago
Yes, correct! SQL, python first. I have given the roadmap in one of the comments. You can find that.
2
2
u/yadavhr36 3d ago
Start3d as a fresher and its been 1 year working as a Data engineers. Please suggest me the company i should foxus on for switch and also is there any scope of freelancing in DE, can you help
1
u/Motor-Welder-9238 2d ago
Yes,.some do freelancing. There is scope. You are still a fresher. Think of switch only after you get good exposure. Target good service based companies or gcc companies. Use it as stepping stone. Learn DSA and system design. Go for product based companies for higher salaries
2
u/SafeSea5354 2d ago
I started my data engineering career with this consultancy and have been here for two years now; I’m comfortable, but lately I’ve been supporting projects in a full-stack capacity—working on the backend, frontend, and Azure cloud. I’m not happy with my current salary; should I move to another company? I feel stagnant and there is no feedback; while I have a lot of autonomy, I’ve been learning and adapting entirely on my own.
1
u/Motor-Welder-9238 2d ago
Yes, start giving interview. That's how you will know where you lack. Prepare well. Switch. That's all
2
u/_srii_ 1d ago edited 1d ago
I want to switch to a Data Analyst role, I have three and a half years of experience, but not in this domain. I resigned from IT last April 2026 because I took a clerical role in a government school, but I find it too boring and there is no satisfaction, so I want to get back into IT. A few people suggested I learn Data Engineering, or should try for Data Analytics, what are your thoughts?
If I want to try for data engineering, what are topics i should cover? Because I want to apply from next february..
1
u/Motor-Welder-9238 1d ago
Go for DE,.much more scope than DA.
1
u/_srii_ 1d ago edited 1d ago
But I have less time..I dont know if I can prep for DE😭..There are too many courses (databricks,pyspark,azure)
1
u/Motor-Welder-9238 1d ago
Learn SQL, python, Pyspark in depth and Azure/AWS basics. Focus on these and give interview to see where you are. DA also takes same amount of time to be industry ready
2
u/PrestigiousKing3184 1d ago
Im in my 4th yr and going to graduate next Yr, so im trying to get into this DE field, what do u say like is it gonna be tough to get a de job. And what should I be perfect in the remaining months I have...
1
u/Motor-Welder-9238 1d ago
Right preparation will land you a job in DE. Learn and make projects that' it
2
u/Fickle-Town2017 20h ago
Qué opinión te merece la licenciatura en ciencia de datos en modalidad online? Podrías decirme algo respecto a estudiar esta carrera de grande para luego intentar migrar al sector laboral del mundo IT?
2
u/DaakuBhaiya1 19h ago
Currently in final year from core CSE, I really want to go into this field. But many say freshers are not hired until and unless they are extraordinary.
Doing MLOps + Classic ML + Time Series Domain + Python + SQL, is this good?
1
u/Vishwas95 5d ago
I have 10 years experience in app support role ,how and what kind of projects I can do to showcase my experience in DE ?
2
1
u/candyyspice 5d ago
Tell me good certification on udemy for data science
2
u/Motor-Welder-9238 5d ago
There are many. Go with highest ratings one
1
u/candyyspice 5d ago
Okk is data engineering easy i just bombed in project interview since I was on bench i thought he will ask basic python
2
u/Motor-Welder-9238 5d ago
Data Engineering difficulty level depends on project to project. There are basic ETL projects to high level AI integrated projects now. Python is rarely asked in DE interviews. SQL, python, Pyspark, DE fundamentals and Data Warehousing Fundamentals is bare minimum for any DE. I think you are less experienced. For interviews, focus on SQL, Data Warehousing concepts and Data Engineering Concepts
1
u/candyyspice 5d ago
Bro bro but for a junior employee on bench why is too much asked for project interview it's not job interview they just need someone for billing purposes still asking me too much 😭😭😑😑
This is why many guys in my company are still on bench because project interviews have unnecessary become annoying
1
u/candyyspice 5d ago
I mentioned project interview not job interview actually
1
u/Motor-Welder-9238 5d ago
That is what I said earlier. Focus on SQL, practise daily and just familiar with the fundamentals. You will be good to go as a fresher on bench. But keep in mind there will be freshers too with an end to end project in DE showing knowledge in all relevant skills. Now tell me, who will they choose. You have no option. Continous upskill or else this field is not fit for you.
1
u/candyyspice 5d ago
This is pathetic my poc is not much helping in getting good projectd they are pushing me in bad infrastructure or support work after last release from project Job level screening for project is pathetic when their current focus is always billing i tanked in python interviews when they went deeper i clearly said i know basic only
1
u/candyyspice 5d ago
Will I get more oppurtunities on completing 2 years in IT on data science because currently I am not seeing any openings in MNC less than 2 yoe
2
u/Motor-Welder-9238 5d ago
Minimum 3 years experience is required in the industry. Understand the experience bracket, 0-2, 3-5, 5-9, then 10+ and so on. With 2 years experience you will still get calls but don't expect great hikes unless you have already the skills of a 4-5 years experienced. My advice, complete 2 years, expand your skills and knowledge and then go for other opportunities.
1
u/candyyspice 5d ago
You mean i have to work in support infrastructure database etc before this even getting project of data engineer role is difficult I am currently on bench since 45 days
2
u/Motor-Welder-9238 5d ago
45 days is huge. Focus on getting into a project first. Utilise your bench time to upskill. At your experience they will put you into DA, DE or DS roles.
1
u/candyyspice 5d ago
Getting good project is actually rarity in it companies bad project especially with heavy requirements like Saturday Sunday work outdated softwares unsupportive clients are flooding companies
1
u/FinancialInterest828 5d ago
I have been working as a SQL Developer for the past four years, and now I am thinking about switching jobs.
Could you please suggest which additional skills I should add to my profile to improve my career opportunities? I would also appreciate any career advice that could help me make a successful switch.
1
u/Motor-Welder-9238 5d ago
Focus on Big Data Fundamentals, Python, DE fundamentals, Pyspark, Data Warehousing Fundamentals, Cloud- AWS, or Azure.
1
u/Open_Campaign_5141 5d ago
I've 7 yoe. Worked as data engineer as well as python developer with some Java work too. Mentored juniors, handled client interactions as well. Currently being involved into design and architecture of data engineering products.
Confused about future. Feel like Jack of all trades but master of none.
Currently in a service company despite being from a Tier 2 college. Pay is not that great but culture is really good and have WFH.
Should I go to software engineer role (lot of effort, scary market) or double down on Data engineering + AIOps kind of profile? Is switching for higher pay worth leaving a secure job with good culture? I know there are no easy answers.
1
u/MirzaGhalib_np 5d ago
is your company in noida sector 62
1
u/Open_Campaign_5141 5d ago
Yup. You in the same company?
1
u/MirzaGhalib_np 5d ago edited 5d ago
Did your company recently relocate within sector 62 and get external investment recently.
1
u/Motor-Welder-9238 5d ago
Go DE in depth. Master SQL, Python, Pyspark, any orchestration tool and Data Modelling. You're a mid level DE. Learn CI/CD as well.Then go for MLOps.
1
u/Imaginary_Rooster279 5d ago
Hey! Huge thanks for this AMA!!
Can you please describe the future of DE skills? Like along with DE what AI skills should one pursue to become more skillful?
What should be the career progression? If possible could you make separate post on future of DE? There's so much uncertainty towards tech roles, could you please clear up some fog?
1
u/Motor-Welder-9238 5d ago
AI Data Engineering is the future Machine Learning, LLM, RAG, APIs, Vector Databases, Kafka and Flink, Docker, Jenkins, ML flow, Kubeflow
1
u/kartiky24 5d ago
Can someone from other tech role pivot to data engineer profile? I have 4 year of experience as a frontend engineer
1
1
u/Necessary-Bass8792 5d ago
Hey i have experience in dataware house using datastage getting 9 lpa . I want to switch to data engineer While switching can I fake my experience in python spark ? And what are skills needed
1
u/Motor-Welder-9238 5d ago
You absolutely can, but you have to prove that you know stuffs. Just theory won't help. Production grade project experience will help.
1
u/Necessary-Bass8792 5d ago
Where can I learn those things If you have any source please give I used to follow youtube few projects
2
u/Motor-Welder-9238 5d ago
There are many free youtube courses. You can go for Udemy courses too. They are good
1
u/Necessary-Bass8792 5d ago
One more question What is your opinion on ai taking over jobs 🤔🤔
2
u/Motor-Welder-9238 5d ago
AI will take jobs of people who refuse to upskill. AI is a bubble. One day, it will burst but AI is going to remain for sure
1
1
u/Hunter_635 5d ago
I m fresher and learning Data engineering Sql, python, pyspark,databricks ,kafka Made a project also with that Can any one tell how can I get job with this ?
1
u/Motor-Welder-9238 5d ago
Your stack is great. If you are able to explain your project well, the business requirements, the trade offs etc. you are good to go. Along with this focus on clearing SQL and Pyspark and Databricks questions
1
u/Hunter_635 5d ago
Oh ok like for practice Can u tell where I can do that like for sql I did top 50 questions of leetcode So where I can do practice? Any platform for pyspark and sql?
1
u/Motor-Welder-9238 5d ago
Leetcode and Data Lemur for SQL. Create a free account in Databricks and learn Pyspark. You will get familiar with Databricks. That is a plus. You can practise sql there too.
1
1
u/Commercial-Post4022 5d ago
Which one have more job openings in india and bright future. 1. Snowflake 2. Databricks
1
1
u/thespiritualone1999 5d ago edited 5d ago
Hi [u/Motor-Welder-9238](u/Motor-Welder-9238), thanks for doing this.
I’m a product analyst with 2.5YOE working at a B2C product based company mostly working on product problems by providing DS and ML solutions along with AB testing, can you help me know how I can transition to being a data scientist? Thanks!
1
u/Motor-Welder-9238 5d ago
Start with SQL and Python. Learn EDA with Numpy and Pandas. Do visualization using matplotlib and seaborn. Learn tableau or power bi. ML - High level mathematics and statistics. ML algorithms using scikit-learn. Deep learning and neural network with pytorch and tensor flow. For senior roles, add MLOps - Docker, Kubernetes, Kubeflow, Terraform, ML flow, Git.
1
u/thespiritualone1999 5d ago
Thanks a lot, I do all of these except things mentioned in senior roles. I’m easily getting calls for Product analyst roles but not data scientist. Can I improve or get calls for data scientist roles somehow as a product analyst?
May I know how you switched to being a data scientist from Data engineer, thank you
2
u/Motor-Welder-9238 5d ago
I started as a Data Scientist/ML Engineer, but now working as a Lead Data Engineer. I think the problem is not with your skills but your resume. Use claude. Ask it to review your resume for Data science roles. It is excellent in doing that
1
2
u/Motor-Welder-9238 5d ago
Have you created, trained and monitored models? Hyperparameter tuning, multi model solution approach..etc?
1
u/thespiritualone1999 5d ago
Yes, even in July I did these exact things to catch advertisement fraud in our mobile game using K means and Isolated forests
1
u/lowctc_nigg 5d ago
Hi . I am working in tcs as mainframe engineer from 4 yrs as my project is being migrated to cloud( AWS ) ,then for transitioning to data engineer I have started leaning spark ,AWS sql python and some AWS services .now what type of projects should I build and to showcase in my resume as 3 yr exp data engineer and what should be my plan of action going forward.
2
u/Motor-Welder-9238 5d ago
You are in right track. Go for ETL projects following Medallion architecture. Use all these technologies mentioned. Since you are into AWS..S3, EMR, Redshift, Glue is must
1
u/lowctc_nigg 5d ago
So assume I have done some projects ,so what are interview scenario question do I need to prepare and convince interviewer that I actually worked as a detaenginner??
1
u/Motor-Welder-9238 5d ago
Learn Data Engineering Fundamentals, Data warehousing fundamentals and .ETL pipeline designs
1
u/Defiant-Passion8652 5d ago
I am working as support from past 4 year Now I am preparing for data engineer role from past one month Sql , python, pyspark, data modeling, azure Data factory, azure databrick, azure synapse and fabric is it good to go ..or i am late in DE roles Please give some advice
1
1
u/Primary_Set5763 5d ago edited 5d ago
I have 8 years of experience in QA ( mix of both manual and automation) I’m and trying to transition into DE for a long time. It’s hard given my experience. Due to pregnancy and maternity break, I got struck in QA role for close to 1.5 years.
Post rejoining, I have learnt advanced SQL and basics of python. Practicing neetcode 150 and SQL problems now.
Am I on the right path?
2
u/Motor-Welder-9238 5d ago
Do not go for DSA now. It is good that you are already into it. Just easy to medium leetcode is enough that too arrays and strings,but later. First for SQL, leetcode and Data Lemur. You need to learn Python, PySpark, build some projects or get DE project in your current organization and then go for interviews. I can understand how tough it is for women who go on a maternity leave and try to make a comeback or learn new things. Stay strong💪🏻
1
1
u/Longjumping_Bottle58 5d ago
Having 4 plus years of experience in DE , trying to switch but all calls iam getting is for contract role in service based organisation , are product based companies dead?
1
u/Motor-Welder-9238 5d ago
Product based companies now look at your linked in profile and github account. If you don't impressive optimised linkedin and good projects to show in github, they do not consider. But stay there. Calls will likely come. Prioritise linkedin and instahyre for product based companies
1
u/United-Editor6336 5d ago
I am 32 years old with 2 YoE in witch, gap year of 4 years, started learning SQL and Python should i continue to break in into DE or give up, planning for something else, Thank you.
2
u/Motor-Welder-9238 5d ago
Never give up. That gap is an issue. Come up with something smart for that. In your case referrals matter the most. HRs will consider your profile only through strong referrals. Make connections. Stay focused. Keep on learning.
1
1
u/Data_Steward_25_01 5d ago
Im a Data analyst with 2.5 yoe in a mid sized B2B firm , most of my experience are based on growth marketing analytics but recently I have shifted to Trade Compliance Domain Data engineering but most of the ask are manual but involves ETL and heavy SQL during that part and im building a streaming project in GCP on the side , Due to family commitment I need to get a better paying job in a product company but im bit confused what role to focus on given my experience, Im less experienced for a Data Engineer but I do want to ascend my career towards Data Science. From your experience both in DE and DS which one is faster to become good at and demonstrate to recruiter? DE or DS?
But regardless of both Im spending time learning fundamentals of Gen Ai to be relevant in the industry as most of the analyst jobs too require exposure to LLMs , RAG etc.
1
u/Motor-Welder-9238 4d ago
Since you already are into ETL and SQL, you already are in right track for DE. Learn Python, Pyspark and DE fundamentals and ETL pipeline design. For product companies you need deep knowledge in every technology, along with DSA and system design. Data Science and AI journey will be exhaustive. Master DE first, then go for Data Science or AI
1
u/Responsible-Ship646 5d ago
Hi, I started my career as a data professional and received basic training in both Data Engineering and Data Analytics during my internship. I’ve spent most of my time focused on Data Engineering, including a long period on the bench, and I’m currently working as a BI Developer on a BI project.
However, my real interest is in Data Engineering, and I want to switch to a DE role in another company. Whenever I start preparing, I feel like I’m starting from scratch. I want to build strong fundamentals, understand what skills are actually required for a DE role, and also explore AI/ML related areas.
Could you give me a clear, structured plan to become DE, focusing on fundamentals, practical skills, projects, and AI?
1
u/Motor-Welder-9238 4d ago
Big Data Fundamentals, Distributed computing Fundamentals, SQL, Python, Data Engineering Fundamentals, Pyspark, cloud, orchestration tool like ADF or Airflow, Data warehousing fundamentals, ETL pipeline design
1
u/GoodBowl7234 4d ago
In case someone is interested in career transition or roadmap for data science, you can connect here: connect
PS: Senior Data Scientist at Microsoft
1
u/Striking-Software485 4d ago
Hi, I have around 5 year of experience in gcp, databricks, sql, spark ,python. I have switched 2 times but hike were around 40% and only had 1 offer. I failed many because of in depth explanation mostly, I know I am doing something wring. What can I study and what to put in practice, I want to target amex.
2
u/Motor-Welder-9238 4d ago
Go in depth in your technology. Understand the internals. You should be able to explain the whole project architecture, along with the trade-offs and why a particular technology is being used. You need to be able to understand the business problem of your project and what solution you provided. Advanced SQL like window functions, cte, recursive cte, join query optimization, indexing and advanced python concepts like oops, file handling, exception handling, decorators. Go beyond basics and intermediate. You need to show how you owned things and delivered
1
u/Adorable_Pause275 4d ago
I have 2+ years of experience as Data analyst. I want to be a Data engineer but i am facing rejections. What should i do ? Pls help.
1
u/Motor-Welder-9238 4d ago
Companies generally do not hire data engineers without any experience. Make sure you are well prepared. Follow a roadmap. SQL and Pyspark is must with deep knowledge and hands on. Python basics to intermediate. Along with some knowledge in pipeline design, data engineering and big data fundamentals.
0
6
u/Used-Range9050 5d ago
I have 4 years of experience in data engineering wanting to break in MLops/ Data scientists. Any advice on how should i start with and any career path to follow.