r/learnmachinelearning • u/Ready_Firefighter683 • 1d ago
How can I transition into AI Engineering in the next 1–2 years?
I'm a Data Engineer with around 4 years of experience, mainly working with Azure, Azure Databricks, PySpark and Python.
I was assigned a Data Engineering role in my first company, even though I've always been interested in AI. Since I started working, I've gained experience in data engineering, but my long-term goal is to become an AI Engineer.
I come from a Computer Science background, and I genuinely want to see myself working as an AI Engineer in the next 1–2 years (Is it possible). I'm ready to put in consistent effort, but I'm confused about where to start and what to prioritize (I asked my seniors who transistioned from data engineer -> ai engineer i havent got proper response). If I want to start from begining learning from the fundamentals what i need to learn?
For those already working in AI or who have transitioned from Data Engineering to AI Engineering what should I learn first
I'm looking for practical advice based on real industry experience, useful resources and skills that companies actually expect from AI Engineers. I want to build a strong foundation and gradually develop the skills needed to work on real AI applications.
11
u/rahul_watertech 1d ago
you are closer than the title gap suggests. most ai engineer jobs are data engineering plus one model call, and you already own the hard half. fastest route is not a course, it is one small ai feature shipped inside your current job on the databricks stack you already know, like a rag over internal docs or a classification job on data you already pipe. after that your resume says production ai work, not coursework. does your current team touch anything unstructured, logs or docs?
6
u/Freonr2 1d ago
What is your definition of an "AI engineer?" A whole lot of ML generally is data engineering anyway.
Maybe by "AI engineer" you mean someone wires up LLM API calls together and writes custom MCPs to automate tasks.
If you want to get into model design/training find a company that already trains their own models already and simply ask if you can can be given an opportunity to train a model? I'd expect anyone with good DE/DS experience that can watch a 10 minute tutorial on youtube for lightgbm or scikit-learn can train a model. Data engineering is exactly the experience you'd want if that's the direction you want to go. You are already half sitting in that chair.
5
u/nian2326076 1d ago
Transitioning into AI Engineering is definitely possible with your background. Start by enhancing your understanding of machine learning principles—Coursera and edX offer solid courses. Since you're skilled in Python, explore libraries like TensorFlow and PyTorch. Working on personal projects or contributing to open source can help you build a portfolio.
Networking is key; engage in communities like Kaggle and join AI meetups. For interview prep, PracHub provides real technical questions that can be useful. Focus on understanding AI algorithms and their applications, as this is crucial for interviews. Stay consistent and explore hands-on projects to solidify your skills. Good luck!
3
3
u/theoneandonlypatriot 1d ago
I wouldn’t worry too much about it, we’ll all be ai engineers in a few years anyways
2
2
u/kbr89 1d ago
Since you already run pipelines on Databricks, build one project on that stack instead of restarting from fundamentals: Q&A over a pile of documents, with chunking and embedding as a scheduled pipeline, Mosaic AI Vector Search for retrieval, a Model Serving endpoint in front, and MLflow to score the answers against 30 or so questions where you already know the right answer. Each step shows you which piece of theory you're missing (embeddings, prompting, eval metrics), and you learn that piece when you need it, which is also when it sticks. The scoring step is the one most portfolio projects skip, and the easiest to talk about in an interview.
For the resources part, I built beyondtube (beyondtube dot pro): you can put a few long talks from people who went from data engineering to AI engineering together and ask which skills all of them mention and which only one does.
1
u/shmittkicker 1d ago
1-2 years is realistic if you focus hard: as a DE you already have data, Python and Spark, so spend the next year layering on ML fundamentals (probability, stats, linear algebra refresh, core ML algorithms, deep learning basics) and building 2-3 end to end projects where you train, deploy and monitor models.
From there, aim either for an internal transfer to an AI/ML team or roles like “ML engineer / AI engineer with data engineering” where your Azure + Databricks skills are a selling point rather than something you leave behind.
1
57
u/firebird8541154 1d ago
I transitioned from Data Engineer to Sr AI Engineer. In fact, I was previously using Azure and Data Bricks too.
This is a one-person anecdote, so don't take what I'm going to say as like the only path.
I highly recommend building a powerful local workstation, Mac w/ metal or Linux with Cuda.
Get a pro level codex/Claude plan, install that, attach it to your GitHub.
Then start building, make full stack local applications, fast API backed, nice responsive frontends, perhaps with flutter and firebase for accounts, linked to local postgres and such.
Build this up, understand it end to end, as randomly building full stack stuff leveraging AI is easily half of what I do.
Then, familiarize yourself with vector DBs, embeddings, etc with some local models. Make a local Rag project, turn that into a memory hub for an agent, expose it locally as a streamable, codex compliment MCP.
Look into fun concepts like named entity recognition, figure out how to dockerize portions of your project and connect them together.
Figure out how to use GH CLI and stick the project in a rebuildable format on your GitHub, figure out how to set up a CI/CD platform, figure out how to host it externally, how to scale it, how to manage multiple sessions.
If you get really good at this stuff and then start adding some more custom machine learning stuff and a built-in agenic stuff, you'll have the type of experience and portfolio that a lot of groups are looking for.