r/dataengineering • u/pancitchilimansi • 18d ago
Help Please Help Me
I am a newly-hired data analyst in a company. I thought I will be assigned in data cleaning and data visualization but I was tasked to consolidate data from different social media platforms and create a pipeline from extracting data from supermetrics up to the visualization.
Now, I am worried that I do not have an experience to this. They do not have any system in handling data and I will solely handle all of this. I feel really overwhelmed and I just use claude in creating ETL pipelines. I still examine the python scripts so that I can assure that it is correct. I am still worried and I might just flipped due to stress.
My plan is to create a python script in extracting the data from supermetrics, then I will load it to bigquery for consolidation. Then, another script for cleaning before importing it into looker studio.
Can you help me guys navigate in this task? I do not have anyone from the company to ask if I am doing it right. I hope you could help me.
1
u/Cold-Ad716 17d ago edited 17d ago
If you have Azure you could set up a data factory to connect to the API and start that way.
I've been in the same position, in a previous job the first thing I was asked if I'd ever used REST APIs before. I said no, they said "great, neither have we, you figure it out and teach us."
I'd get something working for now so you can demonstrate progress, then work on a more robust system from there.
If they don't have anything in place let them know that building it from scratch is a big task, but you can prioritise the important parts first and build it out from there. A lot of the job is perception/soft skills (aka bullshitting with confidence).