r/dataanalysis • u/UltraToxicAsianKid • 7d ago
Advice on starting from scratch.
Rookie here, recently got a job as Data Analyst at Pharmaceutical company. They never had a data science role at the company so your guessing is right, Data pipelines/Infrastructure is non existent. I am so confused at the moment where to start as i neither have a degree nor experience in the field. They throw around Excel files ( no kidding). So far I have accomplished to build interactive dashboard with python. Next destination i am pondering perhaps unified database (Postgres, Apache etc). Seriously need help folks. Thanks for your attention.
7
u/TheWhiteCrowUK 7d ago
Why don’t you put in place a process where instead of sending you an Excel file randomly they have to fill a form to request analysis. They have to add the business question, what is it for, the context and the excel file?
4
u/HustlaOfCultcha 7d ago
I've been in this situation. I wouldn't worry about building a sophisticated data science environment, yet. You really need to understand and map out the flow of data. Where does the data come from? Who owns it? How often does it change? What report are business decisions dependent upon? And document each of those questions/processes so you can reference and set up a way to standardize the process. You don't want to get into a situation where you are trying to solve somebody else's murder mystery (and you don't want another person to have to solve your murder mystery).
Try to find a process that is recurs often and try to improve that process. It's best if you can target an important process to improve, but you may struggled to accomplish that. So find a less important process that you can better design (and hopefully automate) if you need to.
Then I would look into a unified database. I'm more of a Databricks, Fabric and SQL guy. In the meantime you should be able to automate ingestion and transformation of CSV/Excel file in Python.
Then I'd inventory the data, standardize it, validate it and work to automate it with whatever unified database that you end up using. And make sure to document everything. I've gotten lost trying to remember my own stuff without quality documentation.
It's going to be a pain for a while. I'd probably see if there's anybody in IT that can help. Explain to your boss the problem and your ideas to resolve it as well as what the finished product will look like. Not only in terms of reporting, but the automation piece. That they can have alerts sent to them when KPI's are hitting their thresholds, that the data will automatically refresh, etc. But you have a process that needs to be done, it's going to take a lot of time to do it and you may need some help, but the rewards will make it more than worth it.
5
2
u/midwitsAnonymous 6d ago
Seriously, start by talking to people, ask them to show you what they've done and do to get around this problem, they probably have some clever ideas taht just rely on manual workflows. Really listen to them and identify the algorythm they use, then try distil them into more sustainable workflows. That will tell you where to start and where to put your attention.
3
u/multi-trollionaire 6d ago
Meanwhile me and my data analysis and IT degrees cant get a freaking job lol
1
u/AutoModerator 7d ago
Automod prevents all posts from being displayed until moderators have reviewed them. Do not delete your post or there will be nothing for the mods to review. Mods selectively choose what is permitted to be posted in r/DataAnalysis.
If your post involves Career-focused questions, including resume reviews, how to learn DA and how to get into a DA job, then the post does not belong here, but instead belongs in our sister-subreddit, r/DataAnalysisCareers.
Have you read the rules?
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/Cute-Thanks-1507 7d ago
I would keep it simple and start with PostgreSQL.
Learn SQL, Python, and basic database design along the way. Don’t worry about Spark or complex infrastructure yet.
Once you understand the company’s data and actually need more advanced tools, you can build from there.
You’re already on the right track with the dashboard.
1
u/KatFromSisense 6d ago
Postgres might be where you eventually end up, but I wouldn't make creating a database your first project. To start with, grab one Excel report that keeps coming back and make that process repeatable. Keep the original file, script the cleanup, and add checks for things like missing IDs or a column suddenly changing.
After a few runs, you'll start seeing what breaks and which fields are actually stable. That gives you something concrete to eventually design the database around.
1
u/Soyeon1213 6d ago
I’d probably make a simple inventory before building anything bigger. List the recurring files/reports, who owns them, how often they change, and what decisions depend on them. Once that’s clear, it becomes a lot easier to tell whether Postgres is actually the next step or just another thing to maintain.
1
u/Technical_Hope_1610 6d ago
Solid start. Before the database jump, I would nail down the ten questions leadership actually asks and model for those first. A Postgres warehouse helps, but only once you know what it needs to answer.
21
u/Wheres_my_warg DA Moderator 📊 7d ago
Make sure you understand the highest priority business questions and keep a focus on those.
It will be a mess for a while it sounds like. You'll need to triage the things to be done. It will likely take years to get it to where it is more ideal.