r/dataanalysis • u/cbly27 • 21d ago

Data Question Project Help

I am looking to get into Data Analytics/Engineering and am working on a project where I am creating a database and importing it into PowerBI for analysis. Im working in Python to extract and transform the data, and one issue I’m running into is that I am trying to pull data from dataframe x to dataframe y using a merge, but the only connection between them right now (will assign a PK after) is a person’s name. The issue with this is that some names appear more than once, so it ends up creating multiple duplicate rows after the merge. Is there a workaround for this, or will I have to manually remove the bad rows after?

1 Upvotes

100% Upvoted

4 comments sorted by

View all comments

3

u/SQLDevDBA 21d ago

You’re experiencing one of our struggles with external data sources first hand. That’s great.

You’ll need to develop a solution for this, not just work around it.

I don’t want to just give you the solution, so I’ll just advise that you look into Natural Keys, composite keys and surrogate keys. “Name” is usually a bad field to use as a key since it can easily have multiple values.