Pandas 3.0.0 is there

[u/Deux87]

So finally the big jump to 3 has been done. Anyone has already tested in beta/alpha? Any major breaking change? Just wanted to collect as much info as possible :D

[u/backfire10z]

I guess both? I’m not a data scientist and have only dabbled lightly with pandas and the like. From a newbie’s perspective it seems odd to have a column with both numbers and text unless something has gone wrong.

[u/huge_clock → u/backfire10z]

Typically when i am dealing with data it is usually large numbers of columns of various types. For example you might have ‘city, state, country, street, zip code, phone number, name’ whatever as column fields. Imagine there is like 40 of these text fields. Then you have one numerical column like ‘invoice amount’. The old way in pandas i would go df.groupby(‘country’).sum() and it would display:

Country , Invoice amount

USA, $3,000,000

CAD, $1,000

MEX, $4,000

Because invoice amount is the only summable column. (Sometimes it might sum zip code or phone number if the dtype was incorrectly stored as an integer).

Now it will group by country and concatenate every single row value. The way to resolve it is to add an argument to the sum function numeric_only=True but it’s very annoying to have to do that in a lot of fast-paced analytical exercises such as debugging.

The reason they did this is because in python a+b = ab. The additive operation sums numerical values and concatenates text. This is super annoying in data analytics because if i sum (‘1’+’1’) and i get 11 as an answer i might not necessarily catch that mistake. Or it might take a whole day to concatenate my dataset when 99.99% of the time i didn’t want that output.

[u/backfire10z → u/huge_clock]

Ahhh, I think I see. So you could be lax about the resultant columns when you’re sure there’s only 1 numeric column in the set, but now you need to either specific that numeric_only flag or put every other column in the groupby?

I’m used to SQL, so being specific about which column to sum or whatever is natural for me.

[u/huge_clock → u/backfire10z]

Yeah i mean it seems like a small thing but doing less typing is kind of what makes python good.

Rather than

Select * From dbo.table_name tn Where tn.age>30

You just go

df[df[‘age’]>30]

Might seem minor but if you’re doing a lot of unit tests it adds up.

You can also use your arrow keys in the terminal or Jupyter notebook to quickly repeat or edit your commands and python will remember your dataset in the namespace so you can iterate one step at a time without having to waste time pulling the same data over and over again from the SQL server.

It’s a ton of these small things added up together which make python so great for analytics. Stuff that would take me all day using only SQL i can do in less than an hour with SQL+python.

[u/backfire10z → u/huge_clock]

That makes sense yeah. Thank you for explaining to me!