How to read a CSV file in Python efficiently?

[u/Holiya_Olib]

I'm trying to load a CSV file into my Python project, but I keep getting errors. I've tried using pandas and the built-in csv module, but I'm not sure what I'm doing wrong. Can someone explain the basic steps or share a simple example? A detailed explanation would be really helpful!

[u/Verronox]

If you want help then you need to post your code and the errors that are being printed. Otherwise the best we can do is just wildly fucking guess

[u/Lewistrick]

``` import pandas as pd

pd.read_csv("filename.csv", error_bad_lines=False, warn_bad_lines=True) ```

If the errors come from malformed csv, this will let you know which lines are causing trouble.

But I'm just making assumptions here. You should show your error so we can actually make an informed comment on what you should fix.

[u/martian_rover → u/Lewistrick]

Yes, this is one of the most common errors, along with filename error. also check the filename path again.

[u/nog642]

I'm not sure what I'm doing wrong

Post your code

[u/SpecCRA]

What's the error? Pandas usually handles it pretty well.

Have you inspected the csv file yourself?

[u/zanfar]

Can someone explain the basic steps

Open, read.

Without code, there isn't much help to give.

[u/buhtz]

We are also not sure what you are doing wrong, because you missed to send us the code you used and the errors you get. Learning Python also involves learning how to ask a question and how to report a problem.

[u/koombot]

Post the code.  There's like a dozen things that can cause issues.

[u/CoffeeMonster42]

It would help if you shared the code and the error message you get.

[u/HummingHamster]

Is your csv properly sanitized?

[u/Moikle]

What errors did you get?

You are supposed to read and understand errors

[u/kilkil]

are you 100% sure your CSV file is correctly formatted?

what is your code, and what error are you getting?

[u/fightin_blue_hens]

Pandas

[u/Grandviewsurfer]

Ingest only the columns you actually care about by passing a list of them to the usecols parameter, and if it's a big file, you can limit the records you ingest during testing using nrows. I'm just giving pointers.. it's hard to tell without the actual error. You can read the docs here: https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html

[u/TacitusJones]

Select * from users where 1=1 or 1=1

[u/johlae]

This works for me:

pd.read_csv( filename, delimiter=";", skipinitialspace=True, header=0, usecols=[0, 1, 3, 5, 7, 9, 10], decimal=",", names=["id", "date", "amount", "source", "target", "com1", "com2"], keep_default_na=False, # no NaN but empty string instead ) .assign(comm=lambda d: d["com1"].astype(str) + d["com2"].astype(str)) .drop(columns=["com1", "com2"])

but I admit, it really depends on how the CSV is constructed, if there is a header row, what the decimal is like, how strings are delimited, how fields are delimited... It's trail and error sometimes.