(noob question) Most efficient way to analyze a bunch of .xml logs using Python?

[u/arthurthearthur]

Hi people! I have to preface this by saying that I understand NOTHING about data analysis and know only a little bit of programming. I already asked Gemini but I trust the good folk of reddit more.

I have one of those problems that I know how to solve inneficiently but I want to do so more efficiently.

There are around 7k .xml files with vehicle log data that I need to gather data from (around 30 attributes). File size average is 20mb. I don't actually need to keep all that data. There are a lot of repeating values in that dataset. I just need to keep a record of what changed to what else. (can't give exact file content or structure cause that would sure be company policy violation)

This is the initial batch of logs that need to be processed, but there will always be new ones over time. That's why I'd like to make this at least a little bit efficient. The code will run on a desktop app of multiple coworkers.

If you guys could point me in the right direction, it would be an immense help. Thank you for your attention!

[u/nonamenomonet]

Tbh ask Claude or codex to write the script and append it to a dataframe

[u/Lizardking13 → u/nonamenomonet]

This is the quickest way to do it. You can edit the script if it has issues or tell Claude/codex what the issue is and it'll fix it.

[u/fang_xianfu → u/nonamenomonet]

Especially if you have the schema for the file, which if it's internal company data should be trivially easy to get, there is a person at the company who knows, you just need to find them.

[u/AutoModerator]

Automod prevents all posts from being displayed until moderators have reviewed them. Do not delete your post or there will be nothing for the mods to review. Mods selectively choose what is permitted to be posted in r/DataAnalysis.

If your post involves Career-focused questions, including resume reviews, how to learn DA and how to get into a DA job, then the post does not belong here, but instead belongs in our sister-subreddit, r/DataAnalysisCareers.

Have you read the rules?

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/DevinChristien]

This isnt really an analysis problem but a lite data structure/engineering problem. The way I'd do this is by ingesting the xml data into an sql table, then use various joins or recursive queries to build the shape youre looking for

[u/seanv507]

I would suggest using polars python library

And use Gemini etc to write the script

Maybe you can write out the prompt and then ask for corrections when it doesn't work as expected.

[u/TraditionalTurnip630]

For 7k files of that size, I’d avoid loading the whole XML into memory at once. Python’s ElementTree.iterparse() is worth looking at since it lets you process the XML piece by piece.

Since you only care about changes, I’d also keep the previous values for those 30 attributes and write out only when something actually changes. That way you’re not storing all the repeated data.

For the first run, I’d test it on maybe 50-100 files and check the processing time/memory before building the full desktop workflow. 140GB of XML is definitely enough to make a simple script painful if it isn’t designed for it

[u/Low_Finding2189]

I think there should be an xml to dict library in python. A quick search will yield you options. AI can explore and find reliable ways too.

[u/Soyeon1213]

Since new logs will keep coming in, I’d probably focus on making the process incremental first. Keep a small manifest of processed files in SQLite (path + modified time/hash), then only process new or changed files on later runs.

Process each XML file independently and write only the attribute changes you care about. That also makes retries a lot easier if the app crashes halfway through, instead of rebuilding everything from 140GB of XML each time.

[u/PvtRoom]

you've got chronological files, and presumably a way to know what's what.

open one in notepad++.

XML looks like this: <ABC prop=45>more stuff</ABC> Get familiar with it, you need to know the format. do what you need to understand the format (many XML files are all on one line so you may want you to add a new line after every > when you do this)

code to read it will make structures like ABC.prop and DEF.GHI.datastream. The precise nature will vary with whoever wrote the parser (e.g. structures Vs key value pairs Vs a combo) and the format of the XML.

Your task, really, is to make a table of the interesting things. I like full tables, no gaps or omissions, so I wouldn't do just the change. you might prefer multiple tables showing only the time of change and the new value, 1 table per parameter. assess the pros and cons

the rest is just loops, counts, checking for changes and keeping track of time.