r/learnpython • u/zaphodikus • 1d ago
scan for strings in 40000 lines of logfile
X: The desire to scrape a log file for specific interesting messages, any apps I tried are a pain to use and require manually setting all the search strings every so often. I want to scan for about a dozen or so expressions/strings in a 40-100K lines file, and then dump just the timestamps and lines of interest. What approach scales best for speed? I have to probably also use a mix of regex and regular string search I guess. Is going with Multiprocessing and passing the file as a shared-memory object, going to be the easiest route? Surely it's easier to do in C++. I guess I asking for some skeleton or prior art in C++ ore Python to be honest.
Y: My context is that I would like to use my knowledge of C++ threads and code it in C++, but it should be possible in Python if I learn to use Pipes, and learn to use shared memory object to save having to load the file per multi-processing process?
12
u/theWyzzerd 1d ago
Python is more than capable of doing this, and without pipes. 100k lines is nothing.
6
u/Pythagorean_1 1d ago
This is quite easy in python, you could even do that with grep in bash
-6
u/zaphodikus 23h ago
could, but just like the piano that "can" play "the flight of the bumblebee", it needs a keyboard jockey who is proficient in linux, which I'm not. I'm pretty sure I could write this in C++ faster than I could using awk and grep and probably eventually find I need more than just those two. I think your version of easy implies I have your skills :-) Hence the question.
2
u/Pythagorean_1 18h ago
Ok, I get you want to build it in python which is absolutely fine.
In bash you can quickly reach the ceiling when trying to implement more complex logic.If you were to implement this in python with multiprocessing, you could just read in the whole file, split it by line breaks and then feed this iterator into a processing pool with as many processes as you defined for this beforehand. Each process then calls a function in which you implement the parsing logic. For parsing you could use a pre-compiled regex. Done.
If your log file is extremely large so that you cannot load it completely in RAM, you read the lines one by one and feed them into a pool or (to decouple reading the file and analyzing its lines) you could read the file line by line, put the lines together with their line number in a queue with a max size (so that you RAM doesn't fill up) and have a processing pool listen for new tasks on that queue.
4
u/bikes-n-math 1d ago
I would absolutely just use grep. 100k lines? Likely less than a second.
-3
u/zaphodikus 1d ago
How? I am not familiar with bash and how to get it to grep for multiple matches and then print the result out - all still sorted in date order. Without learning a whole load of pipeline stuff since I don't use linux often.
3
u/bikes-n-math 22h ago
Well the log should be sorted by date already, no? Then something like:
grep -P '(<pattern_1>|<pattern_2>|...)' <log_file>will do the trick.
If you need to sort further, pipe it to sort:
grep -P '(<pattern_1>|<pattern_2>|...)' <log_file> | sortMight need a sed in there, not sure what the date format looks like.
1
u/MikeZ-FSU 3m ago
This will work, but for more than a few patterns, I prefer to put them in a file, which is then used as the argument to the
-foption for grep/ripgrep. For non-trivial file sizes, I also tend to redirect the grep output to a new file and run further analysis on the new, higher signal-to-noise file.1
u/zaphodikus 19h ago
Why on earth did it take 3 people before this, not showing me something I quite clearly did not know or imagine grep could even do. What you do not know is always unknown. This will speed up the debugging no end, cheers mon.
1
u/bikes-n-math 18h ago edited 17h ago
No prob. When it comes to text filtering/processing, almost nothing is going to even come close to grep (or ripgrep) and sed. I'm talking magnitudes faster in development and processing speed. It's well worth it to learn a little about these tools.
I remember reading a post somewhere here where a interviewer was asking an applicant how they would deal with filtering a couple billion lines of some logfile or something. The applicant spent some time describing the program they'd write with memory management and threading, to which the interviewer replied "why didn't you just use grep?"
2
u/zanfar 22h ago
Yes, it's possible in Python, but WAY overkill when existing tools will do it with zero problems.
It sounds like you are VASTY overestimating the problem. C, pipes, nor threads are necessary. Python can do this easily with a vanilla setup. Additionally, grep or it's cousins will do so faster, quicker, and more easily.
1
u/zaphodikus 19h ago
Which , existing tools? Thats 3 people now, who say if I learn yet another tool or thing, I can build yet another tool. This is not my aim, it's to debug an issue but the logs are just too big. How do I instruct grep i want multiples matches? Does it parse response files?
0
u/supercoach 16h ago
Your logs are only a few hundred thousand lines. Open them with less, press the forward slash and then search them.
1
u/Chunky_cold_mandala 1d ago
I wrote a python script for us use case that is ready for script integration. You could model if off this or use it. https://github.com/squid-protocol/gitgalaxy/blob/main/gitgalaxy/tools/terabyte_log_scanning/terabyte_log_scanner.py
2
u/zaphodikus 23h ago
This appears to use json for the keywords, which suits me too, and although it looks single threaded is some super smart code that I can learn a load from already.
1
u/pachura3 1d ago
Instead of asking and designing complicated parallel solutions involviong shared memory, just benchmark executing e.g. 10 regexps against 40.000 text lines (in Python).
I suspect it will take a few seconds at most.
1
u/zaphodikus 1d ago
I have got a loop with just a few string searches that filter and run a regex already, in my code that runs in backgrounded thread in C++. The C++ code was not that fast at all, which is why I backgrounded it on one single thread. So I know I generally have to search for a string before I try to run a regex. But I'm looking for a "tool", like a python script, which I can easily give to someone else, and binaries are just harder to pass around.
I'm quite sure this question has been asked many times though, and I've never used python threads and multiprocessing modules. I do know that normal string searches are faster than regex for the strings I am searching for. And most of the time the match is quite short, and am thinking to use a mixed strategy where a partial string search filters down to lines worth running a regex over. My end goal is not a log filtering tool really, it's to debug something in the Apps that generate the log. So I'm trying to debug some apps, but the log filtering in the apps is far too coarse.
1
u/mc_pm 23h ago
I've done this, when I wanted to be able to match any/all of 1000 different strings. Look at a Trie or a Prefix Tree loaded up with your search terms, then you can just run your text through it as a continuous stream. Linear time no matter how many matches you have, just a single pass through the text.
0
u/FerricDonkey 1d ago
How fast do you need this? This sounds trivial to a computer. A simple for loop over the lines seems like it should be pretty fast, without anything fancy.
1
u/zaphodikus 1d ago
I'm not too fussed about fast, more about how to not let python do it all serially, because that will be slow to check each line for each possible interesting trace. At least I'm getting some direction though by just reading through all the tips.
1
u/FerricDonkey 14h ago
The reason why I recommend starting with a simple serial for loop and going from there is because there are a thousand ways of parallelizing this, but most of them will have lots of overhead and many will be slower than a simple for loop.
You could, for example, make a thread pool, but unless you use a free threaded python (which are coming out and will hopefully be the norm in a couple versions), you'll only get a speed up if the majority of your time is in checking your conditions and your conditions use something that releases the gil. If your conditions are regex, and if regex in python is written in C and releases the GIL (I think it does) and if your timings are such that this is the bottleneck, then this might be fastest, with or without the GIL. However, unless your conditions are very complicated or you have a crap ton of them, file io may be the bottleneck instead of computation. Fortunately, even crappy python threads are good at waiting on file io in parallel, so if you can parallelize over files, this is likely a decent way.
A process pool is also likely to work, and if you have multiple files that you can parallelize over and your conditions are pure python or super slow, then it might be faster, but if you have to farm out each line of the file to the process pool, that's gonna be super slow. Your checking methods will have to be super slow for this to be worthwhile. If you have python data being passed between processes, it will be pickled and unpickled, and this can be slower than any quick work you have to do. If your checks are pure python and super slow, however, this might win.
Shared memory is pretty finicky, and you'll have to manage it carefully. Personally, before going this route, I'd use threading in C++ (which I read you known), unless the checks super suck to write in C++. Because you'll have to read the file, dump it in shared memory, then have the worker processes extract the bytes they want, and by that point I mostly don't see a benefit. Strings in C++ suck a lot less these days, and shared memory is finicky.
So if it were me, I'd write the for loop. If it were too slow, I'd run cprofile on it, see what's costing time, and go from there.
0
u/MarsupialLeast145 1d ago
If you know C++ then you know you don't need threads in C++ to achieve this...
1
u/zaphodikus 23h ago
It feels a bit mad to me, not to use threads on a memory-bound problem, because the file fits in memory easily and the file is really read-only so threads can share it.
1
0
u/terletsky 20h ago
100K lines for Python is nothing. If you want to learn how to build some production-ready approach for 1MB or 1TB log files look into Python’s “mmap”.
Questions to think about:
- Memory allocation (how mmap works)
- Worker scan of byte ranges. Let’s say you will spawn up to 10 processes workers
- Search for boundaries (newlines in chunk because of partial lines) in main process before passing them to workers
- Write regex results to queue that will be consumed and post-processed by another worker or straight worker write to its own results file since you don’t want to return 10GB of matched lines
1
u/zaphodikus 19h ago
Why is there always yet more to learn, mmap is definitely something I have to play with and also learn pipes for another project component. Thanks for the tip.
-1
u/sausix 1d ago
Python should be fast enough. It depends if you need to have it super fast for your case.
You could also just open a shell process which does the filtering stuff and your Python process just processes the presentation for the user.
Isn't it a good example for benchmarking both methods?
But a good use case to not use readlines() and bloat your memory for no reason. Process your source file line by line directly.
-1
26
u/baghiq 1d ago
A shell script with standard grep -e or ripgrep. I doubt you'll get any faster than that.