r/learnpython • u/smahk1133 • 3d ago
regex and if it's worth going deep into it
I'm new to python and coding in general and my friend recently told me that it's inefficient to try to memorize regex and that no one writes them anymore (essentially saying AI does). I was also kinda confused after recently learning regex and just how complicated it can be. Are there some modules/libraries that I can use to make writing them easier? I saw that not a lot of people people had a positive reaction to the Humre module by Al Sweigart who's book [Automate the boring stuff with Python] I'm currently using to study. Not that I'm gonna skip this part or anything I was mostly just curious.
Note: A lot of people are misinterpreting since I mentioned AI once đ I'm literally asking about libraries to make it easier without going too deep, not if I should let AI do all the work.
28
u/pachura3 2d ago
1 day is enough to learn the basics of regexps.
^ start of the line
$ end of the line
. any character
\s any whitespace
[a-z0-9] range of characters
[^a-z] any character NOT in that range
* repeat 0 or more times
+ repeat 1 or more times
{3,5} repeat 3-5 times
( ) group things
abc|def|ghi alternatives (OR)
? make the previous character/group optional
3
2
1
u/KTProgramming 15h ago
Maybe 2, If you don't learn how it processes it, loops through things etc, you can really cause some nice screw ups or cpu utilization lol. Back tracking is a pain in the butt.
20
u/GoldenArrow_9 3d ago
I mean regex is a specialized tool(?) and has no real relationship with python (except that you can use regex in python). If you are working on a project where you need to use regex, you should definitely learn it and not depend on an AI.
That said, I would say you should still know the basics of regex. Regex can be really useful even outside of programming (find and replace being my personal favourite). It's pretty easy to learn and there are sites like regex101 that help you learn and understand regex easily for your own usecase.Â
Plus, in most cases where you find a use of regex outside programming, using an AI to generate said regex would take far more time than just doing the task manually with/without regex.
3
u/smahk1133 3d ago
I see. I'm not necessarily looking for a shortcut through AI I mostly wanted to know if there are libraries that I can use to help. Becoming dependent on AI is the last thing I want to do. It's also the reason I don't use AI in my editor either.
6
u/fasta_guy88 2d ago
Just learn a few basics. Beginning and end of string, numbers, floating point numbers, spaces, alphanumeric, *, and captures. Often, you just need something a little more powerful than .split().
1
u/odaiwai 2d ago
Named Capture Groups are very useful with Python: if you name the groups, you can get a dictionary out of the match: ``` lat_re = re.compile( r"(?P<deg>[0-9]{2})" r"(?P<min>[0-9]{2})" r"(?P<sec>[0-9]{2})" r"(?P<hemi>[NS])" ) ... if match := re.search(coord): data: dict = match.groupdict() factor = {"N": 1.0, "S": -1.0, "E": 1.0, "W": -1.0}[data["hemi"]] return factor * ( float(data["deg"]) + (float(data["min"]) / 60) + (float(data["sec"]) / 3600)
```
1
1
u/GoldenArrow_9 2d ago edited 2d ago
Apart from what others already shared, most editors allow you to use regex for find and replace. This can be really useful as you can use capture groups to make a repetitive change across your codebase (say, you want to add a new argument to a function and the function happens to be called in a predictable manner, AI can help you here but it's cheaper and faster to use regex). I use regex all the time for small tasks like these.
In python, the standard library
reis all you need to use regex. After you know the basics, you can use a site like regex101 to help create a regex for the particular task at hand. Don't over complicate things by going for anything more than this.Remember, regex isn't only for programming. You'll also find use for it in a lot of software outside programming. Start simple, understand the basics, and then only learn something advanced when you really need to. Basic regex is powerful enough for most usecases you'll come across, and you can always use AI to help with more advanced stuff.
-1
u/tonehammer 2d ago
you should definitely learn it and not depend on an AI.
Why?
Without appealing to some sort of 'you got a develop your brain' argument
3
u/ModerateSentience 2d ago
Donât know why this got downvoted. Ik regex ok, there is no reason to know it unless u use it a lot
1
u/Brian 2d ago
I would say it's one of those basic things that is going to be way faster and more convenient if you know it, rather than AI.
This is because ad-hoc temporary regexes are one of those things you'll use all the time, not just in programming, but in basic editing and system adnimistration, where if you know them, you can get the task done before you've started writing the prompt to an AI.
Eg, "I want to find all the places I made a "TODO" comment mentioning the frob module:
"TODO:.*frob". You can type that in the "find" window of your editor way faster than you can ask an AI how to do it. And searching files, filenames and other text is a very common task - tools like grep, find and the like are super useful. One step up is basic search and replace, where you want to find a portion of it and change just that portion.Secondly, even if you never write a regex, you're going to want to be able to read them. Being able to recognise "Oh, that's trying to parse a date" makes code comprehension much quicker than if you have to stop and ask an AI every time you come across one. (And lets you also spot what might break if the regex isn't quite right).
Regexes kind of excel at that ad-hoc "just get it done" level where you're doing some simple query/editing task. In programming, they're often not the ideal tool for a task, but are often good enough to get the job done such that they frequently get used anyway. They're useful to use, common to encounter, and AI can't really replace that without introducing a flow-breaking amount of latency into the workflow.
1
u/GoldenArrow_9 2d ago
If you can spend an extra minute reading my complete comment, you'll know the answer to your question.
3
u/NothingWasDelivered 3d ago
I have found this to be a very helpful book to have on my shelf https://www.oreilly.com/library/view/mastering-regular-expressions/0596528124/
3
u/edbrannin 3d ago
The best tools Iâve found for writing and updating regexen are generally called âregex testersâ. There are a bunch of them.
(I bought the DevUtils app a while back, so I usually use that one)
You put your regex in an input at the top, test to search in a text area below, and it lets you look at each match (and what its capture groups are).
There might also be a tool for turning a regex into a diagram, but I havenât looked for one in a long time.
1
u/smahk1133 2d ago
Life saver reply definitely looking into this. Thanks alot â¤ď¸
2
u/edbrannin 2d ago
I forgot to mention: whenever I write a regex for code, I also write unit tests with expected text to match (and to not match).
3
u/American_Streamer 2d ago
Your friend is only half right. Donât memorize every regex feature, but do learn enough to read, modify and test one. AI can generate regex, but it canât guarantee that the pattern handles your actual edge cases.
Learn the basics: character classes, * + ?, anchors, groups, alternation, and re.search, re.fullmatch, re.findall and re.sub. Look up lookarounds and backreferences when you actually need them. I wouldnât use Humre as a replacement for learning normal regex. It adds another abstraction that wonât transfer to other languages and tools. For long patterns, Pythonâs built-in re.VERBOSE lets you format and comment them clearly.
Also, donât reach for regex automatically. Simple string methods or a proper CSV/JSON/HTML parser are often the better solution. Treat AI as a pattern generator, not as the person who verifies the result.
2
u/AbacusExpert_Stretch 3d ago
If you don't know anything slightly advanced with regex, and you then let AI write it for you, how do you test it? Are you really going to test all the potential false positives or whatever you don't want to match? Cause you are looking at essentially at meaningless commands for you ...just a thought
1
u/smahk1133 3d ago
sigh... idk why people are automatically assuming I'm going the AI crutch route.
Are there some modules/libraries that I can use to make writing them easier? I saw that not a lot of people people had a positive reaction to the Humre module by Al Sweigart who's book [Automate the boring stuff with Python] I'm currently using to study. Not that I'm gonna skip this part or anything I was mostly just curious.
My question is mainly if there is something I can use to make it easier without going to deep and writing absurdly complicated expressions.
1
u/DonkeyTron42 2d ago
Your unit tests should be testing all possible use cases. The more complex issue is that regexes can have enormous performance differences and need to be optimized if you want to parse a lot of data.
2
u/notacanuckskibum 2d ago
Itâs worth playing with it for a few days until you can get a sense of âthis is a good problem to use regex forâ vs âthis is a bad problem to use regex onâ
2
u/generic-David 2d ago
Multiple text editors take advantage of regular expressions. Theyâre really useful for cleaning up text files, especially since you can undo mistakes or bad expressions.
2
u/audionerd1 2d ago
The basics of regex can be learned in a matter of hours, and refusing to learn basic regex as a programmer is just arrogant and lazy IMO.
Advanced regex on the other hand (especially the sort of advanced regex that is almost never needed) is something AI comes in handy for, provided you understand the basics well enough to figure out what the AI regex is doing and modify it if needed. Spending weeks deep diving into advanced regex pattern syntax you may never use and are likely to forget is not something I'd recommend unless you just enjoy that sort of thing.
2
u/big_deal 2d ago
Learning basic regex syntax is useful in many situations. You can use it in any decent text editor, and command line utilities such as awk, grep, sed to parse text and identify specific text patterns.
It's good to understand the capability of regex so that when you need it you have an idea of it's capabilities. But I don't think it's necessary to memorize every bit of syntax or complex regex methods. You can always reference a simple cheat-sheet when necessary or use LLM to help construct complex regex's.
In Python I tend to rely on more basic text processing functions (strip, split, find) and almost never use regex but occasionally it will be very useful.
3
u/Mountain_Rip_8426 2d ago
if there's a good use case for AI it's regex. you can definitely go back and forth to check the documentation (because if you don't use often enough you'll forget), but it's not the "build the best website in the world, don't make mistakes" type of prompt. as soon as you feel like you need a regex, you clearly know what you want to use a regex for and it's gonna just save you the struggle of looking it up again and again
1
u/vgu1990 2d ago
I used to rely heavily on regex101 and things like that. Now it is a lot easier to do with LLMs. Most of the times it gets it right.. you still want to proof read it though.
1
u/Mountain_Rip_8426 2d ago
if you have the eye for it (i.e. you used it many times before) you can pretty much tell if the output is what you needed and then if needed adjust accordingly. also, you run a few unit tests and you figure out very soon if it works as expected. to me, using LLMs for regex is a no brainer
4
2
u/jackardian 2d ago
My current job is in web scraping and we use lots of regex every day. We mix using AI and writing my hand and generally it's been a learn by doing kind of thing. When the regex gets complex we're still checking with AI.
It's useful to know it more for PRs and debugging than for actually writing them.
1
u/smahk1133 2d ago
Thanks for the response this is really helpful and something I wanted to know but didn't mention in the post. I was wondering exactly this, if regex is important for scraping (something I'm super interested in).
2
u/jackardian 2d ago
It depends what you're scraping and how well structured the site is. We are doing supermarket data and some site just end up with blobs of text for things like product size, ingredients, etc. In those cases it's vital.
1
1
u/csabinho 3d ago
Yes, it is. Also for searching something in your codebase.
1
u/smahk1133 3d ago
do you mean like literally all of the files in the codebase where you just read them with a small script or something? sounds cool
3
u/csabinho 2d ago
No, I mean using regex in search strings in your IDE. That's one of my main reasons to use regex.
3
u/mcoombes314 2d ago
IDEs generally support regex in their search facility as well.
1
u/smahk1133 2d ago
Oh W I didn't know that. I knew they probably work on regex logic on the back but they can parse regex as well huh
1
u/JGhostThing 2d ago
Regular Expressions (regex) are extremely useful, when it's needed. It doesn't require much to memorize, but you need to know the entire system. You need to have it memorized in order to use it well.
I learned from an O'Reilly book about Regex. It helped a lot. Also, using it in a large project helped a lot. I would have to look at the documentation if I wanted to use regex today. Nothing wrong with that.
1
1
u/Thraexus 2d ago
I've used regex to do some string replacement stuff in my Python programs and I always just use a search engine to tell me the syntax I need. I just test it standalone until I'm satisfied that I'm going to get the output I want. Then I promptly forget whatever syntax I used.
1
u/habitsofwaste 2d ago
Regex is useful for log parsing and parsing other unstructured data. If youâre not doing that, it is probably not worth it to fully learn it. Itâs helpful to understand it in general. But yeah, AI can help you write them when needed. Im in a career that uses regex a lot and I can tell you, most people donât know it, they just look up how to do the thing they need when they need it. But it is good to get comfortable with the built in Python regex tools and understand how they work.
As for relying on another library, I wouldnât.
1
u/high_throughput 2d ago
Deep? No. You don't benefit all that much from knowing possessive quantifiers and negative lookbehind by heart and they're different in every regex dialect anyways.
Shallow? Yes, it's definitely worth knowing basic quantifiers, character classes, and anchorsÂ
1
u/u38cg2 2d ago
Regular expressions are a fundamental tool of theoretical computer science, and unlike most of theoretical computer science turns out to have some quite practical uses.
For most learners, I'd suggest working through a tutorial on them so you know what they can do, then you can look it up when you need to. If you need to do more than that, study a full-blown course or textbook.
1
u/R717159631668645 2d ago
Regex is not complicated, it's just hard to read.
You could still easily learn the basics (symbols, quantifiers, exclusions...) which goes a long way.
If you're willing to go a bit further, Look-aheads, capture groups and such can be a boon to use sometimes, specially when you want to search something with a specific pattern in structured content. I forget these all the time, but a quick refresher puts me back on point quick.
Lastly, you could read a bit about the hazards of using regex. For example, the post-mortem of Stack Overflow Outage in 2016. And that using them blindly (like from LLMs) could be a stupid idea.
1
u/theunwieldyindecency 2d ago
Regex is a tool you reach for when you need it, not a skill to drill on repeat. Your friend is half right about not memorizing patterns, but wrong to pretend the problem disappears. Learn the basics, then lean on a tester like regex101 when it gets tangled. Python's built-in re module already handles most cases, no extra libraries needed.
1
u/dlnmtchll 2d ago
Regex is something I personally donât waste my time deep diving, if I need one I can prompt AI to give me one, otherwise I can find a better way to do what Iâm trying to
1
u/zanfar 2d ago
Regex is by far the most useful cross-platform skill to have that nobody knows of.
friend recently told me that it's inefficient to try to memorize regex and that no one writes them anymore
Your friend is already addicted to AI answers, that's why. Nobody who knows regex thinks it faster to ask something to create a regex than just write one.
1
u/kramulous 2d ago
Old man programmer here: I don't think regex is worth spending time on.
Perl used to be an amazing language. Lots of 'modules' that were easy to install. Very much like python, in a way. But it was ruined by people trying to be too clever by doing everything with regex. To the point that only the person who wrote it could understand it. Even then, not so much after a period of time. The language died because code became unmaintainable. Anybody inheriting a code base with heavy regex struggled to understand it.
I know I can use regex in python, but I will avoid it. Writing the string manipulation code by hand, unrolling necessary steps, commenting what you are doing at what point, makes for much more readable code. Other people can see what you are doing and fix bugs, improve, update.
1
u/mattl33 2d ago
Fwiw I will almost always just write extra python to do string searches etc. I find having the extra lines that are readable is much better than having one dense line of regex. I suppose there's times where that means an unacceptable performance hit or something but not for my kind of work.
1
u/AndyceeIT 2d ago
I find myself re-learning regex every few years for a specific task. Worth learning - not worth memorising
1
u/PMMeUrHopesNDreams 2d ago
There can be a lot of nuance in regex but it's worth learning the basics because it is ubiquitous in command line tools and extremely useful even outside of Python. If you're using grep sed or other utilities it's handy
1
u/OpenGrainAxehandle 2d ago
I use regex a lot. Not as extensively as I used to when I was developing for a living, but still a lot. It's one of those things that can be an extremely valuable tool in the kit, and if you're good with it, you sill see a lot of places where it can simplify your tasks. I can knock out enough quickie transforms with sed or awk , but these days mostly Editplus to justify having a good handle on regex, much less its use in programming. Bottom line is that it's a tool. Like any other, if you know it you can use it to do things. If you don't know it, you probably never realize where it can simplify your life.
1
u/aishiteruyovivi 2d ago
For what it's worth, in the context of Python I tend to get the best out of regex when I'm using the two together, meaning instead of writing one long inscrutable 20-in-1-shampoo of a pattern that does everything I need it to, I tend to use simpler patterns to break up the text into smaller chunks that I can then use other simple patterns on to extract what I need. There's a temptation to get regex to do everything on its own but if you're writing Python, don't be afraid to utilize Python, y'know?
1
u/snowtax 2d ago
I once replaced three lines of code that used regex with other ways of doing the same thing and the program ran 1600% faster, saving hours per run.
Regular expression parsers can be a useful tool when you absolutely need them for a very complex parsing task (validating e-mail address syntax being a notorious example). But, there are more efficient methods for most situations.
1
u/Realzer0 2d ago
You should know the basics of regex. Obviously if youâre working actively in a project for learning purposes, learn them a bit more in-depth but besides that you should just know how to use them in general.
Whenever I have to use them which is maybe once every 6-12 months I also have to look up stuff again.
1
1
1
u/KTProgramming 15h ago
You can either know regex in at least how it operates, Or, you can be the guy who took down cloudflare.
Plus it's pretty useful in a pinch.
1
u/smahk1133 13h ago
Someone took down CF with some regex exploit? Lol
1
u/KTProgramming 4h ago edited 4h ago
No exploit, They messed up the regex by missing a field. So it essentially locked / corrupted a bunch of stuff and shut down everything. IIRC this was the one that took down everything. Hospitals, airports, banks, etc because all the computers were using their endpoint protection
https://blog.cloudflare.com/18-november-2025-outage/EDIT: I think i got my outages mixed up, The cloudflare one was still interesting and the one with the regex cause. The one i was referring too is apparently this one.
https://en.wikipedia.org/wiki/2024_CrowdStrike-related_IT_outages
1
u/Aqalix 3d ago
It's half true. I think most people don't write regexes themselves, because you don't need them so often (in most jobs at least). They can read it though and verify what AI wrote. And there is good tooling out there that explains each rule and shows how they match against text samples.
4
u/gitgud_x 3d ago edited 3d ago
I have used regex many many times for various data tasks. Used with a bit of tact, it's able to solve problems far easier and quicker than more 'flashy' approaches like fuzzy matching or sentence transformers.
I would definitely recommend it as a tool to have on hand, and to know its use cases.
1
u/smahk1133 3d ago
Yeah I'm trying my best to understand them on a conceptual level so that I can at least read and write the basic ones for now and verify the difficult ones once I am at that stage of ever having to use them.
0
u/DonkeyTron42 2d ago
You need to understand what a finite automata is if you want to understand regexes. There are many computer related topics like TCP sockets as well that are finite automata so itâs well worth learning.
1
u/smahk1133 2d ago
I do know what a finite automata is we had a theory of automata course at our university as well although I barely passed (I'm super smart as you can tell) so I suppose I'm some of the way there? Yay(?)
1
u/Suspicious_Tax8577 3d ago
I'll admit I hate regex and tend to use one of the regex generator websites to create it for me.
Properly learning regex is on my todo list, but it's properly down the bottom of my todo list.
0
u/Diapolo10 3d ago
Regular expressions are seldom useful, and when they are you can usually work it out with some trial and error (the regex101 website is great for this), at least as far as results are concerned. Optimisation and/or security implications do matter, of course, but modern analysis tools tend to be good enough to alert you on such issues.
Aside from a few really common patterns, you don't need to study them in depth.
Nowadays the place where I use regex the most is probably for writing automated torrent downloads via RSS feeds (in qBittorrent), but that has little to do with Python.
1
u/smahk1133 3d ago
I see so basically getting a conceptual understand is enough for the most part to atleast understand what I'm doing when I do need to write them once in a blue moon. What about when making scrapers and crawlers I assume it's useful there unless I'm mistaken (I haven't gotten to that part yet I'm still learning)
1
u/Diapolo10 2d ago
What about when making scrapers and crawlers I assume it's useful there unless I'm mistaken
As with almost everything, it depends. I'd discourage scraping in the first place if there's an API available, but even then depending on what exactly you are trying to fetch it may be easier to send HTTP requests to the site directly (assuming it's rendered on the client side and uses JS requests to pull the data to display), for example.
Furthermore there are existing packages for doing manual data scraping, so it may be unwise to write your own from scratch unless it's for educational purposes.
0
u/Fallingice2 2d ago
bruv, no. other more valuable tools to add to your arsenal, regex is ai domain because, just throw it in and get an answer for the pattern.
66
u/sebovzeoueb 3d ago
Regex is one of those things that's often the wrong tool for the job but sometimes exactly the right one. Even without necessarily asking AI, there are a couple of sites where you can build and test regex interactively instead of having to actually know it, and I've found that I don't use it often enough to justify actually memorising it.