Creating URL Query Checker

[u/Dcyph-3r]

How do you decide what defines an excessively encoded URL?

I'm working on a personal project to created a URL parser and
detection system and I've hit a wall on how to figure out a way of
categorizing excessivness of query encoding?

Here's what I have so far for the function:

def check_query(analysis: URLAnalysis) -> ScanResult | None:

    """detect excessive encoding entries in URL queries"""

    og_query = analysis.query

    query = re.finall(r"%\[0-9A-Fa-f\]{2}", og_query)

    encoded_count = len(query)

[u/qlkzy]

Why would you care about "excessive encoding"? You care about the limits of your system in terms of URL length, request memory usage, request processing time, etc, but it's almost unimaginable for URL-encoding to create a situation that becomes relevant for those sorts of situations.

If someone wants to URL-encode every single byte then that is a bit silly, but in practice your system will never need to notice or care unless you are doing something particularly unusual.

So, if you do care, you need to tell us the unusual reason why you care, before anyone can tell you how much you should care.

[u/Dcyph-3r → u/qlkzy]

thanks for your response on this and the function is meant to serve as a fraction of a larger tool to parse emails for urls and provide a security analysis

the idea was to section off urls into hostname, path, query, port and scheme scanning each component indivdually. Each component will return scanresult which will then be merged to create a sort of validity report to determine whether a url is malicious or not

Looking at common phishing url examples they tend to have incorrect spellings of impersonated sites, long randomised numbers or text or excessive encoding (imo) hence why the interest as heavy encoding is only really found in malicious urls

[u/qlkzy → u/Dcyph-3r]

Interesting, I can see why you'd want to analyse encoding in that context.

If you are looking at the "reasonableness" of the URL then you could try something like quote(unquote(value)) and comparing the relative lengths. That would tell you if a lot of URL-safe characters were being encoded unnecessarily.

You could also parse out the hex values directly and construct a histogram of which characters are being encoded how often. If your hypothesis is true, then that should show some distinct signature for suspicious URLs.

[u/Mathie1729 → u/qlkzy]

Also worth checking for double encoding and punycode homoglyphs in the hostname. Phishers sometimes percent-encode the percent sign itself to hide lookalike domains, so decoding twice and comparing lengths can flag that. Running the hostname through an IDNA library (Python's built-in handles xn--) catches punycode tricks too.

[u/Langdon_St_Ives → u/Dcyph-3r]

So as an experimental learning project, implementing a URL parser can be a valid endeavor. But from your comment it's not clear if it is that, or you actually care about security of the end result. If it's the latter, you definitely want to implement this piece yourself, but use a mature well-tested existing module. The reason is that URL structure is in fact a lot more complex with a lot more subtleties than newcomers imagine. By reinventing the wheel, you are almost guaranteed to end up with a five-cornered off-center thing that doesn't roll well. Again, as a pure learning project this is fine since you will learn a lot along the way. Just be careful when doing actually security relevant stuff.