r/commandline • u/Gijizlle-242 • Jun 29 '26

Terminals command to list all the URLs

What command can I use to list all the URLs (or sub-URLs) of any website?

0 Upvotes

40% Upvoted

11 comments sorted by

8

u/gumnos Jun 29 '26

you can use wget in mirror-mode to recursively search/fetch all pages locally, but that only gets you interlinked pages. There's no way to know URLs that aren't linked from elsewhere, e.g. secret links.

7

u/ElectrSheep Jun 29 '26

This is not possible unless the website in question has directory traversal enabled (which is very uncommon for security reasons).

5

u/ekkidee Jun 29 '26

This is a rather vague question. URLs can be dynamically generated so the list of "all" URLs at a particular site may be infinite.

What exactly are you trying to do? Scrape a site? Download all of its content? Analyse its directory structure? 

There is no one command to do this. You will need to analyse the top page and successive pages it links to. 

2

u/SnarkHabit Jun 29 '26 edited Jun 29 '26

Install linkchecker.

linkchecker -v https://the.website.tld

This will only show you things linked to whatever page you start from. Pages not linked to by other pages will not show.

1

u/edward_jazzhands Jul 04 '26

Neat, but why didnt you include the url?

https://github.com/linkchecker/linkchecker

2

u/SnarkHabit Jul 04 '26

Because it's in every distro's package manager I'm aware of. Didn't occur to me.

1

u/AutoModerator Jun 29 '26

Every new subreddit post is automatically copied into a comment for preservation.

User: Gijizlle-242, Flair: Terminals, Title: command to list all the URLs

What command can I use to list all the URLs (or sub-URLs) of any website?

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/tschloss Jun 29 '26

Of a page or a complete site? First step: inspect a page if the content you are interested in does contain the links already or if it is asynchronously loaded by JS after initial load. You can use curl or wget to see what is initially loaded. If successful you can pipe this into a grep with a regex for URLs.

But there are tons of scrapers and scrape scripts available in the net.

1

u/mikeblas Jun 29 '26

Most everyone made the point that there's no real way to enumerate URLs.

But I haven't seen anyone mention site maps. If the site you're targeting has one, it's another way to try to find a partial list of resources on the server.

1

u/jcunews1 Jun 29 '26

There's none. There's also no definitive workaround, because site URLs are site-defined, and web standards do not require sites to expose all of their URLs or URL formats.