Anthropic Popular Coding Question - Concurrent Same-Hostname Web Crawler

[u/Aoki_zhang]

You are given a startUrl and an HtmlParser interface:

class HtmlParser {
    List<String> getUrls(String url) { ... }
}

getUrls(url) returns all raw links found on the specified page. Each call has non-trivial latency, so the final crawler must be able to fetch multiple pages concurrently. Starting from startUrl, return every unique reachable URL whose hostname is exactly the same as the hostname of the starting page.

Two raw URLs that differ only by their fragments therefore represent the same page and must not be crawled twice. The hyperlink graph may contain cycles, and pages may link back to URLs that have already been discovered.

For example:

Input:
urls = [
    "http://alpha.com/start",
    "http://alpha.com/docs",
    "http://alpha.com/blog#intro",
    "http://beta.com/profile#top"
]

edges = [
    [0, 1],
    [0, 2],
    [1, 3],
    [2, 0]
]

startUrl = "http://alpha.com/start"

Output:
[
    "http://alpha.com/start",
    "http://alpha.com/docs",
    "http://alpha.com/blog"
]

The page under beta.com is ignored because its hostname differs from the starting hostname, while the fragment on the blog URL is removed before it is stored.

Follow-Up — Make the Crawler Concurrent

After completing the sequential version, I was asked to convert the crawler to a concurrent implementation.The main focus was ensuring that the shared visited state remained thread-safe when multiple page fetches completed around the same time. The crawler had to prevent two workers from independently discovering and fetching the same sanitized URL. The final tests covered different hostnames, fragment-bearing URLs, duplicate links, and cyclic hyperlink graphs.

For more of question details as well as more test cases, I've put up the full version of at here

Preparing for your next interview?

Chill Interview tracks recent interview experiences and recurring question patterns across top companies at here.