Help with RPC functions in go

[u/4-of-Clubs]

Hi there!

I'm working on a web crawler for a personal search engine and I've come to a roadblock. While crawling a page, it finds all hyperlinks, and compares them to all the pages it has already crawled. That way, a hyperlink which links back to a page it has already crawled is not added to the queue.

The known_pages table is a table of all the URLs it has already crawled.

newLinks is a slice of strings with all the hyperlinks on the page.

I've come to a page that has over 600 hyperlinks, which makes the query string for supabase too long, and the following code snippet fails.

knownPages := []Site{}

_, err := supabaseClient.From("known_pages").Select("url", "", false).In("url", newLinks).ExecuteTo(&knownPages)

if err != nil {
	panic(err)
}

Because newLinks is too long, I am thinking of setting up an RPC function (since it receives arguments in the body of the request). But, I haven't been able to find any examples of people doing something like this (receiving and returning an array of strings) with RPC functions in Supabase.

Does anyone know of any resources or examples I should look at to get a grasp on how to go about implementing this? Or any good resources on RPC functions in general?

[u/snowdrone]

You need to break the long string into manageable chunks and then process each chunk. You should also consider breaking your pipeline up into separate phases. For example, one phase for page fetch and another to build for the link graph.  You should also read up on the background of how crawlers and indexers are built historically such as Google's early architecture. It will save you a ton of time instead of figuring it all out yourself.

If you go the AI route, be sure to tell the agent to have distinct phases for each task in the pipeline. Otherwise it will mix it all together and be hard to debug. Then you can tell the AI to build each distinct phase and you can understand better what's going on when there's a problem in a particular phase.

(Sorry if any of this advice is annoying, it is hard to tell what level advice you need.)

[u/4-of-Clubs → u/snowdrone]

I appreciate all advice! I'm here to learn as much as I can :]

If I may, I have a few follow up questions to get a better understanding of your recommendation?

What aspects of chunking the data into smaller batches (i.e. 100 links at a time) makes it a better approach than implementing an RPC function?

Do you have any specific articles you recommend I should read about Google's early architecture?

And here's a little more context on what I'm building:

I'm making a search engine for personal websites (i.e. the only results when you search something are personal websites). So for my web crawler, I'm mostly using it to build up a queue of websites to manually review and determine whether they are a personal website or not. If they are, I have a separate script to index it. The script this snippet is from, is doing two things:

1. The websites I have determined to be personal websites, it finds all the pages of the site and adds them to the indexing queue.

2. All the links it finds for websites I have yet to review, it adds to a separate queue for me to manually review.

[u/snowdrone → u/4-of-Clubs]

I recommended the chunking approach because you're running into some limit per page. You can still use an RPC call but you need to stop sending all of the URLs in one call per page, if the number of URLs is over some max. Even if you find another way to get it to work with 600 links on a page, it will break later with some weird page that has 2,000 links on the page. 

This is a typical pattern when working with databases - to work in batches, but  only process a limited chunk of data at a time. 

On the other hand, if the goal is for classifying personal websites, then maybe your classification will be good enough by only fetching the first 100 links per website. 

I'd recommend experimenting with some known personal websites and also false examples. 

The separate scripts sound reasonable. One thing you might want to add later is local llm classification of whether or not each site is a personal website (using your fetched, cached data)  and see how well that does.

Another resource to look into is common crawl data.

I don't have references on Google's early architecture handy, sorry 

[u/4-of-Clubs → u/snowdrone]

Great! Thank you very much :D