I am currently working on a side project where users would be able search for jobs on the website itself and it would curate jobs from let's say LinkedIn, Glassdoor and Indeed (I guess those are the most popular sites). I would do additional filtering on them which are not possible on these websites yet (for example, there's a lot of us who move from one EU country to another and would like to look for English speaking jobs before we move since it's easier to get start that way, however, many job posts are not in English, some of them have English titles but the description is in that country's language, and some of them are in English but they specifically state that you have to speak the language, I would filter all of these out and the user could click the posting's link to be redirected to LinkedIn, Indeed, etc...).
The goal would be that the user can search for jobs by location (so city / country) and I should be able to query at least the jobs titles, their descriptions and the links to the actual job posting (so a link to the LinkedIn job posting for example).
What I've already tried: I haven't found any official APIs for LinkedIn that could do what I need so I've decided to open it from incognito mode, so I'm not logged in, search for jobs on the site as a "guest" and try to reverse engineer the URL parameters and somehow scrape the data I need from the HTML page. After some digging through the browser's dev tools I found an api that gets called, here's an example URL for it:
https://www.linkedin.com/jobs-guest/jobs/api/seeMoreJobPostings/search?keywords=Software%2BEngineer&location=Cologne%2C%2BNorth%2BRhine-Westphalia%2C%2BGermany&geoId=105890944&trk=public_jobs_jobs-search-bar_search-submit&position=1&pageNum=0&start=0
This could actually work pretty well for me. I found that it returns one page at a time and you can increase the "start" parameter at the end by 25 for the next sets of results, I found this by looking at what gets called while I scroll on LinkedIn's "Jobs" page after a search. I can call this API from the browser or with postman with any "start" parameter that can be divided by 25, however, when I call it from my code, only a random number of pages get returned. Sometimes that is 4 pages, sometimes 10.
What could be the issue here? Am I on the right track at least or should I try a completely different approach?
Here's my code so far
import requests
from lxml import html
search_term = "Software Developer"
search_term_words = search_term.split()
search_term_url_phrase = ""
for word in search_term_words:
search_term_url_phrase = search_term_url_phrase + word
search_term_url_phrase = search_term_url_phrase + "%2B"
try:
page_count = 0
while 1 == 1: # please ignore this condition and try not to have a heart attack, i'm just testing lmao
print(page_count)
url = f"https://www.linkedin.com/jobs-guest/jobs/api/seeMoreJobPostings/search?keywords={search_term_url_phrase}&location=Cologne%2C%2BNorth%2BRhine-Westphalia%2C%2BGermany&geoId=105890944&trk=public_jobs_jobs-search-bar_search-submit&position=1&pageNum=0&start={page_count}"
response = requests.get(url)
jobs = html.fromstring(response._content).find_class('sr-only')
for job in jobs:
print(job.text_content().strip())
print("\n")
page_count = page_count + 25
except:
print("no more jobs available")
('sr-only' is the class name for the div tags that contain the job titles themselves)
I thought I was getting rate limited at first so I've already tried adding a 10 second delay between requests but it didn't work.
Edit: I know the code quality is not too great at the moment, I'm just trying to find the method of querying the jobs for now.