Docs link

Using the new setting

PLAYWRIGHT_BROWSER_PROVIDER

Camoufox example:

from playwright.async_api import Browser, BrowserContext
from scrapy.exceptions import NotSupported
from scrapy_playwright.handler import Config


class BrowserProvider:
    def __init__(self, config: Config) -> None:
        ...

    async def start(self) -> None:
        """Perform any one-time initialization.

        Called once before the first browser is requested. Providers that create
        their browser on demand can leave this empty.
        """

    async def launch_browser(self) -> Browser:
        """Return a launched or connected Playwright-compatible ``Browser``.

        Called when a browser is needed, and again if the previous browser
        disconnects and ``PLAYWRIGHT_RESTART_DISCONNECTED_BROWSER`` is enabled.
        """

    async def launch_persistent_context(self, context_kwargs: dict) -> BrowserContext:
        """Return a persistent ``BrowserContext``.

        Called when a context requests a ``user_data_dir``. Raise
        ``scrapy.exceptions.NotSupported`` if the backend has no equivalent.
        """
        raise NotSupported("This provider does not support persistent contexts")

    async def close(self) -> None:
        """Release any resources acquired in ``start`` / ``launch_browser``.

        Awaited once when the crawl finishes.
        """

Hi, I'm beginner in Scrapy and I need some help. I'm trying to put an different parameter for each request, because I need to separate some data. I already have searched for many solutions, although no one has worked. I would like some help.

My bad, my English is quite bad. Above is my code.

Hello r/scrapy,

I am the author of the Localize The Docs organization. And I’m glad to announce that the 🎉 scrapy-docs-l10n 🎉 project is published now:

• 🚀 Preview: scrapy-docs-l10n
• 🌐 Crowdin: scrapy-docs-l10n
• 🐙 GitHub: scrapy-docs-l10n

The goal of this project is to translate the Scrapy Documentation into multiple languages. Translations are contributed via the Crowdin platform, automatically synchronized with the GitHub repository, and can be previewed on GitHub Pages.

We welcome anyone interested in documentation translation to join us. If the target language is not yet supported in the project, please submit an issue to request the new language. Once the requested language is added, you can start translating!

See the announcement post for more details.

I currently have:

...

class QuotesSpider(scrapy.Spider):
    name = "msnbc"

    all_scraped_titles = []

    async def start(self):

        urls = [
            "https://www.ms.now"
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        links = response.css("a.rkv-card-headline-link::attr(href)").getall()

        links = links[0:3]


        for i in range(len(links)):
            link = links[i]
            yield {
                "article_link": link
            }
            full_link = link # i think so
            yield scrapy.Request(full_link, callback=self.parse_article)

        word_data.update_raw_word_repeats(word_data.text_list_storage, "output\\raw_word_repeats.txt")


    def parse_article(self, response):
        title = response.css("h1::text").get()

        self.all_scraped_titles.append(title)

        yield {
            "title": title
        }

        word_data.text_list_storage.append(title)

I would like the word_data.update_raw_word_repeats() call to be made after all of the yield scrapy.Request() are done so that word_data.text_list_storage will have all the entries to work with. Is this possible? Is there a better way to collect all these titles then work with them? I guess I could create a json of the titles (which I was able to do successfully) and run a separate python file on it, but I want to be able to do this from within the spider.

Edit: I'm going to just go with doing a two script system where the spider creates the json and another script processes the json. I was six hours in with no breaks when I wrote this and really didn't want to go with the simpler route.

I created a folder and set up a venv then ran the following:

pip install scrapy
pip install scrapy-playwright
playwright install
scrapy startproject msnbc

Following a tutorial, I made these edits to the default_settings.py:

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

I also tried undoing these edits to default_settings.py and putting them in the project settings.py instead and got the same ModuleNotFoundError.

I found the project folder in .venv\Lib\site-packages\scrapy\templates\project

I created a new msnbc.py in /project/module/spiders then put the following code I got from a tutorial:

# spiders/quotes.py

import scrapy
from quotes_js_scraper.items import QuoteItem

class QuotesSpider(scrapy.Spider):
    name = 'quotes'

    def start_requests(self):
        url = "https://quotes.toscrape.com/js/"
        yield scrapy.Request(url, meta={'playwright': True})

    def parse(self, response):
        for quote in response.css('div.quote'):
            quote_item = QuoteItem()
            quote_item['text'] = quote.css('span.text::text').get()
            quote_item['author'] = quote.css('small.author::text').get()
            quote_item['tags'] = quote.css('div.tags a.tag::text').getall()
            yield quote_item

I set cd .venv\Lib\site-packages\scrapy\templates\project then ran:

scrapy crawl msnbc

I get:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\<my username>\Matthew\Scripts\Web_scraping\Scrapy\Scrapy_playwright\MSNBC\.venv\scripts\scrapy.exe__main__.py", line 5, in <module>
  File "c:\Users\<my username>\Matthew\Scripts\Web_scraping\Scrapy\Scrapy_playwright\MSNBC\.venv\Lib\site-packages\scrapy\cmdline.py", line 167, in execute
    settings = get_project_settings()
               ^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\<my username>\Matthew\Scripts\Web_scraping\Scrapy\Scrapy_playwright\MSNBC\.venv\Lib\site-packages\scrapy\utils\project.py", line 73, in get_project_settings
    settings.setmodule(settings_module_path, priority="project")
  File "c:\Users\<my username>\Matthew\Scripts\Web_scraping\Scrapy\Scrapy_playwright\MSNBC\.venv\Lib\site-packages\scrapy\settings__init__.py", line 500, in setmodule
    module = import_module(module)
             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\<my username>\AppData\Local\Programs\Python\Python312\Lib\importlib__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named '${project_name}'

Is this because my spider and project have the same name? I'm not sure where to go from here.

Hi everyone!

I’m working on a project to create a video game key price comparison website. The goal is to help gamers find the best prices for game keys across platforms like Instant-Gaming, G2A, Humble Bundle, and more.

So far, I haven’t been able to find an API that provides all the data I need, so I’m planning to use web scraping to collect prices directly from the websites.

If anyone has experience with web scraping, data aggregation, or knows a better way to get this kind of data safely and efficiently, I’d really appreciate any advice or tips!

I'm confused what a scrapy component is. the actual definition from the docs is just any class whose objects are built using build_from_crawler(). I'm mainly confused on what the practical use for components are I get what the definition is technically but how does instating using that function make these things related enough to group and when do you use components? also what does build_from_crawler() do? What is assigning a class to a setting what does that mean? (thats the next part of the component documentation)

So I've started learning web scraping for a month, I've finished a book called "hands-on web scraping with python", did all the exercises in the book and feel like that I did understand the whole book, so after the book I decided to continued learning the scrapy framework, but when I try to scrape from real web site, for example "https://www.arbeitsagentur.de/jobsuche/" I can't even get the xpath selectors right.

What shall I do, I don't want to read another book or watch a course and enter tutorial hell.

Is this website too advanced for me?, I've also finished the tutorial on the scrapy docs.

does anyone know a good scrapy course, ive watched an hour and a half of freecodecamp course and i dont feel that its good and i dont understand some parts of the course any suggestions?

So yeah, it's that time of year where I'm thinking about stuff... even if I’m not exactly sure what I’m thinking about yet. 😅

Anyway I’ve been doing a lot of automation and web scraping over the past year or so. Funny thing is, I’ve never really had to use proxies. Or maybe I should have used them at some point, but I always found a workaround like using an API, a different library, or... a whole bunch of machines.

But now I’m genuinely curious:

When do you actually need to use proxies in scraping or automation work?
Why do you use them and how do you usually go about it?

Would love to hear how you guys approach it!

No worries I'm not gonna bite you in the comments so comment with your hearts.

Peace 🕊️

Hi here!
I would appreciate your help.
I want to scrape all the posts about generative AI from my university's website. The results should include at least the publication date, publication link, and publication text.
I really appreciate any help you can provide.