Skip to content

Allow passing experimental_options to webdriver - #99

Open
erikkemperman wants to merge 2 commits into
clemfromspace:developfrom
erikkemperman:develop
Open

Allow passing experimental_options to webdriver#99
erikkemperman wants to merge 2 commits into
clemfromspace:developfrom
erikkemperman:develop

Conversation

@erikkemperman

@erikkemperman erikkemperman commented Oct 14, 2021

Copy link
Copy Markdown

This allows us to pass "experimental options" from the scrapy configuration to the webdriver. For example, when using chrome, the following settings lets us avoid downloading any images -- reducing the downloaded bytes significantly:

{
    'SELENIUM_DRIVER_NAME': 'chrome',
    'SELENIUM_DRIVER_EXECUTABLE_PATH': which('chromedriver'),
    'SELENIUM_DRIVER_ARGUMENTS': ['--headless'],
    'SELENIUM_DRIVER_EXPERIMENTAL_OPTIONS': {
        'prefs': {
            'profile.managed_default_content_settings.images': 2
        }
    },
    'DOWNLOADER_MIDDLEWARES': {
        'scrapy_selenium.SeleniumMiddleware': 800
    }
}

@erikkemperman

erikkemperman commented Oct 14, 2021

Copy link
Copy Markdown
Author

PS This also addresses some problems in setup.py which, it seems to me, unnecessarily relied on pip internals.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant