Make writing a habit together! This is the fifth day of my participation in the “Gold Digging Day New Plan · April More text Challenge”. Click here for more details.

Selenium operations are shielded

When using Selenium automated web pages, there is a probability that they will be identified by the target site. Once detected, the target site will block the web actions made by the client.

For example, the login pages of Taobao and Dianping can be entered normally when you manually open the browser and enter the user name and password. However, if you open the login page through Selenium, you will be prompted with verification failure and click the box to try.

This article describes a way to easily bypass site detection without modifying browser properties or injecting JavaScript.

Why was Selenium detected

Each browser visits a site with a specific fingerprint feature, which the site interprets to determine if the visit is automated.

One of the most well known is characterized by the window. The navigator. Webdriver, the characteristics indicate that the browser is webdriver program directly. When a browser is launched through Selenium and you enter this property in the developer tools, you will see that it is marked as true, as opposed to false if the browser is opened manually.

In fact, the browser is not the only feature that can be detected as a WebDriver application, which means that you won’t necessarily be able to bypass the site’s detection by modifying its attributes.

We can use SannySoft to detect browser fingerprints. If the browser is opened through an automated program such as Selenium, there are many features that expose these fingerprints after accessing the site. The values of these features are different from those of manual opening, so they can be easily detected by others.

Some have tried to ditch Selenium for automated tools such as Puppeter and ourselves, but the results are the same.

Selenium avoids detection methods

Browsers are fingerprinted when they start up. If they are already marked when they start up using an automation program, why not just start manually and then use Selenium to connect to the launched browser?

So what do YOU have to do to manually open a browser to make Selenium connect? How does Selenium connect to manually opened browsers?

The steps required are documented in detail in my Selenium Connect to existing Browser 1 article, which summarizes:

1. Add the following parameters when opening the browser:

--remote-debugging-port=9222 --user-data-dir="C:\\selenium\\ChromeProfile"
Copy the code

2. Set the browser options in Selenium and connect to the browser through port 9222 set above:

from selenium.webdriver.chrome.options import Options chrome_options = Options() chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:9222") driver = webdriver. Chrome (options = chrome_options)Copy the code

Run the browser through subprocess

Of course, instead of manually clicking on an icon to open the browser, you can use the command line to launch the browser and then use Selenium to connect.

import subprocess
cmd = '"C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe" ' \\
'--remote-debugging-port=9222 ' \\
'--user-data-dir="C:\\selenium\\ChromeProfile"'

subprocess.run(cmd)
Copy the code

I am nine handle, thank you for reading, see you next time.