1. Introduction to Puppeteer

Puppeteer is a Node library maintained by the Official Chrome team. It provides a set of apis for manipulating Chrome, which is basically a Headless Chrome browser (although you can also configure it to have a UI, which is not available by default).

Since it is a browser, Puppeteer can do all the things that we can do manually on the browser. In addition, Puppeteer translates to “puppet” in Chinese, so it is very easy to manipulate, and you can easily manipulate it to achieve:

1) generate web screenshots or PDF 2) advanced crawler, which can climb a large number of asynchronously rendered web pages 3) simulate keyboard input, form automatic submission, login web page, etc., to achieve automated UI testing 4) capture the site timeline, so as to track your site and help analyze site performance problems

2. Operating environment

Check the official Puppeteer API and you’ll find a screen full of async, await, etc. These are ES7 specifications, so you need to:

  1. Nodejs cannot be less than V7.6.0 and needs to support async, await.
  2. You need the latest Chrome driver, which will be downloaded automatically when you install Puppeteer via NPM
npm install puppeteer --save
Copy the code

3. Basic usage

Let’s start with the official introductory DEMO

const puppeteer = require('puppeteer'); (async () => { const browser = await puppeteer.launch(); const page = await browser.newPage(); await page.goto('https://example.com'); await page.screenshot({path: 'example.png'}); await browser.close(); }) ();Copy the code

The above code is to achieve a screenshot of the web page, first read a few lines of code above:

  1. Start by creating a Browser instance Browser object with puppeteer.launch()
  2. The Page Page object is then created from the Browser object
  3. Then page.goto() jumps to the specified page
  4. Call page.screenshot() to take a screenshot of the page
  5. Close the browser

Isn’t that easy? I find it easier than PhantomJS anyway, not to mention selenium webDriver. Here are a few apis that are commonly used for Puppeteer.

3.1 a puppeteer. Launch (options)

Run the puppeteer with puppeteer.launch(), which returns a promise. It is important to note that almost all operations in Puppeteer are asynchronous.

3.2 Browser objects

When Puppeteer is connected to a Chrome instance, a Browser object is created in two ways:

The Puppeteer. Launch and a Puppeteer. The connect.

The following DEMO implementation reconnects the browser instance after disconnecting

const puppeteer = require('puppeteer'); Puppeteer.launch ().then(async browser => {// Save the Endpoint, Rewire Chromium const browserWSEndpoint = browser.wsendpoint (); // Disconnect from Chromium browser.disconnect(); // Use the endpoint to reconnect to Chromiunm const browser2 = await puppeteer.connect({browserWSEndpoint}); // Close Chromium await browser2.close(); });Copy the code
Browser object API
Method names The return value instructions
browser.close() Promise Close the browser
browser.disconnect() void Disconnect the Browser
browser.newPage() Promise(Page) Create a Page instance
browser.pages() Promise(Array(Page)) Gets all open Page instances
browser.targets() Array(Target) Get targets for all activities
browser.version() Promise(String) Get the browser version
browser.wsEndpoint() String Returns the socket connection URL of the browser instance that can be used to reconnect the Chrome instance

4. Some of the API

4.1 Obtaining Elements

The Page object provides two apis to retrieve Page elements

(1).page.$(selector) gets a single element, and the underlying call is document.querySelector(), so the selector format follows CSS selector specifications

let inputElement = await page.$("#search", input => input); // let inputElement = await page.$('#search');Copy the code

(2). Page. $$(selector) to obtain a set of elements, the bottom of the call is the document querySelectorAll (). Returns an Array of Promise(Array(ElemetHandle) elements.

const links = await page.$$("a"); // const links = await page.$$("a", links => links);Copy the code

All that is returned is the ElemetHandle object

4.2 Obtaining Element Attributes

Puppeteer: Puppeteer: Puppeteer: Puppeteer: Puppeteer: Puppeteer: Puppeteer: Puppeteer: Puppeteer: Puppeteer: Puppeteer: Puppeteer: Puppeteer: Puppeteer: Puppeteer: Puppeteer: Puppeteer If you look at the ElemetHandle API, you will find that there is no API for retrieving element attributes.

Puppeteer actually provides a set of apis for obtaining attributes, Page.$eval() and Page.$$eval().

Page.$$eval(selector, pageFunction[,… args]) to get the attributes of a single element.

const value = await page.$eval('input[name=search]', input => input.value);
const href = await page.$eval('#a", ele => ele.href);
const content = await page.$eval('.content', ele => ele.outerHTML);
Copy the code

4.3 Executing custom JS Scripts

The Page object in Puppeteer provides a set of Evaluate methods that you can use to execute custom JS code. It provides the following three apis

(1). Page. The evaluate (pageFunction,… PageFunction represents the function to be executed on the page. Args represents the argument passed to pageFunction. PageFunction and args represent the same meaning below.

const result = await page.evaluate(() => {
    return Promise.resolve(8 * 7);
});
console.log(result); // prints "56"
Copy the code

This method is very useful, for example, when we take a screenshot of the page, the default is to take a screenshot of the current browser window size, the default is 800×600, so if we need to take a screenshot of the entire page can not be done. The page.screenshot () method provides parameters that can set the screenshot area size, so just get the width and height of the Page after the Page is loaded to solve this problem.

(async () => { const browser = await puppeteer.launch({headless:true}); const page = await browser.newPage(); await page.goto('https://jr.dayi35.com'); await page.setViewport({width:1920, height:1080}); const documentSize = await page.evaluate(() => { return { width: document.documentElement.clientWidth, height : document.body.clientHeight, } }) await page.screenshot({path:"example.png", clip : {x:0, y:0, width:1920, height:documentSize.height}}); await browser.close(); }) ();Copy the code

(2). Page. EvaluateHandle (pageFunction,… Args) performs a pageFunction in the Page context, returning the JSHandle entity

const aWindowHandle = await page.evaluateHandle(() => Promise.resolve(window));
aWindowHandle; // Handle for the window object. 

const aHandle = await page.evaluateHandle('document'); // Handle for the 'document'.
Copy the code

As you can see from the code above, the Page.evaluateHandle () method also returns the final processing result of a Promise directly through the promise.resolve method, but encapsulates the final returned object as a JSHandle object. It’s essentially no different from evaluate.

The following code implements the HTML code that gets the page dynamically (including the element that JS dynamically inserts).

const aHandle = await page.evaluateHandle(() => document.body);
const resultHandle = await page.evaluateHandle(body => body.innerHTML, aHandle);
console.log(await resultHandle.jsonValue());
await resultHandle.dispose();
Copy the code

(3). Page. EvaluateOnNewDocument (pageFunction,… If there is an iframe or frame in the page, the context of function call will become the sub-page, namely iframe or frame. Since it is called before the page is loaded, This function is typically used to initialize the javascript environment, such as resetting or initializing global variables.

4.4 Page. ExposeFunction

In addition to the above three apis, there is a similar and very useful API: page.exposeFunction. This API is used to register global functions on a Page, which is very useful:

There are some functions that need to be used to perform some operations on a Page, although you can use the page.evaluate () API to define functions on a Page, for example:

const docSize = await page.evaluate(()=> {
    function getPageSize() {
        return {
            width: document.documentElement.clientWidth,
            height : document.body.clientHeight,
        }
    }

    return getPageSize();
});
Copy the code

Nodejs has a number of toolkits that make it easy to implement complex functions, such as md5 encryption functions. This is not convenient to implement in pure JS. Using NodeJS takes a few lines of code.

Add the md5 function to the window object of the Page context:

const puppeteer = require('puppeteer');
const crypto = require('crypto');

puppeteer.launch().then(async browser => {
  const page = await browser.newPage();
  page.on('console', msg => console.log(msg.text));
  await page.exposeFunction('md5', text =>
    crypto.createHash('md5').update(text).digest('hex')
  );
  await page.evaluate(async () => {
    // use window.md5 to compute hashes
    const myString = 'PUPPETEER';
    const myHash = await window.md5(myString);
    console.log(`md5 of ${myString} is ${myHash}`);
  });
  await browser.close();
});
Copy the code

As you can see, the Page.exposeFunction API is handy to use, and useful for things like registering the readfile global function on a window object:

const puppeteer = require('puppeteer');
const fs = require('fs');

puppeteer.launch().then(async browser => {
  const page = await browser.newPage();
  page.on('console', msg => console.log(msg.text));
  await page.exposeFunction('readfile', async filePath => {
    return new Promise((resolve, reject) => {
      fs.readFile(filePath, 'utf8', (err, text) => {
        if (err)
          reject(err);
        else
          resolve(text);
      });
    });
  });
  await page.evaluate(async () => {
    // use window.readfile to read contents of a file
    const content = await window.readfile('/etc/hosts');
    console.log(content);
  });
  await browser.close();
});
Copy the code

5. Page. Emulate: Modify the simulator configuration

Puppeteer provides apis for modifying the configuration of the browser terminal

  1. Page.setviewport () changes the browser window size
  2. Page.setuseragent () sets the UserAgent information of the browser
  3. Page.emulatemedia () changes the CSS media type of the Page for emulating media. The options are “screen”, “print”, and “null”. If this parameter is set to null, media emulation is disabled.
  4. Page. Emulate () : emulate devices (iPhone, Mac, Android, etc.)
page.setViewport({width:1920, height:1080}); // Set the window size to 1920x1080 page.setUserAgent('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36'); page.emulateMedia('print'); // Set the printer media styleCopy the code

In addition, we can simulate non-PC devices, such as the following code to simulate an iPhone 6 visiting Google

const puppeteer = require('puppeteer');
const devices = require('puppeteer/DeviceDescriptors');
const iPhone = devices['iPhone 6'];

puppeteer.launch().then(async browser => {
  const page = await browser.newPage();
  await page.emulate(iPhone);
  await page.goto('https://www.google.com');
  // other actions...
  await browser.close();
});
Copy the code

Puppeteer supports many device emulators such as Galaxy, iPhone, IPad, etc. For more details about device support, please click here DeviceDescriptors.

6. Keyboard and mouse

The keyboard and mouse apis are relatively simple. The keyboard apis are as follows:

  • Key. down(key[, options]) Triggers the KeyDown event
  • Key. press(key[, options]) When you press a key, key indicates the name of the key, for example, ‘ArrowLeft’
  • Keyboard.sendcharacter (char) Enters a character
  • Keyboard. Type (text, options) Enter a character string
  • Keyboard. Up (key) Triggers the keyUp event
page.keyboard.press("Shift"); / / by pressing the Shift key page. The rid_device_info_keyboard. SendCharacter (' hi '); page.keyboard.type('Hello'); Page.keyboard. Type ('World', {delay: 100}); // Type slowly like the userCopy the code

Mouse operation:

  • Mouse. Click (x, y, [options]) Move the mouse pointer to the specified position, and then press the mouse

  • Mousedown ([options]) triggers the mouseDown event. Options can be configured:

    • Options. button Which key is pressed. The value can be [left, right, middle]. The default value is left, indicating the left mouse button
    • Options. clickCount Number of times to press, click, double click, or other times
    • Delay Delay time of a key

  • Mouse. Move (x, y, [options]) Moves the mouse to a specified position. Options. steps indicates the move step

  • Mouse. Up ([options]) triggers the mouseup event

7. Several other useful apis

Puppeteer also provides several useful apis, such as:

7.1 Page. WaitFor series API

  • Page. The waitFor (selectorOrFunctionOrTimeout [, options args [,…]]) of the following three comprehensive API
  • Page.waitforfunction (pageFunction[, Options [,… args]]) waits for pageFunction to complete
  • Page.waitfornavigation (options) waits for the basic elements of the page to load, such as synchronized HTML, CSS, JS, etc
  • Page. WaitForSelector (selector[, options]) waits for a selector element to load, which can be loaded asynchronously, which is a very useful API, as you know.

For example, if I want to get an element loaded asynchronously through JS, I can’t get it directly. This can be solved by using page.waitForSelector:

await page.waitForSelector('.gl-item'); // Wait for the element to load, Const links = await page.$$eval('.gl-item >.gl-i-wrap >.p-img > a', links => { return links.map(a => { return { href: a.href.trim(), name: a.title } }); });Copy the code

In fact, the above code can solve our top demand, grab jingdong products, because it is asynchronous loading, so use this way.

7.2 page. GetMetrics ()

Page.getmetrics () provides some page performance data to capture a timeline trace of your site to help diagnose performance problems.

  • Timestamp indicates the Timestamp of the metric sampling
  • Documents Number of Documents on the page
  • Frames Number of Frames on the page
  • Number of event listeners on the JSEventListeners page
  • Nodes Number of DOM Nodes on a page
  • LayoutCount Total number of page layouts
  • RecalcStyleCount style recounts
  • LayoutDuration The combined duration of all page layouts
  • RecalcStyleDuration The combination duration recalculated for all page styles.
  • ScriptDuration Duration for which all scripts are executed
  • TaskDuration Duration of all browser tasks
  • JSHeapUsedSize Size of the heap used by JavaScript
  • JSHeapTotalSize Total JavaScript heap size