Urllib +Re(regular expression) is used to write crawlers to achieve, but today I understand Requests library, it is more convenient to initiate Requests than URllib, I am based on the Chinese university MOOC in Beijing Institute of Technology teacher Song Tian “Python Web crawler and Information Extraction” course for the introduction of learning

Requests the libraryThe website address

Several main methods of the Requests library

These HTTP methods are described below

The get method

import requests
r = requests.get("http://www.baidu.com")
type(r)
Copy the code

r.status_code
Copy the code

r.text
Copy the code

r.headers
Copy the code

r.encoding = 'utf-8'
r.text
Copy the code

The head () method

>>> r = requests.head('http://httpbin.org/get')
>>> r.headers
{'the Content ‐ Length': '238'.'Access ‐ Control ‐ Allow ‐ Origin': The '*'.'Access ‐ Control ‐ Allow ‐ Credentials': 'true'.'the Content ‐ Type': 'application/json'.'Server': 'nginx'.'Connection': 'keep ‐ alive'.'Date': 'Sat, 18 Feb 2017 12:07:44 GMT'}
>>> r.text
' '
Copy the code

Post () method

>>> payload = {'key1': 'value1'.'key2': 'value2'}
>>> r = requests.post('http://httpbin.org/post', data = payload)
>>> print(r.text)
{  ...
"form": {
"key2": "value2"."key1": "value1"}},Copy the code

>>> r = requests.post('http://httpbin.org/post', d  a  t  a = 'ABC')
>>> print(r.text)
{  ...
"data": "ABC"
"form": {},}Copy the code

requests.request()methods

requests.request(method, url, **kwargs)

• Method: indicates the request mode, including get, PUT, and POST
• Url: The URL link of the page to be obtained
• **kwargs: 13 parameters to control access

The six HTTP equivalents actually call the Requests method

r = requests.request('GET', url, **kwargs)
r = requests.request('HEAD', url, **kwargs)
r = requests.request('POST', url, **kwargs)
r = requests.request('PUT', url, **kwargs)
r = requests.request('PATCH', url, **kwargs)
r = requests.request('delete', url, **kwargs)
r = requests.request('OPTIONS', url, **kwargs)
Copy the code

Where **kwargs: control access parameters have the following

Params example

>>> kv = {'key1': 'value1'.'key2': 'value2'}
>>> r = requests.request('GET'.'http://python123.io/ws', params=kv)
>>> print(r.url) http://python123.io/ws? key1=value1&key2=value2Copy the code

The data example

>>> kv = {'key1': 'value1'.'key2': 'value2'}
>>> r = requests.request('POST'.'http://python123.io/ws', data=kv)
>>> body = 'Body content'
>>> r = requests.request('POST'.'http://python123.io/ws', data=body)
Copy the code

Json example

>>> kv = {'key1': 'value1'}
>>> r = requests.request('POST'.'http://python123.io/ws', json=kv)
Copy the code

Headers example

>>> hd = {'the user ‐ agent': 'Chrome/10'}
>>> r = requests.request('POST'.'http://python123.io/ws', headers=hd)
Copy the code

Example files

>>> fs = {'file': open('data.xls'.'rb')}
>>> r = requests.request('POST'.'http://python123.io/ws', files=fs)
Copy the code

The timeout example

>>> r = requests.request('GET'.'http://www.baidu.com', timeout=10)
Copy the code

Proxies example

>>> pxs = { 'http': 'http://user:[email protected]:1234'
'https': 'https://10.10.10.1:4321' }
>>> r = requests.request('GET'.'http://www.baidu.com', proxies=pxs)
Copy the code

Exception handling

ConnectTimeout is a Timeout in the process of connecting to the server, and Timeout is a Timeout in the process of initiating the request (including the time required before and after connecting to the server).

R.r aise_for_status () function

A code framework for crawling web pages

A crawling web page picture and save the code case