1. Import the extension module

1# -*- coding: UTF-8 -*-
2Import network request module requests
3import requests
4# import the random module to generate random numbers
5import random
Copy the code

2. Define crawler address and disguise header information

 1# define the request url interface (interface) need to request, assumes that the interface at https://movie.douban.com/j/new_search_subjects
 2url = 'https://movie.douban.com/j/new_search_subjects'
 3
 4Prepare user-agent in advance for header information camouflage
 5How to generate user-agent
 6user_agent = ['the Mozilla / 5.0 (Windows NT 10.0; Win64; X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 '
 7              'Safari / 537.36'.8              'the Mozilla / 4.0 (compatible; MSIE 7.0; Windows NT 6.0) '.9              'the Mozilla / 5.0 (Windows; U; Windows NT 6.1; En-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 '
10              'Safari / 534.50'
11              ]
12
13Define request header information (dictionary type)
14headers = {
15    Random.randint (0,2) randomly fetkes one defined in the array
16    'User-Agent': user_agent[random.randint(0.2)]
17}
Copy the code

Encapsulate the request parameters

1Set the request parameters, according to the test range must be assigned to fetch data
2params = {
3    'sort': ' '.4    'range': '0, 20'.5    'tags': ' '.6    'start': '0'.7    'genres': 'love'.8}
Copy the code

4. Execute the request and process the return result

 1Execute the GET request
 2response = requests.get(url=url, headers=headers, params=params)
 3
 4Print the response information and return it as JSON data
 5print response.content
 6
 7Serialize output using response.json()
 8data_json = response.json()
 9print data_json
10
11Extract concrete data from data_json
12data_json_array = data_json['data'13]print data_json_array
14
15Get the movie name and score information
16for data_line in data_json_array:
17    print 'Movie Title :', data_line['title'].'-> score :', data_line['rate']

Copy the code

More exciting things to come to wechat public account “Python Concentration Camp”, focusing on Python technology stack, information acquisition, communication community, dry goods sharing, looking forward to your joining ~