from bs4 import BeautifulSoup
import urllib.request

url='https://topic.autohome.com.cn/new/home/sos.jsp?isNonCar=0&nonCar=0&brandId=25&seriesId=0&page=1'
open_it = urllib.request.urlopen(url)  
Copy the code

Urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] Certificate verify failed: Unable to get local issuer certificate (_SSL.c :1108).

The solution is to use the Requests third-party library, which is not Python3’s built-in URllib. request library, but a powerful urllib3-based third-party library.

pip install requests
Copy the code

Find (class_= ‘result-list’) targets_url = bf.find_all(class_= ‘result-list’)

from bs4 import BeautifulSoup
import requests
                                                        
url='https://topic.autohome.com.cn/new/home/list.jsp?typeId=3' 
headers = {'User-Agent': 'the Mozilla / 5.0 (Windows NT 10.0; Win64; X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36'} 
req = requests.get(url = url,headers = headers)  
                       
req.encoding = 'utf-8'                                                  
html = req.text                                                         
bf = BeautifulSoup(html, 'html.parser') 
targets_url = bf.find(class_='result-list')                
url_set = set(a)for each in targets_url.find_all('a') :if 'class' not in each.attrs.keys():  
		url_set.add(each['href']) 
Copy the code
print(url_set)
Copy the code
{'http://topic.autohome.com.cn/new/marketing/2019/12/jetour/'.'https://topic.autohome.com.cn/act/marketing/2019/12/escape/'.'https://topic.autohome.com.cn/act/marketing/2019/12/mustang/'.'https://topic.autohome.com.cn/new/marketing/2019/11/kx3/'.'https://topic.autohome.com.cn/new/marketing/2019/11/tengshi/'.'https://topic.autohome.com.cn/new/marketing/2019/11/wmex5/'.'https://topic.autohome.com.cn/new/marketing/2019/12/compass/'.'https://topic.autohome.com.cn/new/marketing/2019/12/jkdzddg/'.'https://topic.autohome.com.cn/new/marketing/2020/1/corsair/'.'https://topic.autohome.com.cn/new/marketing/2020/3/xingyue/'.'https://topic.autohome.com.cn/new/marketing/2020/4/t77pro/'.'https://topic.autohome.com.cn/new/marketing/2020/4/xiaopeng/'}
Copy the code

Follow wechat official account: “Notes of Data Analyst”