Return request.get to scrapy.response

In response to a subclass

TextResponse object

The TextResponse object adds encoding capabilities to the base Response class, which is meant to be used only for binary data, such as images, sounds, or any media file.

Parameters:

  • Encoding (String) – Is a string containing the encoding used for this response. If you create a TextResponse object with a Unicode body, it will be encoded using that encoding (remember that the body property is always a string). If encoding is None (the default), the encoding is looked up in the response header and body. TextResponse In addition to standard objects, objects also support the following property Response
  • Text – Response body, such as Unicode.
  • Body_as_unicode () – Same text, but available as a method. Retain this method for backward compatibility; Please enjoy the response. The text

HtmlResponse object

A subclass of this HtmlResponse class, TextResponse this adds support for META HTTP-equiv property by automatically finding by viewing HTML encoding. See TextResponse. Encoding.

XmlResponse object

A subclass of this XmlResponse class, TextResponse, this adds automatic discovery support by viewing the XML declaration line encoding. See TextResponse. Encoding

The sample code

url = 'https://www.baidu.com/"
res = requests.get(url,headers=self.headers)
response=HtmlResponse(url=page_url,body=res.text,encoding='utf-8')
Copy the code

So for response, you can use selector[xpath, CSS]