preface

Recently I have been working on monitoring related infrastructure and found that many scripts are based on Python. I’ve heard about it a long time ago. Life is short. I learned Python, and that’s not a joke. With the rise of artificial intelligence, machine learning and deep learning, most of the artificial intelligence codes on the market are written in Python. So in the age of artificial intelligence, it’s time to learn some Python.

Enter the guide

For students without any language development experience, it is recommended to start from the beginning, whether it is a book, video or text tutorial.

If you have experience in other languages, you are advised to start with a case study, such as crawling through a map of a website.

Because you can think about the language, the syntax and so on and you can read the code pretty much as long as you have a sense of the language.

It’s not recommended for experienced developers to start from scratch, whether it’s a video or a book, because it’s just too time-consuming to start learning a language.

Of course, when you go deep into, or to learn systematically, this is later.

Software tools

Python3

The selected version is Python 3.7.1, the latest version

Installation tutorial recommended:

http://www.runoob.com/python3/python3-install.html

Win download address:

https://www.python.org/downloads/windows

Linux Download address:

https://www.python.org/downloads/source

PyCharm

Visual development tools:

http://www.jetbrains.com/pycharm

case

Implementation steps

Take the girl picture for example, it’s actually very simple, divided into the following four steps:

  • Get the page number of the home page and create folders corresponding to the page number

  • Gets the column address of the page

  • Enter the column and obtain the page number of the column (there are multiple pictures under each column, which are displayed in pages)

  • Get the image in the TAB below and download it

Matters needing attention

There are a few other things to note during the crawl process that may help you:

1) Guide library, actually similar to the Framework or tool class in Java, the bottom layer is encapsulated

Installing third-party Libraries


     
    
      
  1. # python3 installed directly under Win
    2. 
      
  2. PIP install requests PIP install requests
    3. 
      
  3. # Linux python2 python3 coexist
    4. 
      
  4. Pip3 install BS4, pip3 Install Requests
    5. 
     
Copy the code

Importing third-party Libraries


     
    
      
  1. # import requests library
    2. 
      
  2. import requests
    3. 
      
  3. # Import file operation library
    4. 
      
  4. import os
    5. 
      
  5. # bS4 BeautifulSoup is one of the most popular libraries for writing Python crawlers that parse HTML tags.
    6. 
      
  6. import bs4
    7. 
      
  7. from bs4 import BeautifulSoup
    8. 
      
  8. Base class library
    9. 
      
  9. import sys
    10. 
      
  10. # Python 3.x resolves the Chinese encoding problem
    11. 
      
  11. import importlib
    12. 
      
  12. importlib.reload(sys)
    13. 
     
Copy the code

2) Define method functions. A crawler may have hundreds of lines, so try not to write it as a lump


     
    
      
  1. def download(page_no, file_path):
    2. 
      
  2. Write the code logic here
    3. 
     
Copy the code

3) Define global variables


     
    
      
  1. Assign a request header to the request to mimic chrome
    2. 
      
  2. Global headers # tells the compiler that this is the global headers variable
    3. 
      
  3. Headers = {' user-agent ': 'Mozilla/5.0 (Windows NT 10.0; Win64; X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36'}
    4. 
      
    5. 
      
  5. Required before use within a function
    6. 
      
  6. Tell the compiler that the a I use in this method is the global headers variable I just defined, not a local variable inside the method.
    7. 
      
  7. global headers
    8. 
     
Copy the code

4) Anti-theft chain

Some sites have added anti-theft, do-it-all Python solutions


     
    
      
  1. headers = {'Referer': href}
    2. 
      
  2. img = requests.get(url, headers=headers)
    3. 
     
Copy the code

5) Switch the version

Linux server uses Ali cloud server, default version PYTHon2, Python3 installed by itself


     
    
      
  1. [root@AY140216131049Z mzitu]# python2 -V
    2. 
      
  2. Python 2.7.5
    3. 
      
  3. [root@AY140216131049Z mzitu]# python3 -V
    4. 
      
  4. Python 3.7.1
    5. 
      
  5. # Default version
    6. 
      
  6. [root@AY140216131049Z mzitu]# python -V
    7. 
      
  7. Python 2.7.5
    8. 
      
  8. # temporarily switch version <whereis python>
    9. 
      
  9. [root @ AY140216131049Z mzitu] # alias python = '/ usr/local/bin/python3.7'
    10. 
      
  10. [root@AY140216131049Z mzitu]# python -V
    11. 
      
  11. Python 3.7.1
    12. 
     
Copy the code

6) Exception capture

There may be abnormal pages in the process of crawling, so we will capture them without affecting subsequent operations


     
    
      
  1. try:
    2. 
      
  2. # Business logic
    3. 
      
  3. except Exception as e:
    4. 
      
  4.    print(e)
    5. 
     
Copy the code

Code implementation

Edit the script: vi mzitu.py


     
    
      
  1. #coding=utf-8
    2. 
      
  2. #! /usr/bin/python
    3. 
      
  3. # import requests library
    4. 
      
  4. import requests
    5. 
      
  5. # Import file operation library
    6. 
      
  6. import os
    7. 
      
  7. import bs4
    8. 
      
  8. from bs4 import BeautifulSoup
    9. 
      
  9. import sys
    10. 
      
  10. import importlib
    11. 
      
  11. importlib.reload(sys)
    12. 
      
    13. 
      
    14. 
      
  14. Assign a request header to the request to mimic chrome
    15. 
      
  15. global headers
    16. 
      
  16. Headers = {' user-agent ': 'Mozilla/5.0 (Windows NT 10.0; Win64; X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36'}
    17. 
      
  17. # map address
    18. 
      
  18. mziTu = 'http://www.mzitu.com/'
    19. 
      
  19. Define the storage location
    20. 
      
  20. global save_path
    21. 
      
  21. save_path = '/mnt/data/mzitu'
    22. 
      
    23. 
      
    24. 
      
  24. Create folder
    25. 
      
  25. def createFile(file_path):
    26. 
      
  26.     if os.path.exists(file_path) is False:
    27. 
      
  27.         os.makedirs(file_path)
    28. 
      
  28. Change the path to the folder created above
    29. 
      
  29.     os.chdir(file_path)
    30. 
      
    31. 
      
    32. 
      
  32. # Download file
    33. 
      
  33. def download(page_no, file_path):
    34. 
      
  34.     global headers
    35. 
      
  35.     res_sub = requests.get(page_no, headers=headers)
    36. 
      
  36. # parse HTML
    37. 
      
  37.     soup_sub = BeautifulSoup(res_sub.text, 'html.parser')
    38. 
      
  38. Get the column address of the page
    39. 
      
  39.     all_a = soup_sub.find('div',class_='postlist').find_all('a',target='_blank')
    40. 
      
  40.     count = 0
    41. 
      
  41.     for a in all_a:
    42. 
      
  42.         count = count + 1
    43. 
      
  43.         if (count % 2) == 0:
    44. 
      
  44. Print ("内 内页 : "+ STR (count))
    45. 
      
  45. # extract href
    46. 
      
  46.             href = a.attrs['href']
    47. 
      
  47. Print (" + href ")
    48. 
      
  48.             res_sub_1 = requests.get(href, headers=headers)
    49. 
      
  49.             soup_sub_1 = BeautifulSoup(res_sub_1.text, 'html.parser')
    50. 
      
  50. # ------ It is best to use exception handling ------ here
    51. 
      
  51.             try:
    52. 
      
  52. Get the maximum number of sets
    53. 
      
  53.                 pic_max = soup_sub_1.find('div',class_='pagenavi').find_all('span')[6].text
    54. 
      
  54. Print (" pic_max ")
    55. 
      
  55.                 for j in range(1, int(pic_max) + 1):
    56. 
      
  56. # print(" + STR (j) ")
    57. 
      
  57. # j int needs to be converted to a string
    58. 
      
  58.                     href_sub = href + "/" + str(j)
    59. 
      
  59.                     print(href_sub)
    60. 
      
  60.                     res_sub_2 = requests.get(href_sub, headers=headers)
    61. 
      
  61.                     soup_sub_2 = BeautifulSoup(res_sub_2.text, "html.parser")
    62. 
      
  62.                     img = soup_sub_2.find('div', class_='main-image').find('img')
    63. 
      
  63.                     if isinstance(img, bs4.element.Tag):
    64. 
      
  64. # extract SRC
    65. 
      
  65.                         url = img.attrs['src']
    66. 
      
  66.                         array = url.split('/')
    67. 
      
  67.                         file_name = array[len(array)-1]
    68. 
      
  68.                         # print(file_name)
    69. 
      
  69. Add Referer
    70. 
      
  70.                         headers = {'Referer': href}
    71. 
      
  71.                         img = requests.get(url, headers=headers)
    72. 
      
  72. # print(' Start saving images ')
    73. 
      
  73.                         f = open(file_name, 'ab')
    74. 
      
  74.                         f.write(img.content)
    75. 
      
  75. # print(file_name, 'image saved! ')
    76. 
      
  76.                         f.close()
    77. 
      
  77.             except Exception as e:
    78. 
      
  78.                 print(e)
    79. 
      
    80. 
      
    81. 
      
  81. # main method
    82. 
      
  82. def main():
    83. 
      
  83.     res = requests.get(mziTu, headers=headers)
    84. 
      
  84. # Use HTML. Parser
    85. 
      
  85.     soup = BeautifulSoup(res.text, 'html.parser')
    86. 
      
  86. Create folder
    87. 
      
  87.     createFile(save_path)
    88. 
      
  88. Get the total number of pages on the first page
    89. 
      
  89.     img_max = soup.find('div', class_='nav-links').find_all('a')[3].text
    90. 
      
  90. # print(" total pages :"+img_max)
    91. 
      
  91.     for i in range(1, int(img_max) + 1):
    92. 
      
  92. Get the URL for each page
    93. 
      
  93.         if i == 1:
    94. 
      
  94.             page = mziTu
    95. 
      
  95.         else:
    96. 
      
  96.             page = mziTu + 'page/' + str(i)
    97. 
      
  97.         file = save_path + '/' + str(i)
    98. 
      
  98.         createFile(file)
    99. 
      
  99. # Download images for each page
    100. 
      
  100. Print (" + page ")
    101. 
      
  101.         download(page, file)
    102. 
      
    103. 
      
  103. 
      
  104. if __name__ == '__main__':    main()
    

    105. 



    Copy the code


Run the following command to run the script on a Linux server



     
    
      
  1. python 3 mzitu.py 
    2. 
      
  2. # or background execution
    3. 
      
  3. nohup python3 -u mzitu.py > mzitu.log 2>&1 &
    4. 
     
Copy the code


At present, only one column of the set of pictures, a total of 17G, 5332 pictures.



     
    
      
  1. [root@itstyle mzitu]# du -sh 
    2. 
      
  2. 17G     .
    3. 
      
  3. [root@itstyle mzitu]# ll -s
    4. 
      
  4. total 5332
    5. 
     
Copy the code


Next, please open your eyes, chicken frozen hearts of the set of time to come.






summary


As a beginner, scripts are bound to have some problems or need to be optimized. If you encounter Python aunts, please feel free to comment.


In fact, the script is very simple, from configuring the environment, installing the INTEGRATED development environment, writing the script to executing the whole script smoothly, it took about four or five hours, and finally the script was executed in a single track. Limited by the server bandwidth and configuration, it took about three or four hours to download the 17GB image. As for the rest 83G, you can download it by yourself.