Python is a language that runs in the interpreter. There is a global lock (GIL) in Python that does not take advantage of multicore in the case of threads. With Multiprocess, you can take advantage of multi-core to really improve efficiency.

Contrast experiment

The data shows that if the process of multithreading is CPU-intensive, it can not improve the efficiency of multithreading. On the contrary, it may reduce the efficiency due to frequent switching of threads, so it is recommended to use multi-process. If it is IO intensive, multithreaded processes can use the idle time while WAITING for THE I/O block to execute other threads, increasing efficiency. So we compare the efficiency of different scenarios according to the experiment

The operating system CPU memory The hard disk
Windows 10 Dual – 8GB Mechanical drive

(1) Introduce the required modules
“`python

import requests import time from threading import Thread from multiprocessing import Process python def count(x, y): While c < 500000: c += 1 x += x y += y

###### (2) defines the CPU-intensive calculation function ###### [](http://blog.atomicer.cn/2016/09/30/Python%E4%B8%AD%E5%A4%9A%E7%BA%BF%E7%A8%8B%E5%92%8C%E5%A4%9A%E8%BF%9B%E7%A8%8B%E7% 9A%84% e5% AF% b9% e6% AF%94/# (3) Define IO - intensive file read/write functions "(3) Define IO - intensive file read/write functions ") (3) Define IO - intensive file read/write functions ###### [](http://blog.atomicer.cn/2016/09/30/Python%E4%B8%AD%E5%A4%9A%E7%BA%BF%E7%A8%8B%E5%92%8C%E5%A4%9A%E8%BF%9B%E7%A8%8B%E7% Python def write(): python def write(): f = open("test.txt", "w") for x in range(5000000): f.write("testwrite\n") f.close() def read(): f = open("test.txt", "r") lines = f.readlines() f.close() ``` ```python _head = { 'User-Agent': 'the Mozilla / 5.0 (Windows NT 10.0; WOW64) AppleWebKit / 537.36 (KHTML, Def http_request(): try: def http_request() = "http://www.tieba.com" webPage = requests.get(url, headers=_head) html = webPage.text return {"context": html} except Exception as e: return {"error": e}Copy the code
(4) Define the network request function
(5) Test the time required by linear IO intensive operation, CPU intensive operation and network request intensive operation


\

CPU intensive operation
t = time.time()
for x in range(10):
    count(1.1)
print("Line cpu", time.time() - t)
IO intensive operation
t = time.time()
for x in range(10):
    write()
    read()
print("Line IO", time.time() - t)
Network request intensive operations
t = time.time()
for x in range(10):
    http_request()
print("Line Http Request", time.time() - t)
Copy the code


\

(6) Test the output of time required by multiple threads to execute CPU-intensive operations concurrently

  • CPU intensive: 95.6059999466, 91.57099986076355 92.52800011634827, 99.96799993515015
  • IO density: 24.25, 21.76699995994568, 21.769999980926514, 22.060999870300293
  • Network request intensive: 4.519999980926514, 8.563999891281128, 4.371000051498413, 4.522000074386597, 14.671000003814697

\

counts = []
t = time.time()
for x in range(10):
    thread = Thread(target=count, args=(1.1))
    counts.append(thread)
    thread.start()
e = counts.__len__()
while True:
    for th in counts:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print(time.time() - t)
Copy the code

Output: 99.9240000248, 101.26400017738342, 102.32200002670288\

(7) Test the time required by multiple threads to execute IO intensive operations concurrently

def io() :
    write()
    read()
t = time.time()
ios = []
t = time.time()
for x in range(10):
    thread = Thread(target=count, args=(1.1))
    ios.append(thread)
    thread.start()
e = ios.__len__()
while True:
    for th in ios:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print(time.time() - t)
Copy the code




\

Output: 25.69700002670288, 24.02400016784668

(8) Test the time required by multiple threads to execute network intensive operations concurrently
t = time.time()
ios = []
t = time.time()
for x in range(10):
    thread = Thread(target=http_request)
    ios.append(thread)
    thread.start()
e = ios.__len__()
while True:
    for th in ios:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print("Thread Http Request", time.time() - t)
Copy the code




\

Output: 0.7419998645782471, 0.3839998245239258, 0.3900001049041748

(9) Test the time required by multiple processes to execute CPU-intensive operations concurrently
counts = []
t = time.time()
for x in range(10):
    process = Process(target=count, args=(1.1))
    counts.append(process)
    process.start()
e = counts.__len__()
while True:
    for th in counts:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print("Multiprocess cpu", time.time() - t)
Copy the code




\

Output: 54.342000007629395, 53.437999963760376

(10) Test the concurrent execution of IO intensive operations by multiple processes
t = time.time()
ios = []
t = time.time()
for x in range(10):
    process = Process(target=io)
    ios.append(process)
    process.start()
e = ios.__len__()
while True:
    for th in ios:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print("Multiprocess IO", time.time() - t)
Copy the code

\

\

\

Output: 12.509000062942505, 13.059000015258789

(11) Test multiple processes to execute Http request-intensive operations concurrently
“`python

t = time.time() httprs = [] t = time.time() for x in range(10): process = Process(target=http_request) ios.append(process) process.start() e = httprs.len() while True: for th in httprs: if not th.is_alive(): e -= 1 if e <= 0: break print(“Multiprocess Http Request”, time.time() – t)

###### > Output: 0.5329999923706055, 0.4760000705718994 *** ##### [](http://blog.atomicer.cn/2016/09/30/Python%E4%B8%AD%E5%A4%9A%E7%BA%BF%E7%A8%8B%E5%92%8C%E5%A4%9A%E8%BF%9B%E7%A8%8B%E7% 9 a E6 B9 AF E5% % % % % 84% AF # % 94 / results "results") experimental results | | | | IO intensive operation cpu-intensive operation network request intensive operation | | -- - | -- -- -- -- -- -- -- -- -- -- -- -- -- - | -- -- -- -- -- -- -- -- -- -- -- -- -- -- | -- -- -- -- -- -- -- -- -- -- -- - | | | | | | 7.3296000004 22.46199995279 94.91824996469 linear operation | multithreaded operations | | | 24.8605000973 101.1700000762 0.5053332647 | | multi-process operating | | | | 0.5045000315 12.7840000391 53.8899999857 through the above results, we can see: \ \> * Multithreading also does not seem to have a great advantage in THE IO intensive operation (perhaps the IO operation is more heavy task can be demonstrated by the advantage), in the CPU intensive operation performance is significantly worse than single-thread linear execution, but for network requests such as busy and blocked thread operations, The advantages of multi-threading are significant * Multi-processes can provide performance advantages in CPU intensive, IO intensive, and network request intensive (frequently thread blocking operations). However, in network request-intensive operations, similar to multithreading, but more CPU and other resources, so in this case, We can choose multithreading to execute \ [! [multithreaded effect] (HTTP: / / https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/21f95e04eac74fa186d32c62c38d3f82~tplv-k3u1fbpfcp-zoom-1.ima ge)](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/21f95e04eac74fa186d32c62c38d3f82~tplv-k3u1fbpfcp-zoom-1.image) [the original link here] (HTTP: / / http://blog.atomicer.cn/2016/09/30/Python%E4%B8%AD%E5%A4%9A%E7%BA%BF%E7%A8%8B%E5%92%8C%E5%A4%9A%E8%BF%9B%E7%A8 %8B%E7%9A%84%E5%AF%B9%E6%AF%94/)\Copy the code