This is the 18th day of my participation in the August Challenge
In the previous article we said that Python could not take full advantage of multithreading to achieve high concurrency due to GIL locks, and that multithreading might be less efficient than single-threading in some cases, hence the introduction of coroutines in Python. Coroutines (also known as microthreads) are lightweight threads that can be paused or resumed at a specific point in a function. At the same time, the caller can obtain or pass state from the coroutine to the coroutine. Process and thread are through the CPU scheduling to achieve the orderly execution of different tasks, and the coroutine is controlled by the user program scheduling, there is no overhead of thread switching, so the execution efficiency is very high.
Generator implementation
Earlier coroutines were implemented using the generator keyword yield, much like generators. Let’s use a producer-consumer model to show you how to switch tasks using yield coroutines.
import time
def consumer() :
r = None
while True:
n = yield r
if not n:
return
print("[consumers] consume '{}'...".format(n))
time.sleep(1)
r = 'Consumable'
def producer(c) :
next(c)
n = 0
while n < 5:
n = n + 1
print("[producer] produces '{}'...".format(n))
res = c.send(n)
print("[producer] consumer returns '{}'".format(res))
c.close()
if __name__ == '__main__':
c = consumer()
producer(c)
Copy the codeThe result is as follows:
- Pass consumer into the producer function and call next(c) to start the producer.
- The producer producer function produces n and switches to consumer via c. end(n).
- The consumer function yields the message n produced by the producer, and returns the result R by yield.
- The producer receives the result of the Consumer processing and continues producing the next message until production is complete. The consumer is shut down by c. Close () and the whole process is complete.
The whole process of this code is completed in one thread. The producer and the consumer cooperate to complete the task. After the producer produces the message, it directly jumps to the consumer to start execution through yield.
greenlet
If you have hundreds of tasks, switching between tasks is cumbersome using yield generators, which the Greenlet module can easily implement. Greenlet is a C extension of Python designed to provide self-scheduled “microthreads”, or coroutines. In the Greenlet module, switching to the specified coroutine is made easier by target.switch().
The greenlet module can be installed using the command PIP install greenlet.
from greenlet import greenlet
import time
def work1() :
while True:
print("Work1 starts execution...")
g2.switch() Switch to run in G2
time.sleep(0.5)
def work2() :
while True:
print("Work2 start execution...")
g1.switch() Switch to RUNNING g1
time.sleep(0.5)
if __name__ == "__main__":
# Define the greenlet object
g1 = greenlet(work1)
g2 = greenlet(work2)
g1.switch() Switch to RUNNING g1
Copy the codeThe result is that work1 and work2 are executed alternately.
gevent
Although the Greenlet module realizes the coroutine and can easily switch tasks, it still needs manual switching rather than automatic task switching. When a task is executed, if IO(such as network, file operation, etc.) is encountered, it will be blocked, and the problem of IO automatic switching to improve efficiency is not solved. In fact, Python also has a more powerful coroutine module gEvent, which is also based on Greenlet and can realize automatic task switching. When a Greenlet encounters AN IO operation, it will automatically switch to another Greenlet. When the IO operation is completed, Switch back to continue at an appropriate time. In this way, the IO operation time is skipped instead of waiting for the I/O to complete, which improves the application efficiency.
The gevent module can be installed using the command PIP install gevent.
import gevent
import time
def work1() :
for i in range(5) :print("Work1 starts execution...", gevent.getcurrent())
time.sleep(0.5)
def work2() :
for i in range(5) :print("Work2 start execution...", gevent.getcurrent())
time.sleep(0.5)
if __name__ == "__main__":
g1 = gevent.spawn(work1)
g2 = gevent.spawn(work2)
Wait for the coroutine to complete before closing the main thread
g1.join()
g2.join()
Copy the codeThe result is as follows:
import gevent
import time
def work1() :
for i in range(5) :print("Work1 starts execution...", gevent.getcurrent())
gevent.sleep(0.5)
def work2() :
for i in range(5) :print("Work2 start execution...", gevent.getcurrent())
gevent.sleep(0.5)
if __name__ == "__main__":
g1 = gevent.spawn(work1)
g2 = gevent.spawn(work2)
Wait for the coroutine to complete before closing the main thread
g1.join()
g2.join()
Copy the codeThe running results are as follows:
The monkey patch
Sleep (), work1 and work2 can be executed alternately. Is there a way to implement work1 without writing it? The answer is yes, monkey patches.
About monkey patch: The term comes from the Zope framework, where Bug fixes are often added at the end of a program. These are called “Guerilla patches.” Guerilla eventually became Gorllia, Then it became monkey, so it’s now called monkey patch.
Monkey patches are mainly used for the following purposes:
- Replace methods, properties, and so on at run time
- Add functionality not previously supported without modifying third party code
- Add patches to objects in memory at run time rather than in disk source code
Monkey patches can be made using the following code:
import gevent
# Patch gEvent to identify time-consuming operations it provides or requests from the network
from gevent import monkey
monkey.patch_all()
import time
def work1() :
for i in range(5) :print("Work1 starts execution...", gevent.getcurrent())
time.sleep(0.5)
def work2() :
for i in range(5) :print("Work2 start execution...", gevent.getcurrent())
time.sleep(0.5)
if __name__ == "__main__":
g1 = gevent.spawn(work1)
g2 = gevent.spawn(work2)
Wait for the coroutine to complete before closing the main thread
g1.join()
g2.join()
Copy the codeThe result is as follows:
Async /await asynchronous coroutine
Prior to Python 2 and Python 3.3, using coroutines was implemented based on third-party libraries such as Greenlet or GEvent, which may have a performance penalty because it was not packaged natively in Python. But in python3.4, the introduction of the standard library asyncio, directly built-in support for asynchronous IO, can be very good support coroutines. You can use the @asyncio.coroutine provided by the Asyncio library to mark a generator function as a Coroutine type, and then use yield from to call another coroutine within the coroutine for asynchronous operations.
In order to simplify and better identify asynchronous IO, Python3.5 introduced new syntax async and await, replacing @asyncio. Coroutine of asyncio library with async and yield from with await. Make Coroutine code cleaner and easier to read.
The async keyword is used to declare a function as an asynchronous function. Asynchronous functions can be suspended during the execution of a function, go to execute other asynchronous functions, and come back to continue the execution after the suspension condition disappears.
The await keyword is used to implement the task suspension operation, for example, an asynchronous task will be suspended to execute other asynchronous programs when it needs a long time to perform a certain step. Note: Await can only be followed by asynchronous programs or objects with an __await__ attribute.
Suppose there are two asynchronous functions async work1 and async work2, and one step in work1 has await. After the program gets await work2() keyword, the asynchronous program suspends and executes another asynchronous work2 function. When the suspension condition disappears, no matter whether work2 is completed or not, Immediately go back to the work1 function from work2 and continue with the original operation.
import asyncio
import datetime
async def work1(i) :
print("Work1 '{}' in execution......".format(i))
res = await work2(i)
print("Work1 '{}' executed......".format(i), datetime.datetime.now())
print("Receive from work2'{}' :", res)
async def work2(i) :
print("Work2 '{}' in execution......".format(i))
await asyncio.sleep(1.5)
print("Work2 '{}' executed......".format(i), datetime.datetime.now())
return "Work2 '{}' return".format(i)
loop = asyncio.get_event_loop() Create an event loop
task = [asyncio.ensure_future(work1(i)) for i in range(5)] Create a list of tasks
time1 = datetime.datetime.now()
Register the task into the event loop
loop.run_until_complete(asyncio.wait(task))
time2 = datetime.datetime.now()
print("Total time:", time2 - time1)
loop.close()
Copy the codeThe running results are as follows:
conclusion
At this point, the use of coroutines in Python is summarized. In normal development, the most common way to write coroutines is to write async/await or the third party library gEvent module. Yield is used only on the writer. As for asynchronous coroutines, Python is getting better and better at supporting and packaging them, and the syntax is very simple to use, but it’s not easy to learn their great ideas through the syntax.
Finally, thank my girlfriend for her tolerance, understanding and support in work and life!
