1, iterations,

1.1 Concept of iteration

The process of iterating through values using the for loop is called iteration, for example, the process of iterating through a list to get a value

for value in [2, 3, 4]:
    print(value)
Copy the code

1.2 Iterable

Standard concept: An iterable is an object created using an __iter__ method defined in a class

Simple memory: Objects that use the for loop to iterate over values are called iterables: lists, tuples, dictionaries, collections, ranges, strings

1.3 Determine whether an object is iterable

From collections import Iterable result = isinstance((3, 5), Iterable) print(" Tuples are Iterable :", Result) result = isinstance([3, 5], Iterable) print(" ", result) result = isinstance({"name": Result = isinstance("hello", Iterable) print(" hello", Iterable) print(" Result) result = isinstance({3, 5}, Iterable) print(" ", result) result = isinstance(range(5), Iterable print("range is Iterable :", result) result = isinstance(5, Iterable) print("range is Iterable :", result) Result) result = isinstance(5, int) print(" int ", result) class Student(object): Pass stu = Student() result = isinstance(stu, Iterable) print("stu is Iterable :", result) result = isinstance(stu, Iterable) Student) print("stu = Student :", result)Copy the code

1.4 Customizing iterables

Implement an __iter__ method in a class

Custom iterable type code

from collections import Iterable
class MyList(object):

    def __init__(self):
        self.my_list = list()

    
    def append_item(self, item):
        self.my_list.append(item)

    def __iter__(self):
        
        pass

my_list = MyList()
my_list.append_item(1)
my_list.append_item(2)
result = isinstance(my_list, Iterable)

print(result)

for value in my_list:
    print(value)
Copy the code

Execution Result:

Traceback (most recent call last):
True
  File "/Users/hbin/Desktop/untitled/aa.py", line 24, in <module>
    for value in my_list:
TypeError: iter() returned non-iterator of type 'NoneType'
Copy the code

As you can see from the execution results, iterators are required to walk through the iterable to get data in sequence

conclusion

Providing an __iter__ inside a class creates objects that are iterable. Iterables are objects that require iterators to complete data iteration

iterators

2.1 Custom iterator objects

Custom iterator objects: Objects created by defining __iter__ and __next__ methods in a class are iterator objects

from collections import Iterable from collections import Iterator class MyList(object): def __init__(self): self.my_list = list() def append_item(self, item): self.my_list.append(item) def __iter__(self): my_iterator = MyIterator(self.my_list) return my_iterator class MyIterator(object): def __init__(self, my_list): Self.my_list = my_list self.current_index = 0 result = isinstance(self, Iterator) print(" my.my_list = my_list self.current_index = 0 ", result) def __iter__(self): return self def __next__(self): if self.current_index < len(self.my_list): self.current_index += 1 return self.my_list[self.current_index - 1] else: raise StopIteration my_list = MyList() my_list.append_item(1) my_list.append_item(2) result = isinstance(my_list, Iterable) print(result) for value in my_list: print(value)Copy the code

Running results:

True MyIterator creates an iterator: True 1 2Copy the code

2.2 Iter () and next() functions

Iter: Gets an iterator for an iterable, which calls the __iter__ method on the iterable

Next function: Gets the next value in an iterator, calling the __next__ method on the iterator object

class MyList(object):

    def __init__(self):
        self.my_list = list()
    
    def append_item(self, item):
        self.my_list.append(item)

    def __iter__(self):      
        my_iterator = MyIterator(self.my_list)
        return my_iterator

class MyIterator(object):

    def __init__(self, my_list):
        self.my_list = my_list
        self.current_index = 0

    def __iter__(self):
        return self
    
    def __next__(self):
        if self.current_index < len(self.my_list):
            self.current_index += 1
            return self.my_list[self.current_index - 1]
        else:           
            raise StopIteration

my_list = MyList()
my_list.append_item(1)
my_list.append_item(2)
my_iterator = iter(my_list)
print(my_iterator)

while True:
    try:
        value = next(my_iterator)
        print(value)
    except StopIteration as e:
        break
Copy the code

2.3 The nature of the For loop

Iterable is iterable

The essence of the for item in Iterable loop is to get the iterator of the Iterable through iter() and then call next() to get the next value and assign it to the item. The loop ends when it encounters an exception called StopIteration.

Iterators are traversed

An Iterator for item in Iterator that calls the next() method to get the next value and assign it to the item. The loop ends when it encounters an exception called StopIteration.

2.4 Application scenarios of iterators

We found that the core function of the iterator is to return the next data value by calling the next() function. If every time the data returned value is not read in an existing data set, but is calculated according to certain rules generated by the program, then means can no longer rely on an existing data set, that is to say, don’t put all want to iteration of the data of disposable cached for subsequent read in turn, this can save a lot of space for storage (memory).

For example, in mathematics there is a famous Fibonacci sequence in which the first number is 0, the second number is 1, and each subsequent number can be obtained by adding the first two numbers:

0, 1, 1, 2, 3, 5, 8, 13, 21, 34…

Now we want to pass for… in… Loop to iterate over the first n numbers in the Fibonacci sequence. So this Fibonacci sequence we can implement with iterators, where each iteration is mathematically calculated to produce the next number.

class Fibonacci(object):

    def __init__(self, num):   
        self.num = num   
        self.a = 0
        self.b = 1   
        self.current_index = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.current_index < self.num:
            result = self.a
            self.a, self.b = self.b, self.a + self.b
            self.current_index += 1
            return result
        else:
            raise StopIteration

fib = Fibonacci(5)

for value in fib:
    print(value)
Copy the code

Execution Result:

0, 1, 1, 2, 3Copy the code

summary

The iterator’s job is to record the current position of the data in order to get the value of the next position

3. Generators

3.1 The concept of generators

Generators are a special class of iterators that don’t need to write __iter__() and __next__() as the above classes do, and are much easier to use. They can still use the next function and the for loop

3.2 Create generator method 1

The first method is as simple as changing a list generator [] to ()

my_list = [i * 2 for i in range(5)]
print(my_list)

my_generator = (i * 2 for i in range(5))
print(my_generator)

for value in my_generator:
    print(value)
Copy the code

Execution Result:

[0, 2, 4, 6, 8]
<generator object <genexpr> at 0x101367048>
0
2
4
6
8
Copy the code

3.3 Create generator method 2

If you see the yield keyword in the def function, it’s a generator

def fibonacci(num):
    a = 0
    b = 1
    
    current_index = 0
    print("--11---")
    while current_index < num:
        result = a
        a, b = b, a + b
        current_index += 1
        print("--22---")
        
        yield result
        print("--33---")

fib = fibonacci(5)
value = next(fib)
print(value)
value = next(fib)
print(value)

value = next(fib)
print(value)
Copy the code

In the generator implementation, we implement the basic logic that was originally implemented in the iterator __next__ method in a function, but replace the return for each iteration with yield, and the newly defined function is no longer a function, but a generator.

Simply put: anything with the yield keyword in def is called a generator

3.4 Generator uses the return keyword

def fibonacci(num): a = 0 b = 1 current_index = 0 print("--11---") while current_index < num: result = a a, b = b, A + b current_index += 1 print("--22-- ") yield result print("--33-- ") return "fib = Fibonacci (5) value = next(fib)  print(value) try: value = next(fib) print(value) except StopIteration as e: print(e.value)Copy the code

Tip:

There is no syntax problem with using the return keyword in generators, but the code will stop iterating until the return statement is executed, raising the stop iterating exception.

3.5 Yield and Return

Functions that use the yield keyword are no longer functions, but generators. (A function that uses yield is a generator.)

Code execution is paused at yield, the result is returned, and the next time the generator is started, execution continues at the paused location

The generator returns a value each time it is started, and multiple starts can return multiple values, i.e. yield can return multiple values

A return can only return a value once, and the code will stop iterating until the return statement is executed

3.6 Start the generator and send parameters using the send method

The send method starts the generator by passing parameters

def gen():
    i = 0
    while i<5:
        temp = yield i
        print(temp)
        i+=1
Copy the code

Execution Result:

In [43]: f = gen()

In [44]: next(f)
Out[44]: 0

In [45]: f.send('haha')
haha
Out[45]: 1

In [46]: next(f)
None
Out[46]: 2

In [47]: f.send('haha')
haha
Out[47]: 3

In [48]:
Copy the code

Note: If the generator is started for the first time using the send method, the argument can only be passed as None, which is usually started for the first time using the next function

summary

  • There are two ways to create generators, typically using the yield keyword method
  • The yield feature is that code is paused at yield, the result is returned, and the generator is restarted to continue execution at the paused location

4, coroutines

4.1 Concept of coroutines

A coroutine, also known as a microthread, a fiber, also known as a user-level thread, is a way to multitask without having to open a thread, that is, to multitask with a single thread, so that multiple tasks are executed alternately in a certain order

Coroutines are also a way to implement multitasking

The code implementation of the coroutine yield

Simple implementation of coroutines

Import time def work1(): while True: print("----work1-- ") yield time.sleep(0.5) def work2(): while True: print("----work1-- ") yield time.sleep(0.5) def work2(): while True: Print ("----work2-- ") yield time.sleep(0.5) def main(): w1 = work1() w2 = work2() while True: next(w1) next(w2) if __name__ == "__main__": main()Copy the code

Running results:

----work1--- ----work2--- ----work1--- ----work2--- ----work1--- ----work2--- ----work1--- ----work2--- ----work1--- ----work2--- ----work1--- ----work2--- ... Omit...Copy the code

summary

Tasks between coroutines are executed alternately in a certain order

5, greenlet

5.1 Introduction to Greentlet

To make it easier to multitask with coroutines, the Greenlet module in Python encapsulates them, making switching easier

Install the Greenlet module using the following command:

pip3 install greenlet
Copy the code

Use coroutines to multitask

Import time import greenlet # task 1 def work1(): for I in range(5): print("work1... Switch () # def work2(): for I in range(5): print("work2...") def work2(): print("work2...") ) time.sleep(0.2) # Switch to the first coroutine to execute the corresponding task g1.switch() if __name__ == '__main__': G1 = greenlet.greenlet(work1) g2 = greenlet.greenlet(work2)Copy the code

Running effect

work1...
work2...
work1...
work2...
work1...
work2...
work1...
work2...
work1...
work2...
Copy the code

6, gevent

6.1 Introduction to GEvent

Greenlet already implements coroutines, but this requires manual switching. Here’s a third-party library that is even more powerful than Greenlet and can automatically switch tasks: GEvent.

The greenlet encapsulated inside gEvent is based on the principle that when a Greenlet encounters IO(input/output, such as network or file operation) operations, such as accessing the network, it will automatically switch to another Greenlet. When the IO operation is completed, Switch back to continue at an appropriate time.

Since IO operations are time-consuming and often leave the program in a wait state, having gEvent automatically switch coroutines for us ensures that greenlets are always running instead of waiting for IO

The installation

pip3 install gevent
Copy the code

6.2 Use of GEvent

import gevent

def work(n):
    for i in range(n):
        
        print(gevent.getcurrent(), i)

g1 = gevent.spawn(work, 5)
g2 = gevent.spawn(work, 5)
g3 = gevent.spawn(work, 5)
g1.join()
g2.join()
g3.join()
Copy the code

The results

<Greenlet "Greenlet-0" at 0x26d8c970488: work(5)> 0
<Greenlet "Greenlet-1" at 0x26d8c970598: work(5)> 0
<Greenlet "Greenlet-2" at 0x26d8c9706a8: work(5)> 0
<Greenlet "Greenlet-0" at 0x26d8c970488: work(5)> 1
<Greenlet "Greenlet-1" at 0x26d8c970598: work(5)> 1
<Greenlet "Greenlet-2" at 0x26d8c9706a8: work(5)> 1
<Greenlet "Greenlet-0" at 0x26d8c970488: work(5)> 2
<Greenlet "Greenlet-1" at 0x26d8c970598: work(5)> 2
<Greenlet "Greenlet-2" at 0x26d8c9706a8: work(5)> 2
<Greenlet "Greenlet-0" at 0x26d8c970488: work(5)> 3
<Greenlet "Greenlet-1" at 0x26d8c970598: work(5)> 3
<Greenlet "Greenlet-2" at 0x26d8c9706a8: work(5)> 3
<Greenlet "Greenlet-0" at 0x26d8c970488: work(5)> 4
<Greenlet "Greenlet-1" at 0x26d8c970598: work(5)> 4
<Greenlet "Greenlet-2" at 0x26d8c9706a8: work(5)> 4
Copy the code

As you can see, the three greenlets run sequentially rather than interchangeably

6.3 GEvent Switchover is performed

import gevent

def work(n):
    for i in range(n):
        
        print(gevent.getcurrent(), i)
        
        gevent.sleep(1)

g1 = gevent.spawn(work, 5)
g2 = gevent.spawn(work, 5)
g3 = gevent.spawn(work, 5)
g1.join()
g2.join()
g3.join()
Copy the code

The results

<Greenlet at 0x7fa70ffa1c30: f(5)> 0
<Greenlet at 0x7fa70ffa1870: f(5)> 0
<Greenlet at 0x7fa70ffa1eb0: f(5)> 0
<Greenlet at 0x7fa70ffa1c30: f(5)> 1
<Greenlet at 0x7fa70ffa1870: f(5)> 1
<Greenlet at 0x7fa70ffa1eb0: f(5)> 1
<Greenlet at 0x7fa70ffa1c30: f(5)> 2
<Greenlet at 0x7fa70ffa1870: f(5)> 2
<Greenlet at 0x7fa70ffa1eb0: f(5)> 2
<Greenlet at 0x7fa70ffa1c30: f(5)> 3
<Greenlet at 0x7fa70ffa1870: f(5)> 3
<Greenlet at 0x7fa70ffa1eb0: f(5)> 3
<Greenlet at 0x7fa70ffa1c30: f(5)> 4
<Greenlet at 0x7fa70ffa1870: f(5)> 4
<Greenlet at 0x7fa70ffa1eb0: f(5)> 4
Copy the code

6.4 Installing Patches for Programs

import gevent import time from gevent import monkey monkey.patch_all() def work1(num): for i in range(num): print("work1...." Def work2(num): print("work2....") def work2(num): print("work2....") ) time.sleep(0.2) if __name__ == '__main__': g1 = gevent.spawn(work1, 3) g2 = gevent.spawn(work2, 3) g1.join() g2.join()Copy the code

The results

work1....
work2....
work1....
work2....
work1....
work2....
Copy the code

6.5 pay attention to

Since the program is in an infinite loop and can perform time-consuming operations, there is no need to add the join method, because the program needs to keep running and never exit

The sample code

import gevent import time from gevent import monkey monkey.patch_all() def work1(num): for i in range(num): print("work1...." Def work2(num): print("work2....") def work2(num): print("work2....") ) time.sleep(0.2) if __name__ == '__main__': g1 = gevent. Spawn (work1, 3) g2 = gevent. Spawn (work2, 3) while True: Print (" execute on main thread ") time.sleep(0.5)Copy the code

Execution Result:

Run work1.... in the main thread work2.... work1.... work2.... work1.... work2.... Execution in main thread Execution in main thread execution in main thread.. Omit..Copy the code

If there are too many coroutines in use, you need to block the main thread one by one using join(), which is too redundant. You can use gEvent.joinall () to start the coroutines

The sample code

Import time import gevent def work1(): for I in range(5): print("work1 works {}". Format (I)) gevent. Sleep (1) def work2(): For I in range (5) : print (" work2 work {} ". The format (I)) gevent. Sleep (1) if __name__ = = "__main__ ': w1 = gevent.spawn(work1) w2 = gevent.spawn(work2) gevent.joinall([w1, w2])Copy the code

7. Process, thread, coroutine comparison

7.1 Relationships among processes, threads, and coroutines

  • A process has at least one thread, and a process can have multiple threads
  • There can be multiple coroutines in a thread

7.2 Comparison between Processes, threads, and threads

  1. A process is a unit of resource allocation
  2. Threads are the unit of operating system scheduling
  3. Process switching requires the most resources and is inefficient
  4. Thread switching requires average resources and efficiency (without the GIL, of course)
  5. Coroutine switching task has small resources and high efficiency
  6. Multi-process, multi-thread depending on the number of CPU cores may be parallel, but the coroutine is in one thread, so it is concurrent

summary

1. Processes, threads and coroutines can complete multiple tasks, and can be used according to their actual development needs

2. Threads and coroutines are most likely to be used because they require fewer resources

3. Developing coroutines requires the least resources