Trip is a network library of coroutines that can stop network delays blocking your application with simple operations like Requests.

Python coroutines can be as simple as this (compatible with Python 2.7-3.7) :

import trip

@trip.coroutine
def main():
    r = yield trip.get('https://httpbin.org/get', auth=('user', 'pass'))
    print(r.content)

trip.run(main)Copy the code

With coroutines, the same amount of code, 100 requests per time.

Trip takes its name from its two dependencies, and also aims to merge the contents of the two: ‘Tornado & Requests In Pair’. A large number of the above two package structures and processing codes were used in compatibility. I just did some simple integration work. Thank Tornado and Requests for making it so easy for me to complete the writing of this project.

With Trip, you can take full advantage of Requests features, including but not limited to: sessions with persistent cookies, browser-style SSL authentication, automatic content decoding, basic/digest authentication, and elegant key/value cookies. At the same time, your request is usually coroutine with Tornado AsyncHTTPClient. Network delay will no longer block your program and you can wait for multiple tasks to be completed at the same time when the program is running normally.

Is crawler taking too long difficult to optimize? Are various coroutine network frameworks difficult to use? Is large crawler framework too bloated to be flexibly customized? Try Trip, you won’t regret it!

Trip works perfectly whether you’re using 2.7, 3.3, or 3.7.

The installation

Setting up Trip is as simple as typing:

python -m pip install tripCopy the code

The document

You can find detailed documentation of the project here.

If you encounter any problems in the process of reading the document, you can also join the QQ group to discuss with us: 462703741.

Advanced applications

Here are some examples of advanced applications:

Use async and await

import trip

async def main():
    r = await trip.get('https://httpbin.org/get', auth=('user', 'pass'))
    print(r.content)

trip.run(main)Copy the code

Persistence of cookies

import trip

@trip.coroutine
def main():
    s = trip.Session()
    r = yield s.get(
        'https://httpbin.org/cookies/set',
        params={'name': 'value'},
        allow_redirects=False)
    r = yield s.get('https://httpbin.org/cookies')
    print(r.content)

trip.run(main)Copy the code

Event hooks

import trip

@trip.coroutine
def main():
    def print_url(r, *args, **kwargs):
        print(r.url)
    def record_hook(r, *args, **kwargs):
        r.hook_called = True
        return r
    url = 'http://httpbin.org/get'
    r = yield trip.get('http://httpbin.org', hooks={'response': [print_url, record_hook]})
    print(r.hook_called)

trip.run(main)Copy the code

timeout

Import trip.coroutine def main(): r = yield trip.get('http://github.com', timeout=0.001) print(r) trip.run(main) import trip.coroutine def main(): r = yield trip.get('http://github.com', timeout=0.001)Copy the code

How to contribute code

  1. You can open issue and exchange your ideas with me.
  2. Or fork the project and make your changes on the Master branch.
  3. Please be sure to bring the relevant code for problems or new features, it will be a great help in our communication.
  4. Finally, if you complete the modification, you can submit it through pull Request. I will complete the test and merge it as soon as possible.