grequests

GRequests: Asynchronous Requests Made Simple

GRequests is a powerful tool that extends the functionality of the popular Requests library by integrating it with Gevent, enabling users to make asynchronous HTTP requests effortlessly. With GRequests, developers can handle multiple HTTP requests simultaneously, significantly improving the efficiency of web scraping, data collection, and other network-related tasks.

Installation

Getting started with GRequests is straightforward. Installation is easily done via pip:

$ pip install grequests

Basic Usage

Using GRequests is as simple as it gets. It allows you to send HTTP requests in parallel with minimal effort. Here's a quick guide on how to utilize GRequests for asynchronous communication.

Start by importing the library and listing the URLs you want to request from:

import grequests

urls = [
    'http://www.heroku.com',
    'http://python-tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://fakedomain/',
    'http://kennethreitz.com'
]

Create a set of unsent requests using GRequests:

rs = (grequests.get(u) for u in urls)

Then, send all these requests simultaneously using the map function:

grequests.map(rs)

This will return the responses from the functioning URLs while handling failed requests gracefully.

Error Handling

Handling errors and exceptions in GRequests is quite straightforward. Users can define custom exception handlers to manage issues like timeouts or connection errors.

Here’s an example of how you can set up an exception handler:

def exception_handler(request, exception):
    print("Request failed")

reqs = [
    grequests.get('http://httpbin.org/delay/1', timeout=0.001),
    grequests.get('http://fakedomain/'),
    grequests.get('http://httpbin.org/status/500'),
]

grequests.map(reqs, exception_handler=exception_handler)

Performance Optimization with `imap`

For users looking to optimize speed and performance further, GRequests provides the imap function. Unlike map, which returns a list of responses, imap yields a generator of responses, offering flexibility in handling responses as they arrive.

for resp in grequests.imap(reqs, size=10):
    print(resp)

imap_enumerated is another useful variant that provides both the index and response of each request, maintaining clarity and control over the requests and their corresponding responses.

rs = [grequests.get(f'https://httpbin.org/status/{code}') for code in range(200, 206)]
for index, response in grequests.imap_enumerated(rs, size=5):
    print(index, response)

Important Notes on Integration with Gevent

Since GRequests relies on Gevent, which uses monkey patching for enabling concurrency, it's crucial to ensure that GRequests is imported before any other libraries, especially Requests. This order of importation helps avoid potential conflicts and issues.

# Correct Order
import grequests
import requests

# Incorrect Order
import requests
import grequests

For those interested in understanding more about such issues, you can check out the ongoing discussions and solutions on the GRequests GitHub page.