How to Set Timeout and Max Retries in Python Request Module

Introduction

In this tutorial we will use python requests module with max retries and timeout values. It is strongly recommended to implement retries and timeout mechanism for production code as connections can close at any time, even in non-error conditions. HTTP applications have to be ready to properly handle unexpected closes. If a transport connection closes while the client is performing a transaction, the client should reopen the connection and retry one time, unless the transaction has side effects. 

1. Set Max Retries Values

We will create python session and define our retry strategy for that. Our retry strategy will consists of connect retries, read retries, backoff factor (delay between attempts after second try), and status_forcelist (list of status, against which we want to retry).

Use following code to create request session, and setup retry strategy.

# import request module

import requests

# define some values of retries. You can update it according to your requirements.

MAX_RETRIES = 3

BACKOFF = 1

STATUS_FORCELIST = [

    413,  # Payload Too Large

    429,  # Too Many Requests

    500,  # Internal Server Error

    502,  # Bad Gateway

    503,  # Service Unavailable

    504  # Gateway Timeout

]

# Set both connect and read timeout. (we can set only one value to represent both.)

REQUEST_TIMEOUT = (3.05, 10)  # (connect, read)

# create request session

session = requests.Session()

# set some default values in session header, these values will be merged with request header, if we want to send some additional headers at runtime.

session.headers.update({'Connection': 'close'})

# define retry strategy, and set required values

retry = Retry(

total=MAX_RETRIES,  # Total number of retries to allow. Takes precedence over other counts.

read=MAX_RETRIES,  # How many connection-related errors to retry on.

connect=MAX_RETRIES,  # How many times to retry on read errors.

backoff_factor=BACKOFF,  # A backoff factor to apply between attempts after the second try

status_forcelist=STATUS_FORCELIST,

)

adapter = HTTPAdapter(max_retries=retry)

session.mount('http://', adapter)

session.mount('https://', adapter)

At this point, we have session ready with required values of retries, back off factor, and status force list. We will use this session to send API requests.

2. Set Timeout Value

Use following code to send API requests with required value of request timeout we have set above

response = session.get('httpbin.org/delay/10', timeout=REQUEST_TIMEOUT)

This request will attempt retries against given list of status force list before raising any exception.

Conclusion

We have configured both timeout and max retries of python request module. Make sure to set some appropriate value of timeout on production, by default value of timeout is None which means it never expire which can cause your production code to hang for too long.

See Also