Introduction
In this tutorial we will use python requests module with max retries and timeout values. It is strongly recommended to implement retries and timeout mechanism for production code as connections can close at any time, even in non-error conditions. HTTP applications have to be ready to properly handle unexpected closes. If a transport connection closes while the client is performing a transaction, the client should reopen the connection and retry one time, unless the transaction has side effects.
1. Set Max Retries Values
We will create python session and define our retry strategy for that. Our retry strategy will consists of connect retries, read retries, backoff factor (delay between attempts after second try), and status_forcelist (list of status, against which we want to retry).
Use following code to create request session, and setup retry strategy.
# import request module
import requests
# define some values of retries. You can update it according to your requirements.
MAX_RETRIES = 3
BACKOFF = 1
STATUS_FORCELIST = [
413, # Payload Too Large
429, # Too Many Requests
500, # Internal Server Error
502, # Bad Gateway
503, # Service Unavailable
504 # Gateway Timeout
]
# Set both connect and read timeout. (we can set only one value to represent both.)
REQUEST_TIMEOUT = (3.05, 10) # (connect, read)
# create request session
session = requests.Session()
# set some default values in session header, these values will be merged with request header, if we want to send some additional headers at runtime.
session.headers.update({'Connection': 'close'})
# define retry strategy, and set required values
retry = Retry(
total=MAX_RETRIES, # Total number of retries to allow. Takes precedence over other counts.
read=MAX_RETRIES, # How many connection-related errors to retry on.
connect=MAX_RETRIES, # How many times to retry on read errors.
backoff_factor=BACKOFF, # A backoff factor to apply between attempts after the second try
status_forcelist=STATUS_FORCELIST,
)
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)
At this point, we have session ready with required values of retries, back off factor, and status force list. We will use this session to send API requests.
2. Set Timeout Value
Use following code to send API requests with required value of request timeout we have set above
response = session.get('httpbin.org/delay/10', timeout=REQUEST_TIMEOUT)
This request will attempt retries against given list of status force list before raising any exception.
Conclusion
We have configured both timeout and max retries of python request module. Make sure to set some appropriate value of timeout on production, by default value of timeout is None which means it never expire which can cause your production code to hang for too long.
