HTTP Requests in Python

BE G I NNE R PYT HO N

Introduction

In today’s ever-evolving technological landscape, proficiency in using Python for HTTP requests is a vital
skill. Whether you’re fetching data from an API, working with databases, or sending user data to a server,
effective web communication is paramount. In this article we will use requests module to master the art of
handling HTTP requests with Python.

Enroll in our free course of Python.

Table of contents

Understanding HTTP Requests

Python Requests Module
Handling Response Status Codes
Response Methods
Handling Errors and Exceptions
Advanced Topics
Frequently Asked Questions

Understanding HTTP Requests

HTTP, or Hypertext Transfer Protocol, forms the basis of data communication on the web. It involves the
exchange of information between a client and a server, with HTTP requests playing a pivotal role in this
interaction. These requests enable clients to seek resources, and servers respond accordingly.
 HTTP
Description
Method
GET GET method is used to retrieve information from the given server using a given URI.
POST request method requests that a web server accepts the data enclosed in the body of the request message, most
POST
likely for storing it
The PUT method requests that the enclosed entity be stored under the supplied URI. If the URI refers to an already
PUT existing resource, it is modified and if the URI does not point to an existing resource, then the server can create the
resource with that URI.
DELETE The DELETE method deletes the specified resource
HEAD The HEAD method asks for a response identical to that of a GET request, but without the response body.
It is used for modify capabilities. The PATCH request only needs to contain the changes to the resource, not the
PATCH
complete resource

Python Requests Module

To facilitate the process of making HTTP requests in Python, the `requests` module comes into play. It
simplifies the interaction with web services, allowing you to seamlessly integrate web functionality into
your Python applications. This section covers the installation of the `requests` module and its proper
importation into your Python environment.

Installing Requests

To install the `requests` library in Python, you can use the following command in your terminal or command
prompt:

pip install requests

Make sure you have Python and pip installed on your system before running this command. If you are
using a virtual environment, activate it before running the command.

Once the installation is complete, you can use the `requests` library in your Python scripts to make HTTP
requests.
Making a Simple GET Request

The GET request is one of the most common HTTP requests, used for retrieving information from a
specified resource. The accompanying Python code demonstrates how to execute a GET request, along
with an explanation of the response received. Understanding this fundamental operation sets the stage for
more complex interactions.

import requests response = requests.get("https://api.example.com/data") print(response.text)

Making a POST Request

When it comes to sending data to a server, the POST request is the go-to method. This section provides a
detailed explanation of the POST request and illustrates the Python code required. Grasping the intricacies
of making POST requests is essential for scenarios involving data submission.

import requests data = {"key": "value"} response = requests.post("https://api.example.com/post-endpoint",

data=data) print(response.json())

Making PUT and DELETE Requests

For modifying existing resources or deleting them, PUT and DELETE requests come into play. This segment
covers the purpose of these requests and guides you through the Python code necessary to execute them.
Understanding these operations expands your capabilities in handling diverse web scenarios.

import requests # Making a PUT request response_put = requests.put("https://api.example.com/put-endpoint",

data={"updated_key": "new_value"}) print(response_put.text) # Making a DELETE request response_delete =

requests.delete("https://api.example.com/delete-endpoint") print(response_delete.status_code)

Handling Response Status Codes

HTTP response status codes provide crucial information about the outcome of a request. This part of the
guide sheds light on understanding these codes, checking the status code in Python, and decoding
common status codes along with their meanings.

import requests response = requests.get("https://api.example.com/data") if response.status_code == 200:

print("Request successful!") else: print(f"Error: {response.status_code}")

Response Methods

The `Response` object in the requests module provides various methods that allow you to perform
different operations on the response. Here are some common methods:

1. response.headers

This attribute contains the HTTP headers sent by the server in the response.

import requests response = requests.get('https://www.example.com') headers = response.headers

print('Headers:', headers)

2. response.encoding

This attribute indicates the encoding of the response content.

import requests response = requests.get('https://www.example.com') encoding = response.encoding

print('Encoding:', encoding)

3. response.elapsed

This attribute returns a `timedelta` object representing the time taken for the request-response cycle.

import requests response = requests.get('https://www.example.com') elapsed_time = response.elapsed

print('Elapsed Time:', elapsed_time)

4. response.close

This method closes the underlying connection to the server.

import requests response = requests.get('https://www.example.com') response.close()

5. response.content

This attribute contains the raw content of the response in bytes.

import requests response = requests.get('https://www.example.com') content = response.content

print('Content (bytes):', content)
6. response.cookies

This attribute provides a `RequestsCookieJar` object containing the cookies sent by the server.

import requests response = requests.get('https://www.example.com') cookies = response.cookies

print('Cookies:', cookies)

7. response.history

This attribute is a list of `Response` objects for each redirection during the request.

import requests response = requests.get('https://www.example.com') redirect_history = response.history

print('Redirect History:', redirect_history)

8. response.is_permanent_redirect

These methods return `True` if the response status code indicates a permanent redirection.

import requests response = requests.get('https://www.example.com') print('Is Permanent Redirect:',

response.is_permanent_redirect)

9. response.is_redirect

The `is_redirect` method returns `True` if the response status code indicates a redirection (3xx).

import requests response = requests.get('https://www.example.com') is_redirect = response.is_redirect

print('Is Redirect:', is_redirect)

10. response.iter_content

This method allows you to iterate over the content in chunks, useful for handling large responses.

import requests response = requests.get('https://www.example.com', stream=True) for chunk in

response.iter_content(chunk_size=1024): print(chunk)

11. response.json

If the response content is in JSON format, this method parses it into a Python dictionary.

import requests response = requests.get('https://api.example.com/data.json') data = response.json()

print('Parsed JSON:', data)

12. response.url
This attribute contains the final URL after all redirections.

import requests response = requests.get('https://www.example.com') final_url = response.url

print('Final URL:', final_url)

13. response.text

This attribute contains the content of the response as a Unicode string.

import requests response = requests.get('https://www.example.com') text_content = response.text

print('Text Content:', text_content)

14. response.status_code

This attribute contains the HTTP status code returned by the server.

import requests response = requests.get('https://www.example.com') status_code = response.status_code

print('Status Code:', status_code)

15. response.request

This attribute contains the `PreparedRequest` object used to create the request.

import requests response = requests.get('https://www.example.com') request_used = response.request

print('Request Used:', request_used)

16. response.reason

This attribute contains the HTTP reason phrase returned by the server.

import requests response = requests.get('https://www.example.com') reason_phrase = response.reason

print('Reason Phrase:', reason_phrase)

17. response.raise_for_status

This method raises an HTTPError if the HTTP request returned an unsuccessful status code.

import requests response = requests.get('https://www.example.com') response.raise_for_status() #

Raises an exception for unsuccessful status codes

18. response.ok

This attribute returns `True` if the status code is less than 400, indicating a successful request.
import requests response = requests.get('https://www.example.com') is_successful = response.ok

print('Is Successful:', is_successful)

19. response.links

This attribute contains parsed HTTP Link headers as a dictionary.

import requests response = requests.get('https://www.example.com') links = response.links

print('Links:', links)

These examples cover a range of methods and attributes provided by the Response object in the requests
library, giving you various ways to inspect and handle the server’s response in your Python code.

Handling Errors and Exceptions

When working with HTTP requests using the `requests` library in Python, it’s important to handle errors
and exceptions gracefully. The `requests` library provides several mechanisms for handling errors and
exceptions that may occur during the HTTP request-response cycle. Here are some common approaches:

1. Checking the Response Status Code:

You can check the HTTP status code in the response to determine if the request was successful or
encountered an error.

import requests response = requests.get('https://www.example.com') if response.status_code == 200:

print('Request was successful') else: print(f'Request failed with status code:
{response.status_code}')

2. response.raise_for_status():

The `raise_for_status()` method raises an HTTPError for bad responses (non-2xx status codes). This allows
you to catch exceptions and handle errors.

import requests response = requests.get('https://www.example.com') try:

response.raise_for_status() print('Request was successful') except
requests.exceptions.HTTPError as err: print(f'Request failed: {err}')

3. Checking response.ok:

The `ok` attribute of the response returns `True` if the status code is less than 400 (indicating success).

import requests response = requests.get('https://www.example.com') if response.ok:

print('Request was successful') else: print('Request failed')

4. Custom Exception Handling:

You can handle specific exceptions to provide more detailed error messages or to perform specific actions
based on the error type.

import requests try: response = requests.get('https://www.example.com')

response.raise_for_status() print('Request was successful') except
requests.exceptions.HTTPError as err: print(f'HTTP error occurred: {err}') except

requests.exceptions.ConnectionError as err: print(f'Connection error occurred: {err}') except

requests.exceptions.RequestException as err: print(f'Request error occurred: {err}')

5. Timeout Handling:

You can set a timeout for the request and handle timeout-related exceptions.

import requests try: response = requests.get('https://www.example.com', timeout=5)

response.raise_for_status() print('Request was successful') except

requests.exceptions.Timeout as err: print(f'Timeout occurred: {err}') except

requests.exceptions.RequestException as err: print(f'Request error occurred: {err}')

These examples cover various ways to handle errors and exceptions when making HTTP requests with the
`requests` library. Depending on your specific use case and requirements, you can choose the approach
that best fits your needs.

Advanced Topics

Sending Query Parameters, Headers, and Cookies

Delving into advanced topics, this section explores how to enhance your HTTP requests by sending query
parameters, headers, and cookies. These features provide additional control and customization,
empowering you to tailor your requests to specific requirements.

import requests

Query parameters

params = {"param1": "value1", "param2": "value2"} response = requests.get("https://api.example.com/endpoint",

params=params) print(response.text)

Headers

headers = {"User-Agent": "MyApp/1.0"} response = requests.get("https://api.example.com/endpoint",

headers=headers) print(response.text)

Cookies
cookies = {"session_id": "123456"} response = requests.get("https://api.example.com/endpoint",

cookies=cookies) print(response.text)

Handling Redirects and Following Links

Understanding how to manage redirects and follow links is crucial for navigating complex web scenarios.
This part of the guide provides insights into handling redirects in Python, ensuring that your application
can seamlessly interact with resources even when they have moved.

import requests

Handling redirects

response = requests.get("https://api.example.com/redirecting-endpoint", allow_redirects=True)

print(response.url)

Following links

response = requests.get("https://api.example.com/page-with-link") link_url =

response.html.find("a").attrs["href"] print(link_url)

Conclusion

In conclusion, this comprehensive guide has equipped you with the knowledge and skills necessary to
confidently make HTTP requests in Python using requests module. From the basics of GET and POST
requests to handling advanced features like query parameters and redirects, you are now well-prepared to
tackle a wide range of web development scenarios. Keep experimenting, and harness the power of Python
to enhance your web applications.

You can also refer to these articles to know more:

What Is The Difference Between Python 2 and Python 3 ?

Exception Handling in Python: A Comprehensive Guide to Error Elimination
How to Print Without Newline in Python?

Frequently Asked Questions

Q1: What is HTTP, and why are HTTP requests impor tant in web development?

A1: HTTP, or Hypertext Transfer Protocol, is the foundation of data communication on the web. It
facilitates the exchange of information between clients and servers. HTTP requests are crucial in web
development as they enable clients to request resources from servers, allowing actions like fetching data,
submitting forms, and interacting with web applications.

Q2: What are the commonly used HTTP methods, and when should each be used?
A2:
GET: Used to retrieve information from a server using a specified URI.
POST: Requests a web server to accept and store data enclosed in the request message, typically used for
data submission.
PUT: Requests the storage of the enclosed entity under the supplied URI, modifying an existing resource or
creating a new one.
DELETE: Deletes the specified resource.
HEAD: Asks for a response identical to that of a GET request but without the response body.
PATCH: Used for modifying capabilities, containing only the changes to the resource.

Q3: How do I install the requests module in Python?

A3: You can install the requests module using the following command in your terminal or command
prompt:
pip install requests
Make sure Python and pip are installed on your system, and if you are using a virtual environment, activate
it before running the command.

Q4: How can I make a simple GET request using the requests module in Python?

import requests
response = requests.get(“https://api.example.com/data”)
print(response.text)
This code demonstrates making a GET request to the specified URL and printing the response text.

Article Url - https://www.analyticsvidhya.com/blog/2024/01/http-requests-in-python/

Aayush Tyagi