← Previous Next →

Making HTTP Requests

In modern software development, interacting with web services and APIs is fundamental. Making HTTP requests allows your Python applications to communicate over the internet, fetch or send data, and integrate with countless online services. This lesson dives deep into the mechanics of making HTTP requests in Python, covering everything from basic GET requests to advanced techniques such as handling sessions, authentication, streaming responses, error handling, and asynchronous requests.

We will explore both the built-in http.client and urllib libraries, but the primary focus will be on the highly popular and user-friendly requests library, which simplifies HTTP interactions and is widely used in production code.

By the end, you will be equipped with the knowledge to perform HTTP operations efficiently, securely, and robustly, enabling you to build powerful applications that leverage web data and services.

💡 A Simple Analogy: HTTP Requests as Sending Letters

Imagine you want to get information from a friend or send them data; you write a letter (the HTTP request) and post it (send it over the internet). Your friend reads the letter and replies with another letter (the HTTP response). Just like different types of mail (e.g., postcards, packages), HTTP has different methods like GET and POST to indicate your intent. Headers are like the envelope details, specifying how the letter should be handled, and the body is the content inside the letter.

🎯 Real-World Use Case: Consuming a REST API

Suppose you want to build a Python app that shows current weather data. You’d make HTTP GET requests to a weather API, parse the JSON response, and present it. Alternatively, if you’re submitting data to a server, like a contact form, you’d send POST requests with form data. Understanding how to make these HTTP requests correctly, handle errors, and manage sessions is essential for any real-world app that interacts with the web.

⚠️ Common Pitfall: Ignoring Response Status Codes

Beginners often assume that a request returning without an exception means success. However, HTTP responses can have various status codes indicating redirects, client errors (4xx), or server errors (5xx). Always check and handle response status codes appropriately to avoid unexpected bugs or failed operations.

1

Understanding HTTP Methods HTTP defines several request methods like GET (retrieve data), POST (submit data), PUT (replace data), DELETE (remove data), and more. Knowing when and how to use these is the foundation of making effective requests.

2

Using the requests Library The requests library abstracts away much of the complexity. We’ll learn how to make GET, POST, and other requests, pass headers, query parameters, and data payloads, and handle JSON responses easily.

3

Handling Errors and Exceptions Learn how to catch connection errors, timeout exceptions, and check HTTP status codes to make your code robust and reliable.

4

Managing Sessions and Cookies Persistent sessions allow you to maintain cookies and headers across multiple requests, essential for sites that require login or stateful interactions.

5

Advanced Techniques Explore streaming large downloads, setting custom timeouts, using proxies, authentication techniques (Basic, OAuth), and asynchronous requests with httpx or aiohttp.

Architecture of Making HTTP Requests
Architecture of Making HTTP Requests

📌 Deep Dive: Basic GET Request Using requests

PYTHON 

# Import the requests library
import requests

# Define the URL to send the GET request to
url = "https://api.github.com/repos/python/cpython"

try:
    # Make a GET request to the URL
    response = requests.get(url)

    # Check if the request was successful (status code 200)
    if response.status_code == 200:
        # Parse JSON content from the response
        data = response.json()
        print("Repository full name:", data['full_name'])
        print("Description:", data['description'])
        print("Watchers count:", data['watchers_count'])
    else:
        print(f"Failed to retrieve data. Status code: {response.status_code}")
except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")
Output
Repository full name: python/cpython Description: The Python programming language Watchers count: 5470

📌 Deep Dive: POST Request with Form Data

PYTHON 

# Import requests to send HTTP requests
import requests

# URL for a test POST endpoint that echoes data
post_url = "https://httpbin.org/post"

# Data to send in the POST request
payload = {
    'username': 'python_learner',
    'password': 'securepassword123'
}

try:
    # Send a POST request with form data
    response = requests.post(post_url, data=payload)

    # Confirm the request was successful
    response.raise_for_status()

    # Parse JSON response
    json_response = response.json()
    print("Form data sent:")
    print(json_response['form'])
except requests.exceptions.RequestException as e:
    print(f"Request failed: {e}")
Output
Form data sent: {'password': 'securepassword123', 'username': 'python_learner'}

📌 Deep Dive: Using Sessions to Persist Cookies

PYTHON 

import requests

# Create a session object to persist parameters across requests
session = requests.Session()

# Set a cookie using the session
set_cookie_url = "https://httpbin.org/cookies/set/sessioncookie/123456789"

# Get cookies to verify
get_cookies_url = "https://httpbin.org/cookies"

try:
    # Set a cookie by visiting the URL
    session.get(set_cookie_url)
    
    # Now retrieve cookies stored in the session
    response = session.get(get_cookies_url)
    cookies = response.json().get('cookies', {})
    
    print("Cookies stored in session:")
    print(cookies)
except requests.exceptions.RequestException as e:
    print(f"Session request failed: {e}")
Output
Cookies stored in session: {'sessioncookie': '123456789'}

📌 Deep Dive: Handling Timeouts and Exceptions

PYTHON 

import requests

url = "https://httpbin.org/delay/5"  # This endpoint delays response by 5 seconds

try:
    # Set a timeout of 2 seconds (less than delay to trigger timeout)
    response = requests.get(url, timeout=2)
    response.raise_for_status()
    print("Request successful")
except requests.exceptions.Timeout:
    print("The request timed out.")
except requests.exceptions.HTTPError as http_err:
    print(f"HTTP error occurred: {http_err}")
except requests.exceptions.RequestException as err:
    print(f"An error occurred: {err}")
Output
The request timed out.

📌 Deep Dive: Sending JSON and Custom Headers

PYTHON 

import requests

url = "https://httpbin.org/post"

# JSON data to send
json_payload = {
    "name": "Alice",
    "age": 30
}

# Custom headers
headers = {
    'User-Agent': 'MyApp/1.0',
    'Content-Type': 'application/json'
}

try:
    # Send POST request with JSON data and custom headers
    response = requests.post(url, json=json_payload, headers=headers)
    response.raise_for_status()
    print("Server received JSON data:")
    print(response.json()['json'])
except requests.exceptions.RequestException as e:
    print(f"Error sending JSON data: {e}")
Output
Server received JSON data: {'age': 30, 'name': 'Alice'}

📌 Deep Dive: Basic Authentication with requests

PYTHON 

import requests
from requests.auth import HTTPBasicAuth

url = "https://httpbin.org/basic-auth/user/passwd"

try:
    # Make a GET request with Basic Authentication credentials
    response = requests.get(url, auth=HTTPBasicAuth('user', 'passwd'))
    response.raise_for_status()
    print("Authentication successful!")
    print("Response JSON:", response.json())
except requests.exceptions.HTTPError as e:
    print(f"Authentication failed: {e}")
except requests.exceptions.RequestException as e:
    print(f"Request error: {e}")
Output
Authentication successful! Response JSON: {'authenticated': True, 'user': 'user'}

📌 Deep Dive: Streaming Large File Download

PYTHON 

import requests

url = "https://speed.hetzner.de/100MB.bin"  # Large test file

try:
    with requests.get(url, stream=True) as response:
        response.raise_for_status()
        with open("100MB.bin", "wb") as f:
            # Download in chunks to avoid loading entire file in memory
            for chunk in response.iter_content(chunk_size=8192):
                if chunk:
                    f.write(chunk)
    print("Download completed successfully.")
except requests.exceptions.RequestException as e:
    print(f"Download failed: {e}")
Output
Download completed successfully.

📌 Deep Dive: Asynchronous HTTP Requests with httpx

PYTHON 

import asyncio
import httpx

async def fetch_url(client, url):
    try:
        response = await client.get(url)
        response.raise_for_status()
        print(f"{url}: {len(response.text)} characters received")
    except httpx.RequestError as exc:
        print(f"An error occurred while requesting {url} - {exc}")

async def main():
    urls = [
        "https://httpbin.org/get",
        "https://api.github.com",
        "https://www.python.org"
    ]
    async with httpx.AsyncClient() as client:
        tasks = [fetch_url(client, url) for url in urls]
        await asyncio.gather(*tasks)

# Run the asynchronous main function
asyncio.run(main())
Output
https://httpbin.org/get: 306 characters received https://api.github.com: 203 characters received https://www.python.org: 49325 characters received
← Previous Next →