Multiprocessing

In modern programming, speed and efficiency are king. Python, while incredibly versatile and easy to learn, has one notable limitation: its Global Interpreter Lock (GIL) which restricts the execution of multiple threads to one at a time. This can be a bottleneck if you want to perform CPU-bound tasks in parallel.

This is where multiprocessing shines. Unlike threading, multiprocessing allows you to spawn separate processes that run independently and simultaneously, each with its own Python interpreter and memory space. This lets you fully leverage multiple CPU cores to speed up your programs.

Why Multiprocessing?

Imagine you need to process a large dataset, perform complex calculations, or run multiple independent tasks. If you did this sequentially, it would take significant time. Threads may not help much due to the GIL, but separate processes can truly run in parallel, making your program faster and more efficient.

Getting Started with Python’s multiprocessing Module

Python provides a built-in multiprocessing module designed to create and manage processes easily. The core concepts you'll work with include:

• Process: A single process that runs a target function.
• Pool: A convenient way to manage a pool of worker processes.
• Queues and Pipes: For inter-process communication.

The Process Class

The simplest way to use multiprocessing is by creating a Process object. Here's how it works:

import multiprocessing
import time

def worker(num):
    print(f'Worker {num} starting')
    time.sleep(2)
    print(f'Worker {num} finished')

if __name__ == '__main__':
    processes = []
    for i in range(3):
        p = multiprocessing.Process(target=worker, args=(i,))
        processes.append(p)
        p.start()

    for p in processes:
        p.join()

    print('All workers completed')

In this example, three worker processes run concurrently. The join() call ensures the main program waits until all workers finish.

Using a Pool of Workers

When you have many tasks, creating a process for each can be inefficient. The Pool class manages a fixed number of worker processes and distributes tasks to them.

from multiprocessing import Pool
import time

def square(n):
    time.sleep(1)
    return n * n

if __name__ == '__main__':
    with Pool(4) as pool:
        results = pool.map(square, range(10))
    print(results)

Here, the Pool creates 4 worker processes and distributes the square function across the input numbers. The map method collects results in order.

Inter-Process Communication (IPC)

Since each process has its own memory space, sharing data is not as straightforward as in threading. Python’s multiprocessing module offers tools to enable communication between processes:

• Queue: A thread- and process-safe FIFO queue.
• Pipe: A direct two-way communication channel.
• Manager: Offers a way to create shared objects like lists and dictionaries.

Example: Sharing Data with a Queue

import multiprocessing

def producer(q):
    for i in range(5):
        print(f'Producing {i}')
        q.put(i)

def consumer(q):
    while True:
        item = q.get()
        if item is None:
            break
        print(f'Consumed {item}')

if __name__ == '__main__':
    q = multiprocessing.Queue()
    p1 = multiprocessing.Process(target=producer, args=(q,))
    p2 = multiprocessing.Process(target=consumer, args=(q,))

    p1.start()
    p2.start()

    p1.join()
    q.put(None)  # Sentinel to signal consumer to exit
    p2.join()

The producer puts items into the queue, and the consumer fetches them. The special None value signals the consumer to stop.

Multiprocessing vs Threading: When to Use What?

Best Practices and Tips

• Protect your entry point: Always wrap multiprocessing code inside if __name__ == '__main__': to prevent recursive process spawning on Windows.
• Limit process count: Match the number of processes to the CPU cores available (can be found with multiprocessing.cpu_count()).
• Use Pools for many tasks: Managing processes manually is error-prone; Pool handles it elegantly.
• Be mindful of data copying: Passing large objects between processes can be slow because they get serialized and copied.
• Debugging: Multiprocessing can be tricky to debug; use logging and small, isolated tests.

Summary

Multiprocessing in Python is a powerful tool to overcome the limitations of the GIL and utilize multiple CPU cores effectively. By creating separate processes, you run code truly concurrently, speeding up CPU-heavy tasks.

Key takeaways:

• Use multiprocessing.Process for simple, manual process creation.
• multiprocessing.Pool is great for managing worker pools and distributing tasks.
• Use Queues, Pipes, or Managers for sharing data between processes safely.
• Always guard your multiprocessing code with if __name__ == '__main__':.
• Choose multiprocessing over threading for CPU-bound tasks.

With these concepts and tools, you can write Python programs that truly harness the power of your machine's multiple cores.