Iterators & Generators

Welcome to this comprehensive lesson on Iterators and Generators in Python — two foundational concepts that empower you to work efficiently with sequences and streams of data. Whether you want to loop over large datasets without loading everything into memory or want to create your own custom sequence objects, mastering iterators and generators is essential.

In this lesson, we'll unravel these concepts step-by-step, starting with what iterators really are, how Python uses them, and then delving into generators — a powerful and elegant Python feature that makes creating iterators simple and clean.

Understanding Iterators: The Core of Python’s Looping Mechanism

At a high level, an iterator is any object in Python that allows you to traverse through all the elements of a collection, one element at a time, without needing to know the internal structure of that collection.

Think of an iterator as a bookmark moving through a book. You don’t read the entire book at once; instead, you move from one page to the next. Similarly, an iterator “remembers” its current position and fetches the next item on demand.

How Python Uses Iterators Under the Hood

When you write a simple for-loop like this:

numbers = [10, 20, 30]
for num in numbers:
    print(num)

Python actually translates this internally to:

numbers = [10, 20, 30]
it = iter(numbers)   # Get an iterator object
while True:
    try:
        num = next(it)    # Get the next item
        print(num)
    except StopIteration:
        break          # Exit loop when no more items

Here, iter() produces an iterator object from the list, and next() fetches successive elements. When there are no elements left, next() raises a StopIteration exception to signal the end of iteration.

What Makes an Object an Iterator?

In Python, an object is an iterator if it implements two methods:

• __iter__() — returns the iterator object itself.
• __next__() — returns the next value or raises StopIteration.

Most built-in collections like lists, tuples, dictionaries, and sets are iterables, which means they can produce iterators (via iter()), but they themselves are not iterators.

Creating Your Own Iterator Class

To deepen your understanding, let’s create a custom iterator from scratch. Suppose we want to iterate over the first n square numbers (0, 1, 4, 9, ...) one by one.

class SquareNumbers:
    def __init__(self, n):
        self.n = n          # Number of squares to generate
        self.i = 0          # Current index

    def __iter__(self):
        return self         # Iterator returns itself

    def __next__(self):
        if self.i >= self.n:
            raise StopIteration
        result = self.i ** 2
        self.i += 1
        return result

# Usage
squares = SquareNumbers(5)
for num in squares:
    print(num)

This class keeps track of its current position and yields the square of the index until it reaches n. Notice how clean the iteration becomes when you implement the iterator protocol.

Generators: Python’s Elegant Way to Create Iterators

While building iterator classes manually is instructive, it often involves boilerplate code. Python offers a simpler and more powerful way called generators.

A generator is a special kind of iterator defined with a function that uses the yield statement. When the generator function is called, it returns a generator object that can be iterated over.

Creating a Generator Function

Let’s rewrite our squares iterator using a generator function:

def square_numbers(n):
    for i in range(n):
        yield i ** 2

# Usage
for num in square_numbers(5):
    print(num)

The yield keyword pauses the function, returns a value, and saves the function’s state so it can resume where it left off on the next call. This makes generators memory-efficient and perfect for large or infinite sequences.

Key Benefits of Generators

• Memory Efficiency: Generate items on the fly without storing the entire sequence.
• Cleaner Syntax: No need to implement __iter__ and __next__ methods manually.
• Composability: Easy to chain and combine generators for complex data pipelines.

Exploring Generator Expressions

Python also supports generator expressions, a concise syntax similar to list comprehensions but using parentheses instead of square brackets.

Example: Generate squares of numbers from 0 to 4.

gen_expr = (x ** 2 for x in range(5))

for val in gen_expr:
    print(val)

Unlike list comprehensions which create the entire list in memory, generator expressions yield one item at a time, making them ideal for large data streams.

Practical Use Cases for Iterators and Generators

Understanding when and why to use iterators and generators can improve program performance and readability significantly.

• Processing large files line by line without loading the entire file into memory.
• Generating infinite sequences like Fibonacci numbers or random numbers — no need to store all values.
• Streaming data pipelines where data is processed in chunks as it arrives.
• Custom iterable objects that encapsulate complex iteration logic cleanly.

Advanced Generator Features: send(), throw(), and close()

Generators are not just simple iterators; they can also be controlled externally via special methods:

• send(value) — Resumes generator execution and sends a value that can be used inside the generator.
• throw(type, value=None, traceback=None) — Raises an exception inside the generator at the current yield point.
• close() — Stops the generator by raising GeneratorExit.

These advanced features enable coroutines and sophisticated control flows but are beyond the scope of this beginner lesson. Keep them in mind as you advance!

Summary: What You Should Take Away

• Iterators are objects that implement __iter__() and __next__() to allow progressive retrieval of data.
• Iterables produce iterators via iter() and can be used in for-loops.
• You can build custom iterators by implementing the iterator protocol in your classes.
• Generators simplify iterator creation by using yield, allowing stateful iteration with minimal code.
• Generator expressions provide a concise syntax for creating generators on the fly.
• Use generators to work efficiently with large or infinite data sequences.

Now that you have a solid understanding of iterators and generators, you can write more efficient and pythonic code that handles data streams gracefully.