🟡 Advanced Python · Lesson 22
Generators and Iterators in Python
In this lesson: Learn iterators, iter/next and generators with yield in Python — memory-saving examples with output. In Hindi & English.
Why iterators and generators?
Every for loop in Python uses an iterator – an object that hands out one item at a time. A generator is the easiest way to build your own iterator: a function that uses yield to produce values one by one, only when they are needed.
This matters when data is large: reading a 5 GB log file, streaming lakhs of database rows, or generating an endless series. A list would load everything into memory; a generator keeps only the current item.
How a for loop really works: iter() and next()
Python – iter_next.py
marks = [78, 91, 66]
it = iter(marks) # ask the list for an iterator
print(next(it))
print(next(it))
print(next(it))
try:
next(it) # nothing left
except StopIteration:
print("StopIteration – the loop ends here")78
91
66
StopIteration – the loop ends here
A for loop simply calls iter() once, then next() again and again, and stops quietly when StopIteration is raised.
| Term | Meaning | Examples |
|---|---|---|
| Iterable | Something you can loop over; iter(x) works | list, tuple, str, dict, set, file, range |
| Iterator | Object with __next__; remembers its position; used once | what iter(list) returns, file objects, generators |
| Generator | An iterator created by a function with yield or a generator expression | def count(): yield 1, (x*x for x in data) |
Writing an iterator class (the long way)
Python – countdown_class.py
class Countdown:
def __init__(self, start):
self.current = start
def __iter__(self):
return self
def __next__(self):
if self.current <= 0:
raise StopIteration
value = self.current
self.current -= 1
return value
for n in Countdown(3):
print(n, end=" ")
print("🚀")3 2 1 🚀
It works, but you must manage state yourself. A generator does the same job in four lines.
Generator functions with yield
Python – yield_basics.py
def countdown(start):
print("generator started")
while start > 0:
yield start # pause here and hand out a value
start -= 1
print("generator finished")
gen = countdown(3)
print(type(gen).__name__) # nothing has run yet
print(next(gen))
print(next(gen))
print(list(gen)) # takes the restgenerator
generator started
3
2
generator finished
[1]
- Calling
countdown(3)runs no code – it only creates a generator object. - Each
next()runs until the nextyield, returns that value and pauses, keeping all local variables. - When the function ends,
StopIterationis raised automatically.
| return | yield |
|---|---|
| Ends the function | Pauses the function |
| Gives one value | Can give many values over time |
| Local variables are lost | Local variables are kept |
Memory: list vs generator
Python – memory.py
import sys
squares_list = [n * n for n in range(1_000_000)]
squares_gen = (n * n for n in range(1_000_000)) # round brackets
print("list size :", sys.getsizeof(squares_list) // 1024, "KB")
print("gen size :", sys.getsizeof(squares_gen), "bytes")
print("same total:", sum(squares_list) == sum(squares_gen))list size : 8250 KB
gen size : 200 bytes
same total: True
💡 Generator expression
Replace the square brackets of a list comprehension with round brackets and you get a generator. Inside a function call you can drop the extra brackets: sum(x*x for x in data).
⚠️ A generator can be used only once
After it is exhausted it stays empty. If you need the data twice, create a new generator or store it in a list.
Real example: a data pipeline over a large log file
Generators can be chained like pipes. Each stage pulls one line at a time, so this program uses the same small memory for a 10-line file or a 10-crore-line file.
Python – log_pipeline.py
def read_lines(path):
with open(path, encoding="utf-8") as f:
for line in f: # files are iterators too
yield line.rstrip("\n")
def only_errors(lines):
for line in lines:
if " ERROR " in line:
yield line
def parse(lines):
for line in lines:
date, time, level, *message = line.split()
yield {"time": time, "message": " ".join(message)}
errors = parse(only_errors(read_lines("server.log")))
for e in errors:
print(e["time"], "-", e["message"])09:02:14 - payment gateway timeout
09:07:03 - database connection refused
Infinite generators and itertools
Python – infinite.py
import itertools
def invoice_numbers(prefix="INV"):
n = 1
while True: # never ends – that is fine for a generator
yield f"{prefix}-{n:04d}"
n += 1
numbers = invoice_numbers()
print(next(numbers), next(numbers))
# take only what you need with islice
print(list(itertools.islice(invoice_numbers("SS"), 3)))
# batches of 2 students (itertools.batched exists from Python 3.12; this works everywhere)
def batches(items, size):
it = iter(items)
while chunk := list(itertools.islice(it, size)):
yield chunk
print(list(batches(["Asha", "Ravi", "Neha", "Kabir", "Isha"], 2)))INV-0001 INV-0002
['SS-0001', 'SS-0002', 'SS-0003']
[['Asha', 'Ravi'], ['Neha', 'Kabir'], ['Isha']]
yield from: delegating to another generator
Python – yield_from.py
def walk(folder):
"""Yield every file name inside nested folders (a dict tree)."""
for name, value in folder.items():
if isinstance(value, dict):
yield from walk(value) # hand over to the inner generator
else:
yield name
project = {"index.html": 1, "css": {"style.css": 1}, "js": {"app.js": 1, "lib": {"chart.js": 1}}}
print(list(walk(project)))['index.html', 'style.css', 'app.js', 'chart.js']
Common mistakes
| Mistake | Result | Fix |
|---|---|---|
| Looping over the same generator twice | Second loop prints nothing | Create it again or use a list |
Calling len() on a generator | TypeError | Count with sum(1 for _ in gen) or use a list |
| Expecting code to run when the generator is created | Nothing happens until next()/loop | Remember: generators are lazy |
| Using a list for a huge file | MemoryError / slow program | Iterate the file or yield lines |
Practice tasks
- Write
even_numbers(limit)that yields even numbers up to the limit. - Write a generator that reads
marks.csvline by line and yields only students with more than 75 marks. - Create an infinite generator of dates starting today (use
datetime.dateandtimedelta) and print the next 7 days. - Compare the memory of
[x for x in range(10**6)]and(x for x in range(10**6))withsys.getsizeof.
Summary
- Iterable → can be looped; iterator → gives items with
next()and remembers its place. - A function with
yieldreturns a generator; it runs lazily and pauses at each yield. - Generator expressions
( … )save memory compared with lists[ … ]. - Chain generators to build memory-efficient pipelines; use
itertoolsfor slicing and combining. - A generator is consumed once.
Frequently Asked Questions
What is the difference between an iterator and a generator?
Every generator is an iterator, but not every iterator is a generator. An iterator is any object with __iter__ and __next__. A generator is an iterator that Python builds for you from a function with yield or from a generator expression.
When should I use a generator instead of a list?
Use a generator when the data is large, comes from a file or network, is infinite, or is needed only once in a loop. Use a list when you need indexing, length, sorting or multiple passes.
Can a generator return a value?
Yes. A return statement inside a generator ends it, and the value is stored in the StopIteration exception. yield from receives that value, which is useful when one generator delegates to another.
💻 Live Code Editor
This page's programs are ready here — run them, edit them, and learn. No installation needed.
Powered by OneCompiler. The code loads into the editor automatically — press Run to see the output. If the editor does not open, open it in a new tab.
