Generators vs Lists in Python

CodingNow · 22 July 2026

generators vs lists python

Generators vs Lists in Python: Which One Should You Use?

Introduction: The Memory Dilemma

You're processing a massive dataset—millions of records. Do you load everything into a list and risk running out of memory, or is there a smarter way? Enter generators, Python's secret weapon for handling large data efficiently.

In this guide, we'll compare generators and lists, explore when to use each, and show you how to write cleaner, more memory-efficient code.

The Short Answer

Feature	Lists	Generators
Memory usage	High (stores all items)	Low (yields one at a time)
Speed	Faster for iteration	Slightly slower
Indexing	Yes (`list[0]`)	No
Reusability	Reusable multiple times	Single-use (exhaustible)
Length	Known (`len(list)`)	Unknown
Storage	All items in memory	Generates items on-the-fly

What's a List?

A list stores all elements in memory at once:

python

# Creating a list of 10 million numbers
numbers_list = [x for x in range(10_000_000)]
# Memory: ~320 MB

# You can:
print(numbers_list[0])        # Access by index
print(len(numbers_list))       # Know the length
print(numbers_list[-1])        # Access from end

Pros:

Fast random access
Can be reused multiple times
Supports slicing, sorting, reversing

Cons:

Memory-hungry
Slow to create for large datasets

What's a Generator?

A generator yields items one at a time without storing them:

python

# Creating a generator of 10 million numbers
numbers_gen = (x for x in range(10_000_000))
# Memory: ~0 bytes (only stores the range object)

# You can:
print(next(numbers_gen))      # Get first item (0)
print(next(numbers_gen))      # Get next item (1)
# But cannot: numbers_gen[0]  # TypeError!

# You MUST iterate:
for num in numbers_gen:
    if num > 5:
        break

Pros:

Extremely memory-efficient
Lazy evaluation (compute only what you need)
Can represent infinite sequences

Cons:

Single-use only
No random access
Slightly slower for small datasets

Real-World Memory Comparison

Let's see the difference:

python

import sys

# List comprehension
list_comp = [x**2 for x in range(1000000)]
print(sys.getsizeof(list_comp))  # ~8 MB

# Generator expression
gen_exp = (x**2 for x in range(1000000))
print(sys.getsizeof(gen_exp))    # ~104 bytes!

# That's 80,000x smaller!

Creating Generators

1. Generator Expressions (One-liners)

python

# List comprehension vs generator expression
squares_list = [x**2 for x in range(10)]
squares_gen = (x**2 for x in range(10))

print(squares_list)  # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
print(squares_gen)   # <generator object at 0x...>

2. Generator Functions with `yield`

python

def fibonacci(limit):
    a, b = 0, 1
    while a < limit:
        yield a
        a, b = b, a + b

# Use it
for num in fibonacci(1000):
    print(num)  # 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987

3. Generator Methods

python

def counter():
    i = 0
    while True:
        received = yield i
        if received == 'reset':
            i = 0
        else:
            i += 1

gen = counter()
print(next(gen))        # 0
print(next(gen))        # 1
print(gen.send('reset')) # Resets to 0
print(next(gen))        # 0

When to Use Lists

Use lists when you:

Need random access (access by index)
Need to iterate multiple times
Need to know the length upfront
Have a small dataset that fits in memory
Need to modify, sort, or reverse the data
Need to use list methods like .append(), .pop(), .insert()

python

# Good use case for lists
def get_top_users(users):
    sorted_users = sorted(users, key=lambda x: x['score'], reverse=True)
    return sorted_users[:10]  # Needs indexing and sorting

# Good use case for lists
user_logs = []
for entry in log_stream:
    if entry['error']:
        user_logs.append(entry)  # Need to store for later analysis

When to Use Generators

Use generators when you:

Process large files (logs, CSVs, images)
Read streaming data (API responses, database cursors)
Compute infinite sequences (Fibonacci, primes)
Have memory constraints (limited RAM)
Only need to iterate once
Want lazy evaluation (compute only what's needed)

python

# Good use case for generators
def read_large_file(file_path):
    with open(file_path) as f:
        for line in f:
            yield line.strip()

# Process line by line without loading entire file
for line in read_large_file('huge_file.txt'):
    process(line)  # Memory efficient!

# Infinite sequence
def infinite_evens():
    num = 0
    while True:
        yield num
        num += 2

evens = infinite_evens()
print(next(evens))  # 0
print(next(evens))  # 2
print(next(evens))  # 4

Performance Comparison

Let's benchmark:

python

import time
import sys

# Create list
start = time.time()
list_data = [x**2 for x in range(10_000_000)]
list_time = time.time() - start
list_mem = sys.getsizeof(list_data)

# Create generator
start = time.time()
gen_data = (x**2 for x in range(10_000_000))
gen_time = time.time() - start
gen_mem = sys.getsizeof(gen_data)

print(f"List: {list_time:.2f}s, {list_mem/1024/1024:.1f} MB")
print(f"Generator: {gen_time:.2f}s, {gen_mem} bytes")

# Iteration speed comparison
start = time.time()
for num in list_data[:1000000]:  # Only first million
    pass
list_iter_time = time.time() - start

start = time.time()
for num in (x**2 for x in range(1000000)):
    pass
gen_iter_time = time.time() - start

print(f"List iteration: {list_iter_time:.4f}s")
print(f"Generator iteration: {gen_iter_time:.4f}s")

Typical results:

List creation: 0.8s, 320 MB
Generator creation: 0.0s, 104 bytes
List iteration: 0.05s
Generator iteration: 0.07s (slightly slower)

Common Pitfalls

1. Exhausting a Generator

python

gen = (x for x in range(3))
print(list(gen))  # [0, 1, 2]
print(list(gen))  # []  # Generator is exhausted!

Solution: Recreate the generator if you need to iterate again.

2. Converting Generators to Lists (Defeats Purpose)

python

# DON'T DO THIS - loads everything into memory
gen = (x**2 for x in range(1000000))
listified = list(gen)  # Memory spike!

# DO THIS - process as needed
gen = (x**2 for x in range(1000000))
for num in gen:
    process(num)  # Memory efficient

3. Using `len()` on Generators

python

gen = (x for x in range(10))
print(len(gen))  # TypeError: object of type 'generator' has no len()

Solution: Track count manually or convert to list (if small enough).

Advanced Generator Patterns

1. Generator Pipelines

Chain generators together for data processing:

python

def read_data():
    for i in range(1000000):
        yield i

def filter_even(data):
    for num in data:
        if num % 2 == 0:
            yield num

def square(data):
    for num in data:
        yield num**2

# Pipeline - data flows through!
data = read_data()
filtered = filter_even(data)
squared = square(filtered)

for result in squared:
    if result > 100:
        break  # Stops processing all generators!

2. `yield from` for Delegation

python

def chain(*iterables):
    for iterable in iterables:
        yield from iterable  # Delegates to sub-iterators

for value in chain([1, 2], (3, 4), "56"):
    print(value)  # 1, 2, 3, 4, '5', '6'

3. Generators as Coroutines

python

def logger():
    while True:
        log_entry = yield
        timestamp = time.time()
        print(f"[{timestamp}] {log_entry}")

log = logger()
next(log)  # Initialize
log.send("Started processing")
log.send("Completed successfully")

Best Practices

Start with lists for small data, switch to generators when memory becomes an issue.
Profile before optimizing - don't prematurely optimize.
Use generator expressions instead of list comprehensions for large datasets.
Document generator behavior - especially if they're single-use.
Choose based on use case:
- Multiple passes, random access, modification → Lists
- Single pass, streaming, memory constraints → Generators

Quick Decision Guide

text

Need random access?
    YES → Use a list
    NO  → ↓

Need to iterate multiple times?
    YES → Use a list
    NO  → ↓

Is data size > 100k items?
    YES → Use a generator
    NO  → ↓

Need to know length?
    YES → Use a list
    NO  → ↓

Working with files/streams?
    YES → Use a generator
    NO  → Use a list (it's simpler)

Conclusion

Lists are your go-to for everyday tasks with small to medium datasets. They're familiar, fast, and flexible.

Generators shine when you're dealing with large data, streaming, or infinite sequences. They're your memory-saving superheroes.

The key is understanding the trade-offs and choosing the right tool for the job. As you gain experience, this decision becomes second nature.

Contact Us

Phone: +91 9667708830
Email: info@codingnow.in
Website: https://codingnow.in/

Address:
2nd Floor, Kapil Vihar (Opp. Metro Pillar No.354)
Pitampura, New Delhi – 110034

Backlink to main website: Explore Python and AI courses at Coding Now – Gurukul of AI

Generators vs Lists in Python

Generators vs Lists in Python: Which One Should You Use?

Introduction: The Memory Dilemma

The Short Answer

What's a List?

What's a Generator?

Real-World Memory Comparison

Creating Generators

1. Generator Expressions (One-liners)

2. Generator Functions with yield

3. Generator Methods

When to Use Lists

When to Use Generators

Performance Comparison

Common Pitfalls

1. Exhausting a Generator

2. Converting Generators to Lists (Defeats Purpose)

3. Using len() on Generators

Advanced Generator Patterns

1. Generator Pipelines

2. yield from for Delegation

3. Generators as Coroutines

Best Practices

Quick Decision Guide

Conclusion

Contact Us

2. Generator Functions with `yield`

3. Using `len()` on Generators

2. `yield from` for Delegation