Skip to content
Open DegreeOpen Degree

Iterate and transform data in Python

I can iterate over a collection with a loop or a comprehension, and use a generator when the data should not all be held in memory at once.

Domain
Python Programming
Level
foundation
Status
draft · v0.1.0
Also called
List comprehensions, Python loops, Generators, Iterators
License
CC BY-SA 4.0

Learn first

Learn it

Scope

Walking over data and producing new data from it: the shape most Python code takes once it does anything real.

In scope: for loops, comprehensions for lists, dicts and sets, iterating a dictionary properly, and generators for data too large or too slow to materialise. Out of scope: itertools, and parallelism.

Loop or comprehension

A comprehension is a loop whose whole purpose is to build a collection. When that is what the loop does, the comprehension says so in one line:

names = [user.name for user in users if user.active]

When the loop does several things — building, logging, updating a counter — a comprehension hides that, and a plain loop is clearer. The test is not brevity. It is whether one line describes the result or conceals the work.

Iterating a dictionary on purpose

Iterating a dict gives keys. .values() gives values, .items() gives both as pairs. Choosing the right one removes the most common noise in beginner code: looking up d[key] inside a loop that is already holding the value.

What a generator changes

A function that yields returns a generator: nothing runs until something asks for the next item, and only one item exists at a time. Two things follow.

The first is memory. Reading a ten-million-line file into a list needs the whole file; a generator needs one line. The second is when work happens — a generator that is never consumed does nothing at all, which is either exactly what you wanted or a bug that looks like missing output.

A generator is exhausted once consumed. Looping over it twice gives the items and then nothing, with no error. If you need the data twice, keep a list.

Common misconceptions

  • “A comprehension is faster, so always prefer one.” The gap is small and irrelevant next to readability, or next to choosing the right data structure.
  • “A generator is a lazy list.” It has no length, no indexing, and it ends. Calling len() on one is an error, not a slow path.
  • “for i in range(len(items)) is how you loop.” Iterate the items directly, and use enumerate when you genuinely need the index.
  • “Nothing happened, so the generator is broken.” Nothing consumed it. That is the design.

Notes for teachers

Have learners write the same transformation three ways — loop, comprehension, generator — over a file big enough to feel, then ask which they would put in a pull request and why. The answer should mention the reader, not the benchmark.

The exhaustion trap is worth meeting on purpose: loop over a generator twice, watch the second loop produce nothing silently, then explain it.

Resources

ResourceKindCostProvider
The Python Tutorial: data structures, including list comprehensionsreadingfreePython Software Foundation
Functional programming HOWTO: iterators and generatorsreadingfreePython Software Foundation
Python Tutor, step-by-step executioninteractivefreePython Tutor

Courses that include it

draft

Python for a Coding Assessment

A refresher path through the Python graph for someone who has written Python before and has an assessment coming: run and read, rebuild the core, then work the way an assessment expects.

Prove it

Evidence of mastery

  • Rewrites a loop that builds a list as a comprehension, and back again, and says which reads better for that case.
  • Iterates over a dictionary's keys, values and items deliberately, choosing the one the task needs.
  • Replaces a function that returns a large list with a generator, and explains what changes about memory and about when the work happens.
  • Explains why a generator can only be consumed once, having been caught by it.

Assessments that measure it

No assessment yet.

Credentials that include it

draft

Python Problem Solving

The holder can take an unfamiliar problem, write a tested Python solution someone else can run, and explain what it costs.

What learners show

Nothing to show yet. The publisher last published on 2026-09-20, and no aggregate for this outcome has reached the 50 learners a number needs. Open Degree does not track learners. About learner evidence.

What comes next