Read and write files and structured data

I can read and write text, CSV and JSON files in Python using paths that work on any machine, and choose the format that fits what the data is for.

Domain: Python Programming
Level: intermediate
Status: draft · v0.1.0
Also called: File handling, CSV in Python, JSON in Python, pathlib, Reading files
License: CC BY-SA 4.0

Learn first

Iterate and transform data in Python foundation
Handle errors deliberately intermediate

Learn it

Scope

Getting data in and out of a program: the part of almost every real task that is not the interesting part, and the part that breaks on someone else’s machine.

In scope: text, CSV and JSON; pathlib for paths; with for handles; encoding; and failing sensibly on a missing file or a bad row. Out of scope: databases, Excel formats, and dataframes.

Always `with`

with open(path) as f: closes the file when the block ends, including when an exception leaves the block early. Without it, a handle stays open until the interpreter happens to collect it, which on Windows means the next process cannot touch the file, and on a long run means hitting the open-file limit.

Paths are not strings

"data/" + name breaks the first time it meets Windows, a space, or a name that already ends in a separator. Path("data") / name is correct everywhere, and carries the useful questions with it: .exists(), .suffix, .stem, .parent, .read_text().

CSV or JSON

CSV is a table: rows of the same fields, readable by a spreadsheet, the lingua franca of data people. It has no types — everything you read is a string until you convert it — and no way to nest anything.
JSON holds nesting and distinguishes numbers, booleans and null. It is the right answer for a record with structure inside it, and the wrong answer for a million flat rows someone wants to open in Excel.

Use csv.DictReader rather than indexing by column number: a file whose columns move should not silently change what your program means.

Encoding, briefly

A file is bytes; text is bytes plus an encoding. Passing encoding="utf-8" explicitly makes your program behave the same on every machine, rather than inheriting whatever the operating system prefers. Most “it works on my laptop” bugs in file handling are this.

Common misconceptions

“Reading a CSV gives numbers.” It gives strings. Convert deliberately, and decide what an empty cell means.
“JSON is a Python dict.” It is a text format. json.load produces dicts and lists; tuples, sets and dates do not survive the round trip.
“Read the whole file, then process.” Fine for a small file, fatal for a large one. Iterating the handle gives one line at a time.
“A missing file is an exception, so the program should stop.” Sometimes. That is a decision, and the previous outcome is how you make it.

Notes for teachers

Give a CSV with a missing value, a number stored as text and a row with too many fields, and ask for a program that reports what it skipped and why. The report is the evidence: it forces every decision about bad data to be made deliberately rather than by crash.

Resources

Resource	Kind	Cost	Provider
The Python Tutorial: input and output, including reading and writing files	reading	free	Python Software Foundation
pathlib: object-oriented filesystem paths	reference	free	Python Software Foundation
csv: CSV file reading and writing	reference	free	Python Software Foundation
json: JSON encoder and decoder	reference	free	Python Software Foundation
Automate the Boring Stuff, chapter 9: reading and writing files	reading	free	Al Sweigart

Courses that include it

draft

Python for a Coding Assessment

A refresher path through the Python graph for someone who has written Python before and has an assessment coming: run and read, rebuild the core, then work the way an assessment expects.

Prove it

Evidence of mastery

Reads a CSV into records, transforms them, and writes the result to a new file without corrupting the original.
Uses a `with` block for every file it opens, and explains what that guarantees over calling close.
Builds paths that work on Windows and Unix alike, rather than joining strings.
Chooses between CSV and JSON for a given dataset and says why, including what each cannot represent.
Handles a missing file and a malformed row without stopping the whole run, where that is the right behaviour.

Assessments that measure it

project

Python Take-Home Task

A timed take-home in the shape real hiring uses: read messy data, compute a result, test it, and hand over a repository someone else can run.

Credentials that include it