Python Essentials: Persistent Data and the Magic of File I/O - Part 6

Tags: Python · FileIO · CSV · Programming · Automation

---

In my previous post, I focused on protecting my code with unit tests — writing checks that catch bugs before they reach real users. But after working through those exercises, I noticed a frustrating pattern that no amount of testing could fix: every time my program ended, all my data disappeared.

Run the program, build some data, close the terminal — gone. Every time. That nagging problem is exactly what this post is about.

---

Why Does Data Disappear in the First Place?

To understand the fix, it helps to understand why this happens.

When your Python program runs, it stores all your variables — your lists, dictionaries, strings — in your computer's RAM (Random Access Memory). RAM is blazing fast, which is why Python uses it. But RAM is also temporary. It only holds data while your computer is actively powered and that program is running. The moment your script ends, the operating system reclaims that memory and wipes it clean.

Think of RAM like a whiteboard: perfect for scribbling ideas mid-meeting, but the cleaners wipe it down every night.

To make data survive beyond a single run, you need to write it to your computer's Hard Drive (or SSD) — permanent storage that keeps its contents even when the power is off. This act of moving data from the temporary whiteboard to permanent storage is called File I/O, short for File Input/Output.

• Input = reading data from a file into your program
• Output = writing data from your program into a file

With this mental model in place, let's start writing data that actually sticks.

---

Step 1 — Reading a File the Manual Way

The most common format for storing simple structured data is CSV, which stands for Comma Separated Values. A CSV file is just a plain text file where each line is one record, and the values within that record are separated by commas. Here's what a names.csv file might look like:

name,email,house
Harry,harry@hogwarts.edu,Gryffindor
Draco,draco@hogwarts.edu,Slytherin

Python's built-in open() function lets you read this file line by line. Here's the most direct approach:

students_list = []

with open("names.csv", "r") as file:
    for line in file:
        name, email, house = line.strip().split(",")
        students_list.append({"name": name, "email": email, "house": house})

Let's break down what's happening:

• open("names.csv", "r") opens the file in read mode ("r").
• The with keyword is important — it automatically closes the file once you're done, even if an error occurs. Always use with when working with files.
• line.strip() removes the invisible newline character (\n) at the end of each line.
• .split(",") breaks the cleaned string into a list of values using the comma as a separator.

This works — until it doesn't.

The hidden flaw: What if one of the values itself contains a comma? Imagine storing an address: "Kamran, Muhalla Mughalpura, Chak Jhumra". Calling .split(",") on that line would shatter your single address into three separate, meaningless pieces. Your program would crash or silently produce wrong data — which is often worse than crashing.

This is a classic edge case, and it's exactly the kind of thing that separates robust code from fragile code. Thankfully, there's a well-built tool to handle it.

---

Step 2 — Using the csv Library Like a Professional

Python's standard library includes a csv module designed specifically to handle the messy realities of CSV parsing — including values with commas, quotes, and other special characters. Instead of splitting strings yourself, you hand the job to a tool that's been tested and refined for years.

Option A: csv.reader

csv.reader gives you each row as a list, where you access values by their position (index):

import csv

students_list = []

with open("names.csv", "r") as file:
    reader = csv.reader(file)
    next(reader)  # Skip the header row manually
    for row in reader:
        students_list.append({"name": row[0], "email": row[1], "house": row[2]})

This is cleaner than manual splitting, but notice the fragility: row[0], row[1], row[2]. Your code is tightly coupled to the order of columns. If anyone ever reorders the columns in the CSV, your code silently reads the wrong data.

Option B: csv.DictReader — The Better Choice

csv.DictReader gives you each row as a dictionary, automatically using the header row as keys:

import csv

students_list = []

with open("names.csv", "r") as file:
    reader = csv.DictReader(file)
    for row in reader:
        students_list.append({"name": row["name"], "email": row["email"], "house": row["house"]})

Now your code accesses values by name, not position. If someone reorders the columns, it simply doesn't matter — row["name"] will always find the name column, wherever it lives.

Here's a side-by-side comparison to make the difference concrete:

The takeaway: Default to csv.DictReader. The slight extra verbosity pays for itself in code that's easier to read and harder to break.

---

Step 3 — Writing Data Back to a File

Reading is only half the story. Once your program processes some data, you'll often want to save the results back to a file. You do this with the same open() function, but in write mode:

import csv

students_list = [
    {"name": "Harry", "email": "harry@hogwarts.edu", "house": "Gryffindor"},
    {"name": "Draco", "email": "draco@hogwarts.edu", "house": "Slytherin"},
]

with open("output.csv", "w", newline="") as file:
    fieldnames = ["name", "email", "house"]
    writer = csv.DictWriter(file, fieldnames=fieldnames)

    writer.writeheader()       # Writes the "name,email,house" header line
    writer.writerows(students_list)  # Writes all the data rows

A few things to note:

• "w" mode overwrites the file if it already exists. Use "a" (append mode) if you want to add to an existing file instead.
• newline="" prevents Python from adding extra blank lines between rows on Windows — a small but important detail.
• csv.DictWriter mirrors DictReader: it takes dictionaries and writes them as properly formatted CSV rows, handling commas and quotes for you automatically.

---

Step 4 — Sorting Your Data with Lambda Functions

Now that you can load a list of student dictionaries, you'll likely want to sort them — alphabetically by name, perhaps. But calling .sort() directly on a list of dictionaries doesn't work, because Python doesn't know which field inside the dictionary to compare.

This is where lambda functions come in. A lambda is a small, anonymous, one-line function. You pass it as the key argument to tell Python how to evaluate each item for sorting:

# Sort students alphabetically by name
for student in sorted(students_list, key=lambda student: student["name"]):
    print(student["name"], student["house"])

Read it as: "For each student in the sorted list, sort by evaluating student["name"]."

You can easily sort by a different field by swapping the key:

# Sort by house instead
sorted(students_list, key=lambda student: student["house"])

Sorting a Dictionary's Own Contents with .items()

Sometimes you're not sorting a list of dictionaries — you have a single dictionary and want to sort its own key-value pairs. For example, imagine a dictionary of spending categories:

spending = {"Food": 1700, "Rent": 4500, "Transport": 800}

Dictionaries don't have a natural order you can sort, so you first convert them into a list of tuples using .items():

# Convert to list of tuples: [("Food", 1700), ("Rent", 4500), ("Transport", 800)]
for category, amount in sorted(spending.items(), key=lambda x: x[1], reverse=True):
    print(f"{category}: ${amount}")

A tuple is an immutable pair (or group) of values accessed by index: x[0] is the key, x[1] is the value. Here we sort by x[1] to rank spending from highest to lowest.

---

Step 5 — Going Beyond Text: Binary Files and Images

So far, all our files have been plain text — human-readable characters. But many of the most important file types are binary files: images, PDFs, audio files, compiled programs. These are stored as raw sequences of 0s and 1s with no direct human-readable meaning.

You can open a binary file in Python by using "rb" (read binary) or "wb" (write binary) mode:

with open("photo.jpg", "rb") as file:
    data = file.read()  # data is now a raw bytes object

But unless you're building a file format parser from scratch, you won't manipulate these bytes directly. Instead, you'll reach for a library — a collection of pre-written code that handles the complex binary logic for you. For images, that library is Pillow (also known as PIL, the Python Imaging Library).

from PIL import Image

# Open an image
img = Image.open("photo.jpg")

# Resize it
img_resized = img.resize((300, 300))

# Convert to grayscale
img_gray = img_resized.convert("L")

# Save the result
img_gray.save("photo_gray.jpg")

With four lines of readable code, you've resized and converted an image — work that would otherwise require understanding JPEG compression formats, pixel buffer manipulation, and binary arithmetic. The library abstracts all of that away.

---

Step 6 — How to Confidently Explore Any Library

Pillow, the requests library, pandas — third-party libraries can feel intimidating when you didn't write the code inside them. For a long time, I'd look at a library function, get back some object I'd never seen before, and freeze up.

The mindset shift that fixed this: everything in Python has a type, and once you know the type, you know how to work with it.

Python gives you three tools to interrogate any unfamiliar object:

import requests

response = requests.get("https://api.example.com/data")

type(response)   # → <class 'requests.models.Response'>
dir(response)    # → ['content', 'json', 'status_code', 'text', ...]
help(response)   # → Full documentation for the Response object

• type() tells you what the thing is
• dir() shows you all the methods and attributes available on it (the "buttons you can press")
• help() gives you the full documentation in your terminal

The question to always ask yourself when using a new library function is: "What does this return?" Once you know the return type — a string, a list, a dictionary, a custom object — you know exactly how to interact with it. You don't need to memorize library documentation; you need to know how to ask Python what it gave you.

---

Step 7 — Iterators vs. Lists: Why csv.reader Doesn't Crash on Huge Files

There's one more concept worth understanding, because it explains a design choice that might otherwise seem puzzling: why does csv.reader give you rows one at a time instead of loading the entire file at once?

The answer is the difference between a list and an iterator.

A list loads everything into memory upfront:

all_rows = [1, 2, 3, 4, 5]  # All 5 values sit in RAM simultaneously

An iterator produces one value at a time, only when asked:

reader = csv.reader(file)  # No rows loaded yet
for row in reader:          # One row loaded, processed, then discarded
    process(row)

Think of a list like buying all your groceries for the month at once and stacking them in your kitchen. An iterator is like having a delivery driver bring you exactly one item each time you need it.

For small files, this distinction doesn't matter. But imagine a CSV with 50 million rows — a common size in data engineering. Loading all 50 million rows into a Python list could consume gigabytes of RAM and crash your program. csv.reader as an iterator means only one row ever lives in memory at a time, regardless of how large the file is.

This is why csv.reader is so well-suited to real-world data work.

---

Putting It All Together

Here's a complete, practical example that combines everything from this post — reading a CSV with DictReader, processing the data, sorting it, and writing the result to a new file:

import csv

# Step 1: Read the input file
students = []
with open("students.csv", "r") as file:
    reader = csv.DictReader(file)
    for row in reader:
        students.append({"name": row["name"], "email": row["email"], "house": row["house"]})

# Step 2: Sort alphabetically by house, then by name within each house
students_sorted = sorted(students, key=lambda s: (s["house"], s["name"]))

# Step 3: Write the sorted result to a new file
with open("students_sorted.csv", "w", newline="") as file:
    writer = csv.DictWriter(file, fieldnames=["name", "email", "house"])
    writer.writeheader()
    writer.writerows(students_sorted)

print("Done! Sorted student list saved to students_sorted.csv")

Notice the tuple (s["house"], s["name"]) in the sort key — Python sorts tuples element by element, so this sorts primarily by house, and uses name as a tiebreaker within the same house. A neat trick that costs nothing extra.

---

Key Takeaways