Read a File Line by Line in Python (Efficient File Handling Guide)

When working with large text files, it’s often inefficient to load the entire content into memory at once. Instead, Python allows you to process files line by line, making your code both memory-efficient and easier to manage. This technique is especially useful when dealing with logs, datasets, or any file where you need to analyze or process information incrementally.

Why Read Files Line by Line?

• Efficiency: Only one line is stored in memory at a time.

• Scalability: Works well for large files that may not fit in RAM.

• Flexibility: You can break or stop the reading loop when you find the data you need.

Using a for Loop with a File Object

The simplest and most Pythonic way to read a file line by line is by iterating directly over the file object.

# Open file in read mode
with open("example.txt", "r") as file:
    for line in file:
        print(line.strip())  # strip() removes trailing newline characters

Explanation:

• open("example.txt", "r") opens the file in read mode.

• The with statement ensures the file is closed automatically.

• Iterating over the file object returns each line one by one.

• strip() cleans up extra whitespace and newline characters.

Using readline()

The readline() method reads one line at a time, allowing you to control the reading flow manually.

with open("example.txt", "r") as file:
    line = file.readline()
    while line:
        print(line.strip())
        line = file.readline()

When to use:

• If you want fine-grained control over the reading process.

• If you need to add custom logic between reading lines.

Using readlines() with Caution

The readlines() method loads all lines into a list. While convenient, it may not be suitable for very large files.

with open("example.txt", "r") as file:
    lines = file.readlines()
    for line in lines:
        print(line.strip())

Downside: Not memory efficient — avoid using this on files that are several gigabytes in size.

Best Practices

1. Use a for loop for memory efficiency and readability.

2. Always use a with statement to ensure proper file closure.

3. Use strip() to clean up unwanted newline characters.

4. Avoid readlines() for large files — prefer iteration instead.

Practical Example

Suppose you have a log file (logs.txt) containing server events, and you want to extract only the lines containing the word ERROR.

with open("logs.txt", "r") as file:
    for line in file:
        if "ERROR" in line:
            print(line.strip())

This approach ensures you can scan through even massive log files without overloading your system’s memory.

By practicing this exercise, you’ll gain confidence in handling text files of varying sizes and learn the importance of efficiency in real-world file processing.