How Do You Write a Python Program to Process a Large File?

Problem scenario
You have a file that is too big to fit into memory. How can Python read the file?

Possible Solution #1
You can use the "open" method and the "read()" or "readlines()" methods. Avoid using read() because that will read the entire file into memory. The readline() and readlines() methods can be given a byte number as an argument. They will retrieve only a portion of the file. Alternatively the file can be processed in a for loop line-by-line.

For more information on the implementation, see these postings:

https://www.w3schools.com/python/ref_file_readline.asp
https://www.w3schools.com/python/ref_file_readlines.asp
https://www.journaldev.com/14408/python-read-file-open-write-delete-copy

Possible Solution #2
Add memory to your system. If you need assistance, see this posting.

Leave a comment

Your email address will not be published. Required fields are marked *