How to read from file python

Python Read Text File

Summary: in this tutorial, you learn various ways to read text files in Python.

The following shows how to read all texts from the readme.txt file into a string:

Steps for reading a text file in Python

To read a text file in Python, you follow these steps:

1) open() function

The open() function has many parameters but you’ll be focusing on the first two:

The path_to_file parameter specifies the path to the text file.

If the program and file are in the same folder, you need to specify only the filename of the file. Otherwise, you need to include the path to the file as well as the filename.

To specify the path to the file, you use the forward-slash ( ‘/’ ) even if you’re working on Windows.

For example, if the file readme.txt is stored in the sample folder as the program, you need to specify the path to the file as c:/sample/readme.txt

The mode is an optional parameter. It’s a string that specifies the mode in which you want to open the file. The following table shows available modes for opening a text file:

For example, to open a file whose name is the-zen-of-python.txt stored in the same folder as the program, you use the following code:

The open() function returns a file object which you will use to read text from a text file.

2) Reading text methods

The file object provides you with three methods for reading text from a text file:

3) close() method

The file that you open will remain open until you close it using the close() method.

It’s important to close the file that is no longer in use for the following reasons:

The following shows how to call the close() method to close the file:

To close the file automatically without calling the close() method, you use the with statement like this:

In practice, you’ll use the with statement to close the file automatically.

Reading a text file examples

We’ll use the-zen-of-python.txt file for the demonstration.

The following example illustrates how to use the read() method to read all the contents of the the-zen-of-python.txt file into a string:

The following example uses the readlines() method to read the text file and returns the file contents as a list of strings:

The reason you see a blank line after each line from a file is that each line in the text file has a newline character (\n). To remove the blank line, you can use the strip() method. For example:

The following example shows how to use the readline() to read the text file line by line:

A more concise way to read a text file line by line

The open() function returns a file object which is an iterable object. Therefore, you can use a for loop to iterate over the lines of a text file as follows:

This is a more concise way to read a text file line by line.

Read UTF-8 text files

The code in the previous examples works fine with ASCII text files. However, if you’re dealing with other languages such as Japanese, Chinese, and Korean, the text file is not a simple ASCII text file. And it’s likely a UTF-8 file that uses more than just the standard ASCII text characters.

To open a UTF-8 text file, you need to pass the encoding=’utf-8′ to the open() function to instruct it to expect UTF-8 characters from the file.

Reading and Writing Files in Python

Author: PFB Staff Writer
Last Updated: December 3, 2021

Overview

When you’re working with Python, you don’t need to import a library in order to read and write to a file. It’s handled natively in the language, albeit in a unique manner. Below, we outline the simple steps to read and write to a file in Python.

The first thing you’ll need to do is use the built-in python open file function to get a file object.

The open function opens a file. It’s simple. This is the first step in reading and writing files in python.

When you use the open function, it returns something called a file object. File objects contain methods and attributes that can be used to collect information about the file you opened. They can also be used to manipulate said file.

For example, the mode attribute of a file object tells you which mode a file was opened in. And the name attribute tells you the name of the file.

You must understand that a file and file object are two wholly separate – yet related – things.

File Types

What you may know as a file is slightly different in Python.

In Windows, for example, a file can be any item manipulated, edited or created by the user/OS. That means files can be images, text documents, executables, and excel file and much more. Most files are organized by keeping them in individual folders.

A file In Python is categorized as either text or binary, and the difference between the two file types is important.

Text files are structured as a sequence of lines, where each line includes a sequence of characters. This is what you know as code or syntax.

Each line is terminated with a special character, called the EOL or End of Line character. There are several types, but the most common is the comma <,>or newline character. It ends the current line and tells the interpreter a new one has begun.

A backslash character can also be used, and it tells the interpreter that the next character – following the slash – should be treated as a new line. This character is useful when you don’t want to start a new line in the text itself but in the code.

A binary file is any type of file that is not a text file. Because of their nature, binary files can only be processed by an application that know or understand the file’s structure. In other words, they must be applications that can read and interpret binary.

Reading Files in Python

In Python, files are read using the open() method. This is one of Python’s built-in methods, made for opening files.

The open() function takes two arguments: a filename and a file opening mode. The filename points to the path of the file on your computer, while the file opening mode is used to tell the open() function how we plan to interact with the file.

By default, the file opening mode is set to read-only, meaning we’ll only have permission to open and examine the contents of the file.

On my computer is a folder called PythonForBeginners. In that folder are three files. One is a text file named emily_dickinson.txt, and the other two are python files: read.py and write.py.

The text file contains the following poem, written by poet Emily Dickinson. Perhaps we are working on a poetry program and have our poems stored as files on the computer.

Success is counted sweetest
By those who ne’er succeed.
To comprehend a nectar
Requires sorest need.

Not one of all the Purple Host
Who took the Flag today
Can tell the definition
So clear of Victory

As he defeated, dying
On whose forbidden ear
The distant strains of triumph
Burst agonized and clear.

Before we can do anything with the contents of the poem file, we’ll need to tell Python to open it. The file read.py, contains all the python code necessary to read the poem.

Any text editor can be used to write the code. I’m using the Atom code editor, which is my editor of choice for working in python.

This screenshot shows my setup in Atom.

I’ve used Python comments to explain each step in the code. Follow this link to learn more about what a Python comment is.

The example above illustrates how using a simple loop in Python can read the contents of a file.

When it comes to reading files, Python takes care of the heaving lifting behind the scenes. Run the script by navigating to the file using the Command Prompt — or Terminal — and typing ‘python’ followed by the name of the file.

Windows Users: Before you can use the python keyword in your Command Prompt, you’ll need to set up the environment variables. This should have happened automatically when you installed Python, but in case it didn’t, you may need to do it manually.

Data provided by the open() method is usually stored in a new variable. In this example, the contents of the poem are stored in the variable “myfile.”

Once the file is created, we can use a for loop to read every line in the file and print its contents to the command line.

This is a very simple example of how to open a file in Python, but student’s should be aware that the open() method is quite powerful. For some projects it will be the only thing needed to read and write files with Python.

Writing Files in Python

Before we can write to a file in Python, it must first be opened in a different file opening mode. We can do this by supplying the open() method with a special argument.

In Python, write to file using the open() method. You’ll need to pass both a filename and a special character that tells Python we intend to write to the file.

Add the following code to write.py. We’ll tell Python to look for a file named “sample.txt” and overwrite its contents with a new message.

Passing ‘w’ to the open() method tells Python to open the file in write mode. In this mode, any data already in the file is lost when the new data is written.

If the file doesn’t exist, Python will create a new file. In this case, a new file named “sample.txt” will be created when the program runs.

Run the program using the Command Prompt:

Python can also write multiple lines to a file. The easiest way to do this is with the writelines() method.

We can also write multiple lines to a file using special characters:

Using string concatenation makes it possible for Python to save text data in a variety of ways.

However, if we wanted to avoid overwriting the data in a file, and instead append it or change it, we’d have to open the file using another file opening mode.

File Opening Modes

By default, Python will open the file in read-only mode. If we want to do anything other than just read a file, we’ll need to manually tell Python what we intend to do with it.

Note: These examples assume the user is working with text file types. If the intention is to read or write to a binary file type, an additional argument must be passed to the open() method: the ‘b’ character.

Closing Files with Python

After opening a file in Python, it’s important to close it after you’re done with it. Closing a file ensures that the program can no longer access its contents.

Close a file with the close() method.

Opening Other File Types

The open() method can read and write many different file types. We’ve seen how to open binary files and text files. Python can also open images, allowing you to view and edit their pixel data.

Before Python can open an image file, the Pillow library (Python Imaging Library) must be installed. It’s easiest to install this module using pip.

With Pillow installed, Python can open image files and read their contents.

The Pillow library includes powerful tools for editing images. This has made it one of the most popular Python libraries.

With Statement

You can also work with file objects using the with statement. It is designed to provide much cleaner syntax and exceptions handling when you are working with code. That explains why it’s good practice to use the with statement where applicable.

One bonus of using this method is that any files opened will be closed automatically after you are done. This leaves less to worry about during cleanup.

To use the with statement to open a file:

Now that you understand how to call this statement, let’s take a look at a few examples.

You can also call upon other methods while using this statement. For instance, you can do something like loop over a file object:

You’ll also notice that in the above example we didn’t use the “file.close()” method because the with statement will automatically call that for us upon execution. It really makes things a lot easier, doesn’t it?

Splitting Lines in a Text File

As a final example, let’s explore a unique function that allows you to split the lines taken from a text file. What this is designed to do, is split the string contained in variable data whenever the interpreter encounters a space character.

But just because we are going to use it to split lines after a space character, doesn’t mean that’s the only way. You can actually split your text using any character you wish – such as a colon, for instance.

The code to do this (also using a with statement) is:

If you wanted to use a colon instead of a space to split your text, you would simply change line.split() to line.split(“:”).

The output for this will be:

The reason the words are presented in this manner is because they are stored – and returned – as an array. Be sure to remember this when working with the split function.

Conclusion

Reading and writing files in Python involves an understanding of the open() method. By taking advantage of this method’s versatility, it’s possible to read, write, and create files in Python.

Files Python can either be text files or binary files. It’s also possible to open and edit image data using the Pillow module.

Once a file’s data is loaded in Python, there’s virtually no end to what can be done with it. Programmers often work with a large number of files, using programs to generate them automatically.

As with any lesson, there is only so much that can be covered in the space provided. Hopefully you’ve learned enough to get started reading and writing files in Python.

How to read a text file into a list or an array with Python

I am trying to read the lines of a text file into a list or array in python. I just need to be able to individually access any item in the list or array after it is created.

The text file is formatted as follows:

I’m using the following code to try to read the file into a list:

The output I get is:

Apparently it is reading the entire file into a list of just one item, rather than a list of individual items. What am I doing wrong?

6 Answers 6

Trending sort

Trending sort is based off of the default sorting method — by highest score — but it boosts votes that have happened recently, helping to surface more up-to-date answers.

It falls back to sorting by highest score if no posts are trending.

Switch to Trending sort

You will have to split your string into a list of values using split()

EDIT: I didn’t realise there would be so much traction to this. Here’s a more idiomatic approach.

You can also use numpy loadtxt like

So you want to create a list of lists. We need to start with an empty list

next, we read the file content, line by line

A common use case is that of columnar data, but our units of storage are the rows of the file, that we have read one by one, so you may want to transpose your list of lists. This can be done with the following idiom

Another common use is to give a name to each column

so that you can operate on homogeneous data items

Update While in Python 2 zip(*list_of_lists) returns a different (transposed) list of lists, in Python 3 the situation has changed and zip(*list_of_lists) returns a zip object that is not subscriptable.

If you need indexed access you can use

that gives you a list of lists in both versions of Python.

On the other hand, if you don’t need indexed access and what you want is just to build a dictionary indexed by column names, a zip object is just fine.

Reading and Writing Files in Python (Guide)

Table of Contents

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Reading and Writing Files in Python

One of the most common tasks that you can do with Python is reading and writing files. Whether it’s writing to a simple text file, reading a complicated server log, or even analyzing raw byte data, all of these situations require reading or writing a file.

In this tutorial, you’ll learn:

This tutorial is mainly for beginner to intermediate Pythonistas, but there are some tips in here that more advanced programmers may appreciate as well.

Free Bonus: Click here to get our free Python Cheat Sheet that shows you the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

Take the Quiz: Test your knowledge with our interactive “Reading and Writing Files in Python” quiz. Upon completion you will receive a score so you can track your learning progress over time:

What Is a File?

Before we can go into how to work with files in Python, it’s important to understand what exactly a file is and how modern operating systems handle some of their aspects.

At its core, a file is a contiguous set of bytes used to store data. This data is organized in a specific format and can be anything as simple as a text file or as complicated as a program executable. In the end, these byte files are then translated into binary 1 and 0 for easier processing by the computer.

Files on most modern file systems are composed of three main parts:

File Paths

When you access a file on an operating system, a file path is required. The file path is a string that represents the location of a file. It’s broken up into three major parts:

Here’s a quick example. Let’s say you have a file located within a file structure like this:

Line Endings

One problem often encountered when working with file data is the representation of a new line or line ending. The line ending has its roots from back in the Morse Code era, when a specific pro-sign was used to communicate the end of a transmission or the end of a line.

Later, this was standardized for teleprinters by both the International Organization for Standardization (ISO) and the American Standards Association (ASA). ASA standard states that line endings should use the sequence of the Carriage Return ( CR or \r ) and the Line Feed ( LF or \n ) characters ( CR+LF or \r\n ). The ISO standard however allowed for either the CR+LF characters or just the LF character.

Windows uses the CR+LF characters to indicate a new line, while Unix and the newer Mac versions use just the LF character. This can cause some complications when you’re processing files on an operating system that is different than the file’s source. Here’s a quick example. Let’s say that we examine the file dog_breeds.txt that was created on a Windows system:

This same output will be interpreted on a Unix device differently:

This can make iterating over each line problematic, and you may need to account for situations like this.

Character Encodings

Another common problem that you may face is the encoding of the byte data. An encoding is a translation from byte data to human readable characters. This is typically done by assigning a numerical value to represent a character. The two most common encodings are the ASCII and UNICODE Formats. ASCII can only store 128 characters, while Unicode can contain up to 1,114,112 characters.

ASCII is actually a subset of Unicode (UTF-8), meaning that ASCII and Unicode share the same numerical to character values. It’s important to note that parsing a file with the incorrect character encoding can lead to failures or misrepresentation of the character. For example, if a file was created using the UTF-8 encoding, and you try to parse it using the ASCII encoding, if there is a character that is outside of those 128 values, then an error will be thrown.

Opening and Closing a File in Python

When you want to work with a file, the first thing to do is to open it. This is done by invoking the open() built-in function. open() has a single required argument that is the path to the file. open() has a single return, the file object:

After you open a file, the next thing to learn is how to close it.

Warning: You should always make sure that an open file is properly closed. To learn why, check out the Why Is It Important to Close Files in Python? tutorial.

It’s important to remember that it’s your responsibility to close the file. In most cases, upon termination of an application or script, a file will be closed eventually. However, there is no guarantee when exactly that will happen. This can lead to unwanted behavior including resource leaks. It’s also a best practice within Python (Pythonic) to make sure that your code behaves in a way that is well defined and reduces any unwanted behavior.

When you’re manipulating a file, there are two ways that you can use to ensure that a file is closed properly, even when encountering an error. The first way to close a file is to use the try-finally block:

If you’re unfamiliar with what the try-finally block is, check out Python Exceptions: An Introduction.

The second way to close a file is to use the with statement:

The with statement automatically takes care of closing the file once it leaves the with block, even in cases of error. I highly recommend that you use the with statement as much as possible, as it allows for cleaner code and makes handling any unexpected errors easier for you.

Other options for modes are fully documented online, but the most commonly used ones are the following:

Let’s go back and talk a little about file objects. A file object is:

“an object exposing a file-oriented API (with methods such as read() or write() ) to an underlying resource.” (Source)

There are three different categories of file objects:

Each of these file types are defined in the io module. Here’s a quick rundown of how everything lines up.

Text File Types

A text file is the most common file that you’ll encounter. Here are some examples of how these files are opened:

With these types of files, open() will return a TextIOWrapper file object:

Buffered Binary File Types

A buffered binary file type is used for reading and writing binary files. Here are some examples of how these files are opened:

With these types of files, open() will return either a BufferedReader or BufferedWriter file object:

Raw File Types

A raw file type is:

“generally used as a low-level building-block for binary and text streams.” (Source)

It is therefore not typically used.

Here’s an example of how these files are opened:

With these types of files, open() will return a FileIO file object:

Reading and Writing Opened Files

Once you’ve opened up a file, you’ll want to read or write to the file. First off, let’s cover reading a file. There are multiple methods that can be called on a file object to help you out:

The above example can also be done by using list() to create a list out of the file object:

Iterating Over Each Line in the File

However, the above examples can be further simplified by iterating over the file object itself:

This final approach is more Pythonic and can be quicker and more memory efficient. Therefore, it is suggested you use this instead.

Now let’s dive into writing files. As with reading files, file objects have multiple methods that are useful for writing to a file:

Working With Bytes

Sometimes, you may need to work with files using byte strings. This is done by adding the ‘b’ character to the mode argument. All of the same methods for the file object apply. However, each of the methods expect and return a bytes object instead:

Opening a text file using the b flag isn’t that interesting. Let’s say we have this cute picture of a Jack Russell Terrier ( jack_russell.png ):

A Full Example: dos2unix.py

Tips and Tricks

Now that you’ve mastered the basics of reading and writing files, here are some tips and tricks to help you grow your skills.

__file__

“the pathname of the file from which the module was loaded, if it was loaded from a file.” (Source

Note: To re-iterate, __file__ returns the path relative to where the initial Python script was called. If you need the full system path, you can use os.getcwd() to get the current working directory of your executing code.

Here’s a real world example. In one of my past jobs, I did multiple tests for a hardware device. Each test was written using a Python script with the test script file name used as a title. These scripts would then be executed and could print their status using the __file__ special attribute. Here’s an example folder structure:

Running main.py produces the following:

I was able to run and get the status of all my tests dynamically through use of the __file__ special attribute.

Appending to a File

Sometimes, you may want to append to a file or start writing at the end of an already populated file. This is easily done by using the ‘a’ character for the mode argument:

When you examine dog_breeds.txt again, you’ll see that the beginning of the file is unchanged and Beagle is now added to the end of the file:

Working With Two Files at the Same Time

There are times when you may want to read a file and write to another file at the same time. If you use the example that was shown when you were learning how to write to a file, it can actually be combined into the following:

Creating Your Own Context Manager

__enter__() is invoked when calling the with statement. __exit__() is called upon exiting from the with statement block.

Here’s a template that you can use to make your custom class:

Now that you’ve got your custom class that is now a context manager, you can use it similarly to the open() built-in:

Don’t Re-Invent the Snake

Additionally, there are built-in libraries out there that you can use to help you:

There are plenty more out there. Additionally there are even more third party tools available on PyPI. Some popular ones are the following:

You’re a File Wizard Harry!

You did it! You now know how to work with files with Python, including some advanced techniques. Working with files in Python should now be easier than ever and is a rewarding feeling when you start doing it.

In this tutorial you’ve learned:

If you have any questions, hit us up in the comments.

Take the Quiz: Test your knowledge with our interactive “Reading and Writing Files in Python” quiz. Upon completion you will receive a score so you can track your learning progress over time:

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Reading and Writing Files in Python

Get a short & sweet Python Trick delivered to your inbox every couple of days. No spam ever. Unsubscribe any time. Curated by the Real Python team.

About James Mertz

James is a passionate Python developer at NASA’s Jet Propulsion Lab who also writes on the side for Real Python.

Each tutorial at Real Python is created by a team of developers so that it meets our high quality standards. The team members who worked on this tutorial are:

Master Real-World Python Skills With Unlimited Access to Real Python

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

Master Real-World Python Skills
With Unlimited Access to Real Python

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

What Do You Think?

What’s your #1 takeaway or favorite thing you learned? How are you going to put your newfound skills to use? Leave a comment below and let us know.

Commenting Tips: The most useful comments are those written with the goal of learning from or helping out other students. Get tips for asking good questions and get answers to common questions in our support portal. Looking for a real-time conversation? Visit the Real Python Community Chat or join the next “Office Hours” Live Q&A Session. Happy Pythoning!

Related Tutorial Categories: intermediate python

Read a file line by line in Python

Prerequisites:

Python provides inbuilt functions for creating, writing, and reading files. There are two types of files that can be handled in python, normal text files and binary files (written in binary language, 0s, and 1s). In this article, we are going to study reading line by line from a file.

Method 1: Read a File Line by Line using readlines()

readlines() is used to read all the lines at a single go and then return them as each line a string element in a list. This function can be used for small files, as it reads the whole file content to the memory, then split it into separate lines. We can iterate over the list and strip the newline ‘\n’ character using strip() function.

Example:

Python3

Output:

Method 2: Read a File Line by Line using readline()

readline() function reads a line of the file and return it in the form of the string. It takes a parameter n, which specifies the maximum number of bytes that will be read. However, does not reads more than one line, even if n exceeds the length of the line. It will be efficient when reading a large file because instead of fetching all the data in one go, it fetches line by line. readline() returns the next line of the file which contains a newline character in the end. Also, if the end of the file is reached, it will return an empty string.

Example:

Python3

Output:

Method 3: Read a File Line by Line using for loop

An iterable object is returned by open() function while opening a file. This final way of reading a file line-by-line includes iterating over a file object in for a loop. In doing this we are taking advantage of a built-in Python function that allows us to iterate over the file object implicitly using a for loop in a combination with using the iterable object. This approach takes fewer lines of code, which is always the best practice worthy of following.

Example:

Python3

Output:

Method 4: Read a File Line by Line using for loop and list comprehension

A list comprehension consists of brackets containing the expression, which is executed for each element along with the for loop to iterate over each element. Here, we will read the text file and print the raw data including the new line character in another output we removed all the new line characters from the list.