File objects are Python code’s main interface to external files on your computer. They can be used to read and write text memos, audio clips, Excel documents, saved email messages, and whatever else you happen to have stored on your machine. Files are a core type, but they’re something of an oddball—there is no specific literal syntax for creating them. Rather, to create a file object, you call the built-in open function, passing in an external filename and an optional processing mode as strings.
For example, to create a text output file, you would pass in its name and the 'w' processing mode string to write data:
>>>f = open('data.txt', 'w')# Make a new file in output mode ('w' is write) >>>f.write('Hello\n')# Write strings of characters to it 6 >>>f.write('world\n')# Return number of items written in Python 3.X 6 >>>f.close()# Close to flush output buffers to disk
This creates a file in the current directory and writes text to it (the filename can be a full directory path if you need to access a file elsewhere on your computer). To read back what you just wrote, reopen the file in 'r' processing mode, for reading text input—this is the default if you omit the mode in the call. Then read the file’s content into a string, and display it. A file’s contents are always a string in your script, regardless of the type of data the file contains:
>>>f = open('data.txt')# 'r' (read) is the default processing mode >>>text = f.read()# Read entire file into a string >>>text'Hello\nworld\n' >>>print(text)# print interprets control characters Hello world >>>text.split()# File content is always a string ['Hello', 'world']
Other file object methods support additional features we don’t have time to cover here. For instance, file objects provide more ways of reading and writing (read accepts an optional maximum byte/character size, readline reads one line at a time, and so on), as well as other tools (seek moves to a new file position). As we’ll see later, though, the best way to read a file today is to not read it at all—files provide an iterator that automatically reads line by line in for loops and other contexts:
>>> for line in open('data.txt'): print(line)
We’ll meet the full set of file methods later in this book, but if you want a quick preview now, run a dir call on any open file and a help on any of the method names that come back:
>>>dir(f)[...many names omitted...'buffer', 'close', 'closed', 'detach', 'encoding', 'errors', 'fileno', 'flush', 'isatty', 'line_buffering', 'mode', 'name', 'newlines', 'read', 'readable', 'readline', 'readlines', 'seek', 'seekable', 'tell', 'truncate', 'writable', 'write', 'writelines'] >>>help(f.seek)...try it and see...
The prior section’s examples illustrate file basics that suffice for many roles. Technically, though, they rely on either the platform’s Unicode encoding default in Python 3.X, or the 8-bit byte nature of files in Python 2.X. Text files always encode strings in 3.X, and blindly write string content in 2.X. This is irrelevant for the simple ASCII data used previously, which maps to and from file bytes unchanged. But for richer types of data, file interfaces can vary depending on both content and the Python line you use.
As hinted when we met strings earlier, Python 3.X draws a sharp distinction between text and binary data in files: text files represent content as normal str strings and perform Unicode encoding and decoding automatically when writing and reading data, while binary files represent content as a special bytes string and allow you to access file content unaltered. Python 2.X supports the same dichotomy, but doesn’t impose it as rigidly, and its tools differ.
For example, binary files are useful for processing media, accessing data created by C programs, and so on. To illustrate, Python’s struct module can both create and unpack packed binary data—raw bytes that record values that are not Python objects—to be written to a file in binary mode. We’ll study this technique in detail later in the book, but the concept is simple: the following creates a binary file in Python 3.X (binary files work the same in 2.X, but the “b” string literal prefix isn’t required and won’t be displayed):
>>>import struct>>>packed = struct.pack('>i4sh', 7, b'spam', 8)# Create packed binary data >>>packed# 10 bytes, not objects or text b'\x00\x00\x00\x07spam\x00\x08' >>> >>>file = open('data.bin', 'wb')# Open binary output file >>>file.write(packed)# Write packed binary data 10 >>>file.close()
Reading binary data back is essentially symmetric; not all programs need to tread so deeply into the low-level realm of bytes, but binary files make this easy in Python:
>>>data = open('data.bin', 'rb').read()# Open/read binary data file >>>data# 10 bytes, unaltered b'\x00\x00\x00\x07spam\x00\x08' >>>data[4:8]# Slice bytes in the middle b'spam' >>>list(data)# A sequence of 8-bit bytes [0, 0, 0, 7, 115, 112, 97, 109, 0, 8] >>>struct.unpack('>i4sh', data)# Unpack into objects again (7, b'spam', 8)
Text files are used to process all sorts of text-based data, from memos to email content to JSON and XML documents. In today’s broader interconnected world, though, we can’t really talk about text without also asking “what kind?”—you must also know the text’s Unicode encoding type if either it differs from your platform’s default, or you can’t rely on that default for data portability reasons.
Luckily, this is easier than it may sound. To access files containing non-ASCII Unicode text of the sort introduced earlier in this chapter, we simply pass in an encoding name if the text in the file doesn’t match the default encoding for our platform. In this mode, Python text files automatically encode on writes and decode on reads per the encoding scheme name you provide. In Python 3.X:
>>>S ='sp\xc4m'# Non-ASCII Unicode text >>>S'spÄm' >>>S[2]# Sequence of characters 'Ä' >>>file = open('unidata.txt', 'w', encoding='utf-8')# Write/encode UTF-8 text >>>file.write(S)# 4 characters written 4 >>>file.close()>>>text = open('unidata.txt', encoding='utf-8').read()# Read/decode UTF-8 text >>>text'spÄm' >>>len(text)# 4 chars (code points) 4
This automatic encoding and decoding is what you normally want. Because files handle this on transfers, you may process text in memory as a simple string of characters without concern for its Unicode-encoded origins. If needed, though, you can also see what’s truly stored in your file by stepping into binary mode:
>>>raw = open('unidata.txt', 'rb').read()# Read raw encoded bytes >>>rawb'sp\xc3\x84m' >>>len(raw)# Really 5 bytes in UTF-8 5
You can also encode and decode manually if you get Unicode data from a source other than a file—parsed from an email message or fetched over a network connection, for example:
>>>text.encode('utf-8')# Manual encode to bytes b'sp\xc3\x84m' >>>raw.decode('utf-8')# Manual decode to str 'spÄm'
This is also useful to see how text files would automatically encode the same string differently under different encoding names, and provides a way to translate data to different encodings—it’s different bytes in files, but decodes to the same string in memory if you provide the proper encoding name:
>>>text.encode('latin-1')# Bytes differ in others b'sp\xc4m' >>>text.encode('utf-16')b'\xff\xfes\x00p\x00\xc4\x00m\x00' >>>len(text.encode('latin-1')), len(text.encode('utf-16'))(4, 10) >>>b'\xff\xfes\x00p\x00\xc4\x00m\x00'.decode('utf-16')# But same string decoded 'spÄm'
This all works more or less the same in Python 2.X, but Unicode strings are coded and display with a leading “u,” byte strings don’t require or show a leading “b,” and Unicode text files must be opened with codecs.open, which accepts an encoding name just like 3.X’s open, and uses the special unicode string to represent content in memory. Binary file mode may seem optional in 2.X since normal files are just byte-based data, but it’s required to avoid changing line ends if present (more on this later in the book):
>>>import codecs>>>codecs.open('unidata.txt', encoding='utf8').read()# 2.X: read/decode text u'sp\xc4m' >>>open('unidata.txt', 'rb').read()# 2.X: read raw bytes 'sp\xc3\x84m' >>>open('unidata.txt').read()# 2.X: raw/undecoded too 'sp\xc3\x84m'
Although you won’t generally need to care about this distinction if you deal only with ASCII text, Python’s strings and files are an asset if you deal with either binary data (which includes most types of media) or text in internationalized character sets (which includes most content on the Web and Internet at large today). Python also supports non-ASCII file names (not just content), but it’s largely automatic; tools such as walkers and listers offer more control when needed, though we’ll defer further details until Chapter 37.
The open function is the workhorse for most file processing you will do in Python. For more advanced tasks, though, Python comes with additional file-like tools: pipes, FIFOs, sockets, keyed-access files, persistent object shelves, descriptor-based files, relational and object-oriented database interfaces, and more. Descriptor files, for instance, support file locking and other low-level tools, and sockets provide an interface for networking and interprocess communication. We won’t cover many of these topics in this book, but you’ll find them useful once you start programming Python in earnest.