Before we get started, I also want to clarify what we won’t be covering here. Chapter 4 briefly previewed Unicode strings and files—tools for dealing with non-ASCII text. Unicode is a key tool for some programmers, especially those who work in the Internet domain. It can pop up, for example, in web pages, email content and headers, FTP transfers, GUI APIs, directory tools, and HTML, XML and JSON text.
At the same time, Unicode can be a heavy topic for programmers just starting out, and many (or most) of the Python programmers I meet today still do their jobs in blissful ignorance of the entire topic. In light of that, this book relegates most of the Unicode story to Chapter 37 of its Advanced Topics part as optional reading, and focuses on string basics here.
That is, this chapter tells only part of the string story in Python—the part that most scripts use and most programmers need to know. It explores the fundamental str string type, which handles ASCII text, and works the same regardless of which version of Python you use. Despite this intentionally limited scope, because str also handles Unicode in Python 3.X, and the separate unicode type works almost identically to str in 2.X, everything we learn here will apply directly to Unicode processing too.
For readers who do care about Unicode, I’d like to also provide a quick summary of its impacts and pointers for further study. From a formal perspective, ASCII is a simple form of Unicode text, but just one of many possible encodings and alphabets. Text from non-English-speaking sources may use very different letters, and may be encoded very differently when stored in files.
As we saw in Chapter 4, Python addresses this by distinguishing between text and binary data, with distinct string object types and file interfaces for each. This support varies per Python line:
In Python 3.X there are three string types: str is used for Unicode text (including ASCII), bytes is used for binary data (including encoded text), and bytearray is a mutable variant of bytes. Files work in two modes: text, which represents content as str and implements Unicode encodings, and binary, which deals in raw bytes and does no data translation.
In Python 2.X, unicode strings represent Unicode text, str strings handle both 8-bit text and binary data, and bytearray is available in 2.6 and later as a back-port from 3.X. Normal files’ content is simply bytes represented as str, but a codecs module opens Unicode text files, handles encodings, and represents content as unicode objects.
Despite such version differences, if and when you do need to care about Unicode you’ll find that it is a relatively minor extension—once text is in memory, it’s a Python string of characters that supports all the basics we’ll study in this chapter. In fact, the primary distinction of Unicode often lies in the translation (a.k.a. encoding) step required to move it to and from files. Beyond that, it’s largely just string processing.
Again, though, because most programmers don’t need to come to grips with Unicode details up front, I’ve moved most of them to Chapter 37. When you’re ready to learn about these more advanced string concepts, I encourage you to see both their preview in Chapter 4 and the full Unicode and bytes disclosure in Chapter 37 after reading the string fundamentals material here.
For this chapter, we’ll focus on the basic string type and its operations. As you’ll find, the techniques we’ll study here also apply directly to the more advanced string types in Python’s toolset.