Python UnicodeError – Tutorial with Examples

Unicode is a computing industry standard used to represent text in almost every modern software application. Python uses a Unicode string type. It is designed to support all characters available in Unicode. Unicode is an essential part of the Python programming language, but sometimes you may face challenges when working with it. One common error you may encounter is the UnicodeError.

This article will explain what this error is and why you may encounter it. We will also look at examples of how to solve this error in different scenarios.

What is UnicodeError?

The UnicodeError indicates that there are problems with Unicode-related processing. This error will arise when you try to encode, decode or print a string that is not capable of being processed as Unicode. It is an exception that usually occurs unexpectedly, and when it does occur, it halts your program and raises an error message.

The UnicodeError is raised when an error occurs during Unicode-related operations like encoding or decoding bytes. Various reasons can cause this issue. Some of them include:

• Attempting to encode or decode invalid or incomplete byte sequences.
• Trying to concatenate strings with different encodings.
• Attempting to write to a file with an unsupported encoding.
• Trying to print non-ASCII characters to the output console in the terminal.

Examples of UnicodeError and How to Fix Them

We will now look at some examples of UnicodeError and see how to solve them.

Example 1: Encountering UnicodeError when reading a file with an unsupported encoding

Assume you have a file encoded in a different character set that is not compatible with the default Python encoding. Python will try to read the file using the default ASCII encoding, and you will get a UnicodeError.

Let’s create a sample file containing non-ASCII characters that can cause an exception.

content = 'This is a sample file with non-ASCII characters: Måløy.'

# write the content in a file with encoding='iso-8859-1'
with open("testfile.txt", "w", encoding="iso-8859-1") as f:
    f.write(content)

# read the file with the default ASCII encoding
with open("testfile.txt") as f:
    text = f.read()

print(text)

When we run the above code, we get the UnicodeDecodeError with the following exception message:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 39: invalid continuation byte

The error message indicates that there is an invalid byte sequence that Python could not decode as UTF-8. It is unable to read the sample file using ASCII encoding since it contains non-ASCII characters.

To solve this error, you can specify the encoding while opening the file in read mode, using the same encoding it was written with.

with open("testfile.txt", encoding="iso-8859-1") as f:
    text = f.read()

print(text)

The above code will specify the encoding to be iso-8859-1. Now when we run the code, the file is opened correctly without any errors.

Example 2: Concatenating strings with different encodings

When concatenating strings with different encodings, you may end up causing a UnicodeError. For example:

string1 = "東京"
string2 = b"\xe3\x81\x9f\xe3\x81\x8c\xe3\x81\x84"

# concatenate the byte string with a unicode string
print(string1 + string2)

In the above code, we define two strings, one in Unicode and the other in bytes. We then try to concatenate them using the + operator. When we run the code, we’ll get an UnicodeDecodeError with the following message:

TypeError: can't concat str to bytes

The TypeError message shows that you cannot concatenate a byte string with a unicode string. The way to solve this is to encode the Unicode string in the same encoding as the byte string:

string1 = "東京"
string2 = b"\xe3\x81\x9f\xe3\x81\x8c\xe3\x81\x84"

# encode the Unicode string to UTF-8 bytes
encoded_string1 = string1.encode("utf-8")

# concatenate the byte string with the encoded Unicode string
print(encoded_string1 + string2)

The above code will encode the Unicode string to UTF-8 bytes before concatenating it with the bytes string. Now, when we run the code, both strings concatenate correctly without any errors.

Example 3: Printing non-ASCII characters to the output console

When printing non-ASCII characters to the console output, you may get a UnicodeError. For example:

print("Hérè herè")

When we run this code, we’ll get a UnicodeEncodeError with the following error message:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 4-7: ordinal not in range(128)

The error message means that the terminal/console cannot handle the non-ASCII characters. The solution to this error is to force the print statement to use encoding that supports non-ASCII characters like UTF-8 or Unicode.

import sys
sys.stdout.buffer.write('Hérè herè\n'.encode('utf8'))

The above code will force the print statement to use UTF-8 encoding. The sys.stdout.buffer.write command writes the string to the output stream, which supports UTF-8 characters, and the \n character is used to add a newline.

Conclusion

This article has provided an overview of the UnicodeError, its causes, and examples of how to solve it. Unicode is an essential part of Python, but it can be challenging to work with in certain scenarios. By following the solutions to the examples listed here, you can handle the UnicodeError exception and avoid any errors in your Python programs.