Python UnicodeEncodeError – Tutorial with Examples

Unicode is a universal character encoding standard that includes almost every character and script used in modern computing. When we print or display Unicode data, sometimes we may encounter an error that says "UnicodeEncodeError: 'charmap' codec can't encode character". This error occurs when Python is unable to map the given character to the specified encoding while trying to convert the string to a byte string. In this tutorial, we will discuss what UnicodeEncodeError is in Python, its causes, and how to solve it with examples.

Prerequisites

To follow this tutorial, you should have a basic understanding of Python programming language, and be familiar with concepts such as string manipulation, encoding, decoding, and file I/O.

What is UnicodeEncodeError?

When we print or display strings that contain non-ASCII characters, Python has to convert the string from Unicode to byte string by a process called encoding. This process requires us to specify an encoding standard or a codec, such as 'utf-8', 'ascii', or 'iso-8859-1'. Sometimes, when we try to encode a string with a certain codec, we may encounter an <code>UnicodeEncodeError</code> that says "charmap codec can't encode character". This error occurs when Python is unable to map the given character to the specified encoding, which could be due to a wrong encoding, a character not supported by the encoding, or an invalid byte sequence.

Examples

Let's take some examples to understand the <code>UnicodeEncodeError</code> better.

Example 1: Non-ASCII characters with ascii() function

The <code>ascii()</code> function in Python returns a string containing a printable representation of an object. This function replaces non-ASCII characters with escape sequences, such as '\uXXXX'. When we pass a string that contains non-ASCII characters to the <code>ascii()</code> function, we may encounter a <code>UnicodeEncodeError</code> that says "charmap codec can't encode character", because the ascii codec cannot encode non-ASCII characters.

# Example 1: Non-ASCII characters with ascii() function

string = "Hello, 世界"
print(ascii(string))  # UnicodeEncodeError: 'charmap' codec can't encode character '\u4e16' in position 7: character maps to <undefined>

In the above example, the string contains non-ASCII character '世' (U+4E16) which is not supported by the ascii codec. Therefore, the <code>ascii()</code> function raises a <code>UnicodeEncodeError</code> with the message "charmap codec can't encode character '\u4e16' in position 7: character maps to &lt;undefined&gt;".

Example 2: Non-ASCII characters with str.encode() method

The <code>str.encode()</code> method in Python returns an encoded version of the string. This method requires us to pass an encoding standard or a codec as an argument. When we pass a string that contains non-ASCII characters to the <code>str.encode()</code> method with a codec that cannot encode the characters, we may encounter a <code>UnicodeEncodeError</code> that says "charmap codec can't encode character", because the given codec cannot map the characters to its byte representation. 

# Example 2: Non-ASCII characters with str.encode() method

string = "Hello, 世界"
encoded_string = string.encode("ascii")  # UnicodeEncodeError: 'ascii' codec can't encode character '\u4e16' in position 7: ordinal not in range(128)

In the above example, we are trying to encode the non-ASCII characters of the string with the ascii codec, which can only encode ASCII characters. Therefore, the <code>str.encode()</code> method raises a <code>UnicodeEncodeError</code> with the message "'ascii' codec can't encode character '\u4e16' in position 7: ordinal not in range(128)".

Example 3: Invalid byte sequence with str.encode() method

When we pass a string that contains an invalid byte sequence to the <code>str.encode()</code> method, we may encounter a <code>UnicodeEncodeError</code> that says "charmap codec can't encode character", because the specified codec cannot map the invalid byte sequence to a character. An invalid byte sequence is a sequence of bytes that does not represent any character in the specified encoding.

# Example 3: Invalid byte sequence with str.encode() method

string = "São Paulo"
encoded_string = string.encode("ascii")  # UnicodeEncodeError: 'ascii' codec can't encode character '\xe3' in position 1: ordinal not in range(128)

In the above example, the string contains an invalid byte sequence '\xe3' which represents the character 'ã' in Latin-1 encoding but not in ascii encoding. Therefore, when we try to encode the string with the ascii codec, the <code>str.encode()</code> method raises a <code>UnicodeEncodeError</code> with the message "'ascii' codec can't encode character '\xe3' in position 1: ordinal not in range(128)".

How to Solve UnicodeEncodeError?

There are several ways to solve the <code>UnicodeEncodeError</code> in Python. Some of them are:

• Specify a suitable encoding standard or a codec that can encode the given characters.
• Encode the string with Unicode encoding, such as ‘utf-8’, that can handle almost all characters.
• Use the replace() method to replace unencodable characters with a suitable replacement character.
• Use the ignore() method to ignore unencodable characters.
• Use the backslashreplace() method to replace unencodable characters with a backslashed escape sequence.
• Handle the error with a try-except block.

Example 4: Specifying a suitable encoding

We can solve the <code>UnicodeEncodeError</code> by specifying a suitable encoding standard or a codec that can encode the given characters. For example, if the string contains non-ASCII characters, we can use 'utf-8' or 'iso-8859-1' encoding instead of 'ascii', which can only encode ASCII characters. Alternatively, we can try using other encodings, such as 'latin-1', 'cp1252', etc., depending on the system and the application.

# Example 4: Specifying a suitable encoding

string = "Hello, 世界"
encoded_string = string.encode("utf-8")
print(encoded_string)  # b'Hello, \\xe4\\xb8\\x96\\xe7\\x95\\x8c'

In the above example, we are encoding a string that contains non-ASCII characters with 'utf-8' encoding, which can encode any Unicode character. Therefore, the encoded string is produced without raising a UnicodeEncodeError. Note that the output is in byte string format, meaning that we need to decode it back to Unicode string before using it further.

Example 5: Replacing unencodable characters with replace() method

We can use the <code>replace()</code> method to replace unencodable characters with a suitable replacement character. The <code>replace()</code> method replaces all occurrences of a string with another string.

# Example 5: Replacing unencodable characters with replace() method

string = "São Paulo"
encoded_string = string.encode("ascii", errors="replace")  # b'S?o Paulo'
decoded_string = encoded_string.decode("ascii")
print(decoded_string)  # S?o Paulo

In the above example, we are trying to encode a string with 'ascii' encoding, which cannot encode the character 'ã'. Therefore, we are using the <code>replace()</code> method to replace the unencodable character with a question mark '?'. The output encoded string is produced without raising a UnicodeEncodeError. Note that the output may not be accurate as the replaced character may not represent the original character correctly.

Example 6: Ignoring unencodable characters with ignore() method

We can use the <code>ignore()</code> method to ignore unencodable characters. The <code>ignore()</code> method removes all occurrences of a string from another string.

# Example 6: Ignoring unencodable characters with ignore() method

string = "São Paulo"
encoded_string = string.encode("ascii", errors="ignore")  # b'So Paulo'
decoded_string = encoded_string.decode("ascii")
print(decoded_string)  # So Paulo

In the above example, we are trying to encode a string with 'ascii' encoding, which cannot encode the character 'ã'. Therefore, we are using the <code>ignore()</code> method to remove the unencodable character. The output encoded string is produced without raising a UnicodeEncodeError. Note that the output may not be accurate as the removed character may change the meaning of the string.

Example 7: Replacing unencodable characters with backslashreplace() method

We can use the <code>backslashreplace()</code> method to replace unencodable characters with a backslashed escape sequence. The <code>backslashreplace()</code> method replaces all occurrences of a string with a backslashed escape sequence, such as '\uXXXX' or '\xXX'.

# Example 7: Replacing unencodable characters with backslashreplace() method

string = "São Paulo"
encoded_string = string.encode("ascii", errors="backslashreplace")  # b'S\\xe3o Paulo'
decoded_string = encoded_string.decode("ascii")
print(decoded_string)  # S\xe3o Paulo

In the above example, we are trying to encode a string with 'ascii' encoding, which cannot encode the character 'ã'. Therefore, we are using the <code>backslashreplace()</code> method to replace the unencodable character with a backslashed escape sequence '\xe3'. The output encoded string is produced without raising a UnicodeEncodeError. Note that the output may be less readable due to the inclusion of escape sequences.

Example 8: Handling the error with a try-except block

We can handle the <code>UnicodeEncodeError</code> with a try-except block. The try block contains the code that may raise the exception, and the except block contains the code that handles the exception.

# Example 8: Handling the error with a try-except block

string = "São Paulo"
try:
   encoded_string = string.encode("ascii")
except UnicodeEncodeError as err:
   print("Got an encoding error:", err)
   encoded_string = string.encode("utf-8")
print(encoded_string)  # b'S\xe3o Paulo'

In the above example, we are trying to encode a string with 'ascii' encoding, which cannot encode the character 'ã'. Therefore, the <code>str.encode()</code> method raises a UnicodeEncodeError. We are handling the error by catching the exception with a try-except block. In the except block, we are printing an error message, and encoding the string with 'utf-8' encoding, which can handle any Unicode character. The output encoded string is produced without raising a UnicodeEncodeError.

Conclusion

In this tutorial, we have learned about the <code>UnicodeEncodeError</code> in Python, its causes, and how to solve it with examples. We have seen that the error occurs when Python is unable to map the given character to the specified encoding, which could be due to a wrong encoding, a character not supported by the encoding, or an invalid byte sequence. We have discussed several ways to solve the error, such as specifying a suitable encoding, encoding the string with Unicode encoding, using the <code>replace()</code>, <code>ignore()</code>, or <code>backslashreplace()</code> method, and handling the error with a try-except block. Remember to choose the appropriate method depending on the application requirements and the system configuration.