Lesson 06: They Look Identical. Python Says They're Different. Do You Know Why?

Unicode, String Equality, and Normalization Explained Through Varastra — Discover Why Visually Identical Strings Can Fail Equality Checks, How Unicode Code Points Work, and How to Fix String Comparison Bugs in Python.

The interview insight: Never assume that two strings that look the same on screen have the same underlying representation.

Two strings can look exactly the same on screen and still be different in Python. This quietly breaks searches, imports and duplicate checks. Here is why, and how to fix it.

The problem at Varastra

Varastra is an online store for Indian ethnic wear. One Monday, a customer writes to support. She searched for a saree and found the same one twice.

Both listings have the same photo. Both are named "Café Brown Silk Saree". One says "In stock". The other says "Sold out".

Sam, a senior engineer, opens the supplier file. The name appears in two rows. Dev wrote the import code. It lowercases both names and compares them. Sam asks him to run the check.

The answer is False.

"But they look the same!" Dev says.

Tara, a new intern, does not trust her eyes. She checks the length of the colour word in each row. She reads the numbers out loud: "Four. And five."

Nobody speaks. Same word on the screen. Different length in the computer. Why?

What a string really is

A string is a row of characters. In Python, it works much like an array, with one big difference. You cannot change it. This is called immutable. Every "change" makes a new string.

Under the hood, Python stores a string as code points. A code point is a number that stands for one character in the Unicode standard. Python has no separate character type. One character is just a string of length 1.

Here is the trap. One letter you see can be one code point, or two.

The letter é in "café" can be stored two ways:

• Form A: one code point, U+00E9. The word has 4 code points.
• Form B: an e (U+0065) followed by a combining accent (U+0301). The word has 5 code points.

Pause and predict

Before you read on: form A and form B look the same on screen. Will len() give the same number for both?

Answer: No. Form A gives 4 and form B gives 5.

The diagram shows both forms. They look the same on screen. We checked in Python:

a = "caf" + chr(0x00E9)         # one code point for the accented e
b = "cafe" + chr(0x0301)        # e plus a combining accent

len(a)     # 4
len(b)     # 5
a == b     # False

That is exactly what Tara saw. Four and five.

Here is how to see the code points and bytes yourself.

for ch in a:
    print(hex(ord(ch)), end=" ")   # 0x63 0x61 0x66 0xe9
print()
for ch in b:
    print(hex(ord(ch)), end=" ")   # 0x63 0x61 0x66 0x65 0x301
print()

print(len(a.encode("utf-8")))      # 5 bytes
print(len(b.encode("utf-8")))      # 6 bytes

Bytes are a third count. When Python saves text to a file, it encodes it as bytes. In UTF-8, form A takes 5 bytes and form B takes 6 bytes. So "length" can mean code points or bytes. Know which one you are counting.

The same thing happens in many languages, not only French words. Any letter with an accent or mark can have more than one form.

The fix: one standard form

Tara tries three ideas first. Each one fixes one problem and misses another.

• Compare with ==. The two names fail, as you saw.
• Lowercase both, then compare. This is Dev's code. It fixes capital letters. It does not fix the accent problem. It also fails on some letters in other languages. In German, lower() leaves "ß" as it is, so "STRASSE" and "straße" do not match.
• Trim spaces at the ends. A supplier may type two spaces between words. "Banarasi Saree" and "Banarasi Saree" are still different strings.

What she needs is one cleaning step that handles all three. Python gives her the tools.

import unicodedata

def clean_name(s):
    s = unicodedata.normalize("NFC", s)
    s = s.casefold()
    return " ".join(s.split())

def same_name(a, b):
    return clean_name(a) == clean_name(b)

Line by line:

• normalize("NFC", s) converts text to one standard form. After this, both ways of writing "café" become the same 4 code points.
• casefold() is a stronger lower(). It turns "ß" into "ss".
• s.split() with no argument splits on any run of spaces. " ".join(...) puts the words back with one space each. This also trims the ends.

Each step reads the string once. So the cost is O(n) time and O(n) extra space, where n is the length of the name.

a = "caf" + chr(0x00E9)
b = "cafe" + chr(0x0301)

assert a != b
assert same_name(a, b)
assert same_name("STRASSE", "stra" + chr(0x00DF) + "e")
assert not same_name("Banarasi Saree", "Kanjivaram Saree")
assert same_name("Banarasi  Saree ", "Banarasi Saree")

Tara checks it with tests. The two forms of "café" are not equal raw, and equal after clean_name. "STRASSE" and "straße" are equal. "Banarasi Saree" and "Kanjivaram Saree" stay different.

Try it

Try unicodedata.normalize("NFD", a) and unicodedata.normalize("NFC", b). Check the length of each.

NFD splits the accented letter apart, so the length is 5. NFC joins it, so the length is 4. Pick one form and use it everywhere.

One limit: this treats names as the same only when the characters match after cleaning. It does not know that "saree" and "sari" mean the same thing. That is a different problem.

Tara runs clean_name on the supplier file. In the story, 211 duplicate rows come together.

Two more string traps

isdigit accepts more than 0 to 9

In the story, Varastra checks a six-digit delivery code. A first try looks right.

def valid_code_bad(code):
    return len(code) == 6 and code.isdigit()

It accepts "411001". But isdigit() also accepts digits from other scripts, such as Arabic-Indic digits. It also accepts a superscript two, so "41100²" passes. We ran both, and both pass the bad check.

The fix is to say exactly which characters are allowed.

def valid_code(code):
    return len(code) == 6 and all(c in "0123456789" for c in code)

This is a story rule. It is not a claim about real postal codes.

Building a long string in a loop

A string cannot change, so each "add" can copy the whole string built so far. Tara builds a string from n one-letter pieces in three ways. Each time is the best of three runs.

import time

def build_plus(parts):
    s = ""
    for p in parts:
        s += p
    return s

def build_front(parts):
    s = ""
    for p in parts:
        s = p + s
    return s

def build_join(parts):
    return "".join(parts)

def best_of_three(func, parts):
    best = None
    for _ in range(3):
        start = time.perf_counter()
        func(parts)
        elapsed = time.perf_counter() - start
        best = elapsed if best is None else min(best, elapsed)
    return best

for n in (40_000, 80_000, 160_000, 320_000):
    parts = ["a"] * n
    print(n, [round(best_of_three(f, parts), 4)
              for f in (build_plus, build_front, build_join)])

Adding to the front grows fast. When n doubles, the time goes up about four times. That is the O(n²) shape.

The plain s += p loop was fast in her test. The Python documentation warns that building a string by joining pieces in a loop is quadratic. Her test did not show that for this one pattern. We did not find the reason. It may be a shortcut in this Python version, but that is a guess.

So do not rely on it. "".join(parts) was fastest in every row, and it needs no shortcut. The rule is simple: collect the pieces in a list, then join them once at the end.

These timings come from one machine. They will differ on yours.

How to spot a string trap

Watch for three clues.

1. The problem says "ignore case", "ignore spaces" or "compare names." Think about cleaning first. Decide what "the same" means.
2. You build a string in a loop. Collect the pieces in a list. Join them once at the end. Never add to the front.
3. The input may hold non-English text, digits or symbols. Do not assume isdigit, isalpha or lower do what you hope.

Before you write code, ask out loud:

"Can I assume only English letters?"

The answer changes the code. In an interview, this one question shows you know strings have hidden costs.

When Tara could not explain the bug, she did one simple thing. She printed the length. That small step showed her the problem was real, and it is a good first move whenever two strings look equal but are not.

Takeaway: Strings cannot change, so count the copies. Build with join. And decide what "the same" means before you compare.