Encoding: ASCII, ISCII & Unicode

What is encoding?

A computer can store only numbers in binary. So how does it store the word Hello or नमस्ते? We agree on a table in which every character gets a unique number (its code point). Converting characters into these codes is called encoding; converting codes back to characters is decoding.

ASCII

ASCII (American Standard Code for Information Interchange) was created in the 1960s to make computers and teleprinters exchange English text. It uses 7 bits, giving 128 codes (0–127):

• 0–31 and 127: control characters (new line = 10, tab = 9) — not printed.
• 32–126: printable characters — space, digits, A–Z, a–z and symbols.

Extended ASCII uses 8 bits (256 codes) for extra symbols, but different companies filled codes 128–255 differently.

Important ASCII codes to remember

Trick: small letter = capital letter + 32. So D = 68 and d = 100. The word CAT is stored as 67 65 84.

ASCII in Python: ord() and chr()

ord(ch) gives the code of a character; chr(n) gives the character for a code.

for ch in ['A', 'a', '0', ' ']:
    print(repr(ch), 'code =', ord(ch), 'binary =', format(ord(ch), '08b'))
print(chr(72) + chr(105))

A fun use — a simple "secret code" that shifts each letter forward by one:

word = 'CBSE'
codes = [ord(c) for c in word]
print(codes)
secret = ''.join(chr(c + 1) for c in codes)   # shift every letter by 1
print(secret)

ISCII for Indian languages

ISCII (Indian Script Code for Information Interchange) was developed in India in the late 1980s. It is an 8-bit code: the first 128 codes are the same as ASCII, and codes 160–255 are used for letters of Indian scripts such as Devanagari, Bengali, Gurmukhi, Gujarati, Oriya, Tamil, Telugu, Kannada and Malayalam.

Because Indian scripts share the same structure (vowels, consonants, matras), one ISCII table could serve many scripts. Today, however, almost all Indian-language text on the web uses Unicode.

Unicode

Unicode is a universal standard that gives a unique number (written as U+ followed by hex) to every character of every writing system — Hindi, Urdu, Tamil, Chinese, Arabic, maths symbols and emoji. It has room for over a million code points. Its first 128 code points are the same as ASCII, so old English text still works.

UTF-8, UTF-16 and UTF-32

Unicode gives the number; a UTF (Unicode Transformation Format) decides how many bytes are used to store it.

Unicode in Python

Python 3 strings are Unicode, so Hindi works directly. encode('utf-8') shows the actual bytes stored.

for ch in ['A', 'अ', '₹', '😀']:
    b = ch.encode('utf-8')
    print(ch, 'U+' + format(ord(ch), '04X'), len(b), 'byte(s)')
print(len('नमस्ते'), 'characters,', len('नमस्ते'.encode('utf-8')), 'bytes')

So the 6 characters of नमस्ते (न, म, स, ्, त, े) take 18 bytes in UTF-8, while "Hello" takes only 5.

Summary & practice

• Encoding = mapping characters to numbers so they can be stored in binary.
• ASCII: 7-bit, 128 characters, English only. A=65, a=97, 0=48, space=32.
• ISCII: 8-bit code for Indian scripts.
• Unicode: one code point for every character in the world; UTF-8 stores it in 1–4 bytes.