📊 Unit 1 – DataFrame · Lesson 14
Index Changes & Missing Values
Changing the index
Python
import pandas as pd
df = pd.DataFrame({'Roll': [11, 12, 13], 'Name': ['Asha', 'Ravi', 'Neha'], 'IP': [88, 81, 95]})
df2 = df.set_index('Roll') # a column becomes the index
print(df2)
print(df2.reset_index()) # index goes back to a column Name IP
Roll
11 Asha 88
12 Ravi 81
13 Neha 95
Roll Name IP
0 11 Asha 88
1 12 Ravi 81
2 13 Neha 95
reindex()
Python
import pandas as pd s = pd.Series([88, 81, 95], index=['Asha', 'Ravi', 'Neha']) print(s.reindex(['Neha', 'Asha', 'Om'])) # new order; unknown label -> NaN
Neha 95.0
Asha 88.0
Om NaN
dtype: float64
What are missing values?
Real data often has blanks – a student was absent, a form field was left empty. pandas stores such a blank as NaN. Missing values must be found and handled before analysis, otherwise averages and charts can mislead.
| Name | Maths | IP | City | |
|---|---|---|---|---|
| 0 | Asha | 78 | 88 | Delhi |
| 1 | Ravi | NaN | 81 | Pune |
| 2 | Neha | 65 | NaN | NaN |
| 3 | Om | 88 | 74 | Agra |
| 4 | Zoya | NaN | 90 | Pune |
Python
import pandas as pd
import numpy as np
df = pd.DataFrame({'Name': ['Asha', 'Ravi', 'Neha', 'Om', 'Zoya'],
'Maths': [78, np.nan, 65, 88, np.nan],
'IP': [88, 81, np.nan, 74, 90],
'City': ['Delhi', 'Pune', np.nan, 'Agra', 'Pune']})
df.to_csv('class.csv', index=False)
print(df)
print(df.isnull()) Name Maths IP City
0 Asha 78.0 88.0 Delhi
1 Ravi NaN 81.0 Pune
2 Neha 65.0 NaN NaN
3 Om 88.0 74.0 Agra
4 Zoya NaN 90.0 Pune
Name Maths IP City
0 False False False False
1 False True False False
2 False False True True
3 False False False False
4 False True False False
Detecting: isnull() and notnull()
Python
import pandas as pd
df = pd.read_csv('class.csv')
print(df.isnull().sum()) # NaN count in every column
print('Total missing:', df.isnull().sum().sum())
print(df[df['Maths'].notnull()]) # rows where Maths is presentName 0
Maths 2
IP 1
City 1
dtype: int64
Total missing: 4
Name Maths IP City
0 Asha 78.0 88.0 Delhi
2 Neha 65.0 NaN NaN
3 Om 88.0 74.0 Agra
Removing: dropna()
Python
import pandas as pd
df = pd.read_csv('class.csv')
print(df.dropna()) # drop rows having ANY NaN
print(df.dropna(subset=['Maths'])) # only check Maths
print(df.dropna(thresh=3)) # keep rows with at least 3 values Name Maths IP City
0 Asha 78.0 88.0 Delhi
3 Om 88.0 74.0 Agra
Name Maths IP City
0 Asha 78.0 88.0 Delhi
2 Neha 65.0 NaN NaN
3 Om 88.0 74.0 Agra
Name Maths IP City
0 Asha 78.0 88.0 Delhi
1 Ravi NaN 81.0 Pune
3 Om 88.0 74.0 Agra
4 Zoya NaN 90.0 Pune
Filling: fillna()
Python
import pandas as pd
df = pd.read_csv('class.csv')
print(df.fillna(0))
filled = df.fillna({'Maths': df['Maths'].mean(), 'IP': df['IP'].mean(), 'City': 'Unknown'})
print(filled.round(1)) Name Maths IP City
0 Asha 78.0 88.0 Delhi
1 Ravi 0.0 81.0 Pune
2 Neha 65.0 0.0 0
3 Om 88.0 74.0 Agra
4 Zoya 0.0 90.0 Pune
Name Maths IP City
0 Asha 78.0 88.0 Delhi
1 Ravi 77.0 81.0 Pune
2 Neha 65.0 83.2 Unknown
3 Om 88.0 74.0 Agra
4 Zoya 77.0 90.0 Pune
Which method to use?
dropna()
- Removes rows/columns with NaN
- Use when few rows are missing
- Data becomes smaller
fillna()
- Replaces NaN with a value
- Use 0, mean, median or a label
- Keeps all rows
Frequently Asked Questions
How does pandas show a missing value?
As NaN (Not a Number). In CSV files an empty cell is read as NaN.
How do I count missing values in each column?
df.isnull().sum(). For the total use df.isnull().sum().sum().
Should I drop or fill missing values?
Drop when only a few rows are affected; fill (with 0, the mean or a fixed label) when deleting would lose too much data.
