Web Scraping with BeautifulSoup

What is web scraping

Web scraping is using a program to automatically read data from web pages instead of copying by hand. In Python, requests downloads a page and BeautifulSoup reads its HTML so you can pull out exactly what you need — headings, prices, links and more.

requests + BeautifulSoup

Install both libraries once:

pip install requests beautifulsoup4

Then import them. requests gets the page, and BeautifulSoup (from the bs4 package) makes the HTML searchable.

Fetch and parse a page

import requests
from bs4 import BeautifulSoup

url = "https://example.com"
html = requests.get(url).text
soup = BeautifulSoup(html, "html.parser")

print(soup.title.text)

requests.get(url).text downloads the HTML as a string. BeautifulSoup(...) turns it into a searchable object, and soup.title.text reads the page title.

Finding tags and text

# first matching tag
first = soup.find("h2")
print(first.text)

# a tag by its class
box = soup.find("div", class_="price")
print(box.text)

find returns the first matching tag. You can match by tag name, or narrow it with a class using class_="...". Then .text gives the text inside.

Scraping a list

headings = soup.find_all("h2")
for h in headings:
    print(h.text)

find_all returns every matching tag as a list. Looping over it lets you extract many items at once — perfect for lists of products, articles or links.

Scrape responsibly

• Check the rules. Read the site's terms and its robots.txt file.
• Do not hammer the server. Add small delays between requests.
• Prefer an API. If the site offers an official API, use that instead.
• Only public data. Never scrape private or login-protected content.

Common mistakes

• Wrong tag or class name. Inspect the page's HTML first to find the exact names.
• Calling .text on None. If find returns nothing, check before reading text.
• Ignoring the status code. Confirm the page loaded (status 200) before parsing.
• Scraping too fast. Rapid requests can get you blocked — be polite with delays.

Summary

• Web scraping reads data from pages automatically.
• Use requests to fetch and BeautifulSoup to parse.
• find gets the first tag; find_all gets a list to loop.
• Match by tag name or class, then read .text.
• Scrape responsibly: check rules, add delays, prefer an API.

Next, handle the data with lists and dicts, or fetch structured data via an API.