Scraping responsibly — and its limits · Working with Data

Sometimes the data you want lives on a web page and the site has no API. Nepali news headlines, ministry circulars, public school addresses, NEPSE tables that exist only as HTML. Scraping — writing a program that fetches the page and extracts the data — is a real technique, and one most builders use occasionally. It is also surrounded by rules and constraints that beginners often discover the hard way. This section covers when scraping is appropriate, how to do it cleanly, and when to put it down and ask for the data directly.

What scraping actually is

A scraper is a program that does what a human would do in a browser, but in code. It requests a URL, receives HTML, finds the bits of HTML containing the data you want, and extracts them into a usable format.

In Python, the two libraries you reach for are requests (to fetch the HTML — same library as the last section) and BeautifulSoup (to parse the HTML):

pip install beautifulsoup4

The minimal pattern:

import requests
from bs4 import BeautifulSoup

response = requests.get("https://example.com/some-page")
soup = BeautifulSoup(response.text, "html.parser")

for heading in soup.find_all("h2"):
    print(heading.get_text(strip=True))

That fetches a page and prints every <h2> heading on it. Most scraping is some elaboration of this shape.

When scraping is appropriate (and when it isn’t)

Before writing a line of scraper, work through a short checklist:

Does the site have an API? Check the documentation. Check for an /api path. Search “[site name] developer”. Many sites that look like they have no API actually have one if you look.
Could you ask for the data? Some ministries, NGOs, and businesses will email you a CSV if you write a polite email. Costs you 30 minutes; saves you weeks of scraping fragility.
What do the site’s terms of service say? Many sites explicitly forbid scraping. Some allow it for personal or research use but not commercial. Read the terms, even briefly.
What does robots.txt say? Almost every site has a file at /robots.txt (e.g. example.com/robots.txt) that declares what automated agents are and aren’t welcome to access. Respect it — both because it is the polite thing to do and because not respecting it is in some jurisdictions a legal problem.
Are you taking personal data? Names, addresses, photos of identifiable people, anything that could be linked back to an individual. If so, scraping introduces serious privacy concerns we cover properly in Chapter 6.

If you can answer “API/email is not available, terms permit it, robots.txt allows it, and the data is not personal”, you can scrape with a clean conscience. If any of those is no, stop and reconsider.

A polite scraper, end to end

A small scraper for Nepali news headlines, demonstrating every habit you should pick up at once:

import time
import requests
from bs4 import BeautifulSoup

BASE = "https://example-news-site.np"
HEADERS = {
    "User-Agent": "AILiteracyNepal-Scraper/1.0 (contact: you@example.com)",
}


def fetch_headlines(category: str) -> list[str]:
    url = f"{BASE}/category/{category}"
    response = requests.get(url, headers=HEADERS, timeout=10)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")
    headlines = [
        h.get_text(strip=True)
        for h in soup.select("article.news h2")
    ]
    return headlines


if __name__ == "__main__":
    for category in ["politics", "business", "sports"]:
        print(f"--- {category} ---")
        for headline in fetch_headlines(category):
            print(headline)
        time.sleep(2)

Four habits worth taking from this:

A descriptive User-Agent. Identify your scraper honestly. Many sites block requests with no User-Agent or with the default python-requests/2.x. A real string with a contact email also makes it easy for the site to reach out if there is a problem — better than getting silently blocked.
timeout=10 — same lesson as the last section.
time.sleep(2) between requests. Slow down. A human reading a news site does not click ten articles per second. Your scraper should not either.
Targeted selectors ("article.news h2"). Pull only the data you actually need. Do not download entire archives “just in case”.

The HTML usually breaks within a year

A practical reality of scraping: HTML structures change. The selector article.news h2 that works today may be div.headline-wrapper h3 six months from now after the site redesigns. Your scraper silently returns nothing, or worse, returns nonsense.

Defensive moves:

headlines = soup.select("article.news h2")
if not headlines:
    raise RuntimeError(f"No headlines found on {url} — has the page structure changed?")

A loud failure is better than a quiet one. A scraper that returns an empty list when the site changes can run for weeks before someone notices.

Dynamic pages — when `requests` alone is not enough

Some sites do not render their content in the HTML they first send. The page loads, and then JavaScript runs in the browser and fetches the data into the page. A requests.get only gets the first HTML, before the JavaScript runs — so the data you want is missing.

Signs you are dealing with this:

You can see the data when you visit the page in a browser.
The same page fetched with requests.get and printed shows none of the data.
The HTML is mostly empty containers with names like <div id="app">.

Two paths from here:

Find the underlying API the JavaScript itself is calling. Open the browser’s Developer Tools, switch to the Network tab, reload the page, and look for XHR/fetch requests. Often the JavaScript is itself calling a JSON API — and you can call the same one directly with requests, skipping the scraping entirely. This is by far the cleanest path.
Use a headless browser. Libraries like playwright or selenium run a real browser in the background, execute the JavaScript, and let you read the resulting HTML. Heavier (slower, more dependencies) but works when option 1 fails.

For Course 02 we will stick with requests + BeautifulSoup. If you need a headless browser for a real project, the patterns transfer — same selectors, same waiting-for-things-to-load discipline.

When to stop scraping and just ask

A useful instinct to develop: every time you spend more than half a day on a scraping problem — fighting selectors, fighting rate limits, fighting JavaScript — pause and ask whether a polite email to the data owner would have worked. The answer is “yes” surprisingly often. Government data offices, NGOs, and even private businesses with public data often have a contact person whose job it is to share data with researchers. They mostly want to share. They mostly are not asked.

This is also a long-term relationship. A builder who emails a ministry, gets the data politely, and credits the source in their published work will have a much warmer reception than one who scrapes and waits for someone to notice. Reputation in Nepal’s small data community travels.

Check your understanding

Quick check

—

You want to build a small project that uses Nepali school addresses from a Ministry of Education page. The page lists them. The ministry has no obvious API. What is the first thing to try, according to this section?

Write a scraper to download all addresses immediately.
Check the page's robots.txt, then the ministry's site for any developer or open-data section, then politely email the ministry's data contact and ask for a CSV. Scraping is the last resort, not the first.
Pay a third-party service to scrape it for you.
Manually retype every school address.