What Is Web Scraping? (2026)

Web scraping is reading a public page and saving the fields you need. A program requests a URL, receives HTML or JSON the site already sends to a browser, and writes a few of those fields to a row: a title, a price, a name. The glossary definition is that sentence. The rest of this page is the pipeline, what it is not, and where a residential address fits.

People scrape to compare a public price across shops, to record a search result from a given country, or to notice that a public page changed. Those jobs are price monitoring and SERP tracking when the field is a price or a ranking. The tool is ordinary HTTP. The judgment is whether you are allowed to store what you read. That judgment is the legal and ethical note, and it is not optional once the run leaves a single example URL.

The request and the row

Start with a page you are allowed to read. https://example.com is a public placeholder. Replace it with your own public URL. The pattern is the same: one GET, then the <title> pulled out of the body. A real job swaps the title for the fields you actually store.

Code Snippet
import re
import requests

url = "https://example.com"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.8078.48 Safari/537.36",
}
r = requests.get(url, headers=headers, timeout=30)
match = re.search(r"<title>(.*?)</title>", r.text, re.I | re.S)
print(r.status_code, match.group(1).strip() if match else "")

The User-Agent above is a Chrome 156 example from the stable channel on 20 October 2026. What that header does, and what it does not change, is the User-Agent note. Check the status and the body. A 200 whose HTML is a block page is not a successful scrape. Save a row only when the fields are the ones you asked for.

When the address matters

Many public pages answer the snippet above from your own network. A residential proxy is the step you add when the site treats the caller’s IP as part of the visitor: a local price, a country catalog, or a refusal of a hosting address. The proxy is not the scraper. It is the path. What a residential proxy is is the product definition. It is not a VPN, which is a device tunnel and a different job, covered in proxy vs VPN.

Premium HTTP is proxy.aethyn.io port 2099. The username carries the country. The password is the dashboard secret. Quote both only because they sit inside a proxy URL. requests decodes them before it builds the header.

Code Snippet
from urllib.parse import quote

user = "aethyn-XXXXX-country-us"
password = "PASSWORD"
proxy = f"http://{quote(user, safe='')}:{quote(password, safe='')}@proxy.aethyn.io:2099"
r = requests.get(url, headers=headers, proxies={"http": proxy, "https": proxy}, timeout=30)

Pause between requests. A new exit does not replace a polite pace. Stay on pages the public can open without an account. This pipeline does not log in, does not solve a challenge, and does not call a private API the site did not give you. If the gateway refuses the login, the status is 407 and the body says whether the header was missing or the secret was wrong. That is a credential problem, not a parsing problem.