Python scraping template: 10 patterns you edit, not write

Most searches for a python scraping template end at a 40-line tutorial snippet that breaks on the second page of results. What actually saves time is a template per scraping pattern, already handling the boring parts: pagination, retries, dedupe, file output. This page lists the ten patterns we ship as ready-to-edit scripts, shows what each looks like in Python, and is straight about where templates stop helping.

One thing to know up front: the script pack behind this guide is written in Node.js, not Python. The structure ports line for line, and the mapping section below shows exactly which Python libraries replace which parts. If you would rather run the files as-is without porting, they work on their own; nothing in them needs Python.

The 10 templates and what each one is for

Each file is self-contained, opens with a how-to-run comment, and marks everything you are allowed to change with an EDIT constant. The full inventory with runtime notes ships as a free manifest JSON:

You need one template per job, the one that matches the page in front of you. The other nine stay in the folder for the next project.

How each pattern maps to Python

The transferable part of any template is its shape: fetch, parse, dedupe, persist. Each shape translates directly:

The retry wrapper shows the pattern in its smallest form. The shipped version adds jitter and reads the server's Retry-After header; this is the skeleton:

import time, requests

def fetch_retry(url, tries=5):
    delay = 1
    for _ in range(tries):
        r = requests.get(url, headers={"User-Agent": "my-scraper/1.0 (contact: me@example.com)"})
        if r.status_code == 200:
            return r
        if r.status_code in (429, 503):
            time.sleep(delay)
            delay *= 2
        else:
            r.raise_for_status()
    raise RuntimeError(f"gave up on {url} after {tries} tries")

Swap the fetch line for a BeautifulSoup parse or a Playwright page and the rest of a ported template stays identical across all ten patterns.

Why self-hosted scripts instead of a hosted scraper

Hosted scraping platforms rent you the same patterns by the month and keep your runs on their infrastructure. Self-hosted templates trade convenience for control: the files live on your machine, run on your schedule, and stop costing anything the day you stop using them. The data lands in your own CSVs, so nothing about your pipeline depends on another company's uptime or pricing changes.

The cost shows up as maintenance. A hosted platform absorbs layout changes behind its own tooling; with your own templates, you fix a selector when a site redesigns. The cloud vs self-hosted comparison walks both sides with numbers. For a handful of sites you check weekly, templates win on effort. For dozens of sites on tight schedules, the platform math changes.

What templates cannot do

Three situations outgrow templates, in Python or any language. Sites that load data from private APIs inside the page work, but the selector you need may sit behind a delay or scroll, and you will adjust the wait. Sites that require an account: the login template handles the mechanics, but only for an account you own or have written permission to use. Sites that redesign often: each change breaks selectors, and fixing them is a two-minute edit per change, repeated forever.

Templates remove the 90 percent that is boilerplate. The remaining 10 percent is judgment about the specific site, and no template removes that part.

Stay within robots.txt and the site's terms

Four checks before any run. Read the site's robots.txt and follow what it disallows. Read the terms of service for a clause about automated access. Collect public pages only; the login template exists for your own accounts, not for getting around access controls. Keep the request rate low and send a real user agent so site owners can reach you.

Template 07 reads robots.txt and crawl-delay for you. The other nine run at whatever pace you set, so set it slow: one page every few seconds is polite, ten per second is a problem for you and for the site. In Python, time.sleep() between requests does the same job. Laws on automated collection differ by country and site, so treat this section as a starting point, not legal advice.

Get the templates

The web scraping scripts pack page documents every file, lists use cases per script, and links the free manifest plus sample files. Setup is one npm install and one chromium download, and nothing uploads anywhere. The scrape without coding guide shows the edit-one-config workflow step by step if this is your first template.

The manifest lists all 10 files with their pattern and runtime notes, free to download.

FAQ

Is there a Python scraping template I can just edit?

Yes, though the ones documented here ship as Node.js and Playwright scripts. The structure is what you are borrowing: a config block at the top, one loop per pattern, a save step at the end. In Python the same templates map to requests plus BeautifulSoup for static pages, Playwright for Python when a page needs a real browser, and the csv or json modules for output. Porting one template is usually under an hour.

What should a Python scraping template include?

Five parts: an EDIT config block at the top with the start URL and selectors, a retry with exponential backoff for failed requests, a loop that handles the page pattern (pagination, scroll, or poll), a dedupe step so reruns do not duplicate rows, and a plain output writer to CSV or JSONL. A real user agent string belongs in the headers of every request.

Is scraping with Python legal?

It depends on what you scrape and where the site operates. Check robots.txt and the terms of service, collect public pages only, keep request rates low, and never scrape personal data. Login-protected pages need an account you own or written permission. Rules differ by country and site, so treat any checklist as a starting point, not legal advice.