Universal Web Scraping in Python: requests vs Headless, Tokens, Cookies, Proxies and Rotation

Ever celebrated a perfect parser prototype that flew on the demo page, only for it to start catching 403s, 429s, empty HTML, and "where did the cards go" in production? Content renders in JS, the server demands a token, and after an IP change the old session stops working.

This article takes a detailed look at how to collect data reliably and predictably — without excess magic, with an emphasis on real operation.

The whole article and all examples are in Python.

What we'll cover about scraping sites

when requests is enough and when you need a headless browser;
how to behave "like a human": headers, user-agent, cookies, locale and timezone;
sessions and tokens — why "new IP = new session" and what to do about it;
speed, rate limits and backoffs* without getting a subnet banned;
"infinite lists": scrolling or intercepting XHR/GraphQL;
a separate practical block on IP rotation.

*backoff — a retry strategy in which a pause is inserted between failed attempts, with each subsequent pause longer than the previous one.

1. The basic case where requests is genuinely enough

If the page is static or there's an open JSON endpoint, reach for requests plus an HTML parser (lxml, BeautifulSoup, parsel). It's fast, cheap and scales easily.

Key rules for scraping in this case:

send realistic headers (headers with a UA);
pin Accept-Language and Referer if the site is region-dependent;
on 5xx and 429 errors, use backoff with Retry-After.

A minimal example:

# pip install bs4 requests lxml

import requests
from bs4 import BeautifulSoup

headers = { "User-Agent": "Mozilla/5.0 ... Chrome/124.0.0.0 Safari/537.36", "Accept-Language": "ru-RU,ru;q=0.9,en;q=0.7", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}

r = requests.get("https://какой-то.адрес", headers=headers, timeout=30)
r.raise_for_status()

soup = BeautifulSoup(r.text, "lxml")
titles = [t.get_text(strip=True) for t in soup.select(".title")]

When that's not enough:

content is drawn on the client after load;
user interaction is required;
the server checks browser signals and sessions.

So this is quite enough for simple and demo sites.

2. When there's no getting around a headless browser

Signs:

empty HTML, while the browser's DevTools shows a rich DOM;
many XHR/Fetch/GraphQL calls after load;
sensitivity to sec-ch-ua, WebGL, plugins, locales, timezone and the like.

The recommendation is Playwright with Chromium. It gives a realistic environment and proper sec-ch-ua.

A short example of loading a page:

# pip install playwright
# playwright install

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    ctx = browser.new_context(
        user_agent="Mozilla/5.0 ... Chrome/124.0.0.0 Safari/537.36",
        locale="ru-RU",
        timezone_id="Europe/Moscow",
        viewport={"width": 1280, "height": 800},
    )
    page = ctx.new_page()
    page.goto("https://example.com/catalog", wait_until="networkidle", timeout=45000)
    html = page.content()
    print(html)
    browser.close()
    ctx.close()

Practical tips:

don't strip the browser down too much. The more "natural" it looks, the fewer triggers on the protection side;
pin locale and timezone — they affect prices, date formats and the selectors themselves;
if you see XHR with JSON — parse the JSON, not the HTML.

Headers and user-agent

Keeping current User-Agents up to date by hand is tedious. Use libraries that maintain a large set of UA strings and can pick real ones:

fake-useragent
random_user_agent

Usage:

# pip install fake-useragent

from fake_useragent import UserAgent
ua = UserAgent()

headers = { "User-Agent": ua.random, "Accept-Language": "ru-RU,ru;q=0.9,en;q=0.7", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}

Important:

old UAs fall under suspicion more often;
don't add exotic headers that a real browser doesn't send.

4. Cookies, tokens and "new IP = new session"

Many sites bind a session strictly to an IP address. If you picked up a session cookie on IP A and then suddenly start sending it through proxy B, the server will almost certainly treat that as suspicious behavior. At best it just resets the session; at worst it returns a 403 immediately.

Takeaways:

store cookies in an ip - cookies pairing;
when you switch proxies, create a new session — don't reuse the old one;
don't share the same cookies across different IPs;
CSRF tokens and other "one-off" markers are part of a specific session too — refresh them.

A handy Python template

First "warm up" the session in a headless browser — accept cookies, log in (if needed).
Save the cookies to storage_state.
Then use them in a requests.Session — but strictly from the same IP.

The simplest scheme:

def bootstrap_session_from_browser(proxy):
    # return a ready-to-use session

pool = {
    proxy: bootstrap_session_from_browser(proxy)
    for proxy in proxies
}

def fetch(url):
    proxy, sess = choose_proxy(pool)
    r = sess.get(url, timeout=30)

    # if the server dropped the session - recreate it
    if r.status_code in (401, 403):
        pool[proxy] = bootstrap_session_from_browser(proxy)
        r = pool[proxy].get(url, timeout=30)

    r.raise_for_status()
    return r.text

5. Speed, rate limits and backoff

Even perfect headers won't save you if you hammer the site with dozens of requests per second. The rules are simple:

limit request rate per domain.
apply backoff.
honor Retry-After.
schedule queues with a generous time margin.

A minimal backoff scheme:

import time, random, requests def  get_with_backoff(sess, url, retries=5, base=0.5, factor=2.0): for attempt in  range(retries):
        r = sess.get(url, timeout=30) if r.status_code == 429:
            ra = r.headers.get("Retry-After")
            wait = float(ra) if ra else base * (factor ** attempt)
            time.sleep(wait + random.uniform(0, 0.5)) continue r.raise_for_status() return r.text raise RuntimeError("слишком много попыток")

6. Infinite scrolling: two ways

You have surely seen sites where products or posts don't load right away but appear as you scroll down. That is infinite scrolling. It's convenient for users, but it complicates life for a scraper: the page is not complete on its own and has to be scrolled further.

Two ways around it:

Approach 1 - scroll the page like a human

The most direct and simple option is to imitate scrolling in a headless browser. The code literally "turns the mouse wheel", waits for content to load, then scrolls again. That way it can reach the very bottom of the page.

Downside: it's slow, because you actually wait for the content and elements to render.

page.mouse.wheel(0, 2400)
page.wait_for_timeout(800) # wait for the content to load

Approach 2 — find the API

The more "hackerish" and faster option: open DevTools and see where the page pulls its data from. Usually product cards or posts arrive in a separate XHR/GraphQL request — for example, to /api/catalog. If you find that request and replay it directly in code, you immediately get the whole JSON with the list of cards, skipping the scrolling itself.

The scenario looks roughly like this:

open the page through Playwright.
subscribe to page.on("response", ...). events.
filter responses by address (for example, /api/catalog).
read the JSON and extract the data you need.

Bottom line: if you need it fast and reliable — find the API and hit it directly. If the API is hidden or too much of a hassle — you'll have to scroll the page.

7. An important bonus: an outbound IP pool for getting past 429

If your parser runs where outbound traffic goes through a pool of IPs, you get an important bonus: requests leave from rotating addresses. In practice this spreads the load out and reduces the probability of 429 on an address, because your requests don't always leave from the same IP. What this buys you:

No separate paid proxies required.
The request rate from any one specific IP to the external site is lower.
As a side effect, a lower chance of an instant rate limit on the address.

There are also architectural details you should account for:

there may be no explicit button or API to switch the outbound IP yourself, on demand. Rotation happens without a clear rule, randomly under the hood of the infrastructure.
you can't guarantee that a request will leave from a new IP.
if the target binds sessions to an IP, a random change of source can break the session.
set up alerts on errors in your logs — 429s, for example — so you find out right away if something goes wrong with the parsing.

Even with an IP pool, don't give up on backoff and sensible rate limiting. That's still the best protection against 429. One more practical tip: make jobs idempotent, so an unexpected session break in the middle doesn't produce duplicates and junk in your storage.

8. Common parsing failures and quick recipes

403 right away — check how fresh your UA is, Accept-Language, Referer; try Playwright with a realistic context
429 occasionally — lower your RPS.
Empty HTML — the page runs on JS; intercept XHR/GraphQL or use a headless browser.
different language/currency — pin locale and timezone in the context and headers.

Summary

Start with requests — cheap and fast.
If the page lives on JS — go with Playwright.
Sessions and tokens are often tied to an IP — don't move cookies between addresses.
Randomize the User-Agent with a library, keep headers realistic.
Backoff, Retry-After and rate limiting are the foundation of a scraper that runs in production for a long time.
An outbound IP pool can help you avoid 429 without separate paid proxies.
All of this is easy to assemble in Python and maintain in production.