Scrapling for Hermes: get past Cloudflare walls

Your Hermes agent handles easy pages fine. Point it at a Cloudflare-protected site and it stalls, or hands back a "verify you’re human" screen instead of data. Scrapling is the tool you add for exactly that: a local, Python-based stealth fetcher that solves Cloudflare Turnstile and renders JavaScript, all without leaving your machine.

It ships as an optional skill alongside Hermes — not new (it has been in the catalog since March 2026), and not a replacement for the built-in web tools. It’s the extra option you reach for when the light tools hit a wall. This guide covers what Scrapling does, and — honestly — what it really takes to get it running, because the official "one command" leaves out the parts that actually make it work.

Start with the built-in: web_extract

Before Scrapling, know your starting point — because you already have one. Hermes ships with web_extract, a built-in tool that fetches a page and hands it back as clean Markdown (plus a companion, web_search, for search results). You don’t run a command for it: just ask your agent in plain language — "read this page for me" or "what does this URL say?" — and it calls the tool itself.

It works out of the box with no setup — a fresh install rotates across free providers automatically — and you can pin or configure a backend (Firecrawl is the default) by running hermes tools. The full list of providers and how to set them up lives in Nous’ Web Search & Extract docs.

web_extract handles a lot of sites on its own, so it is always your first try. Scrapling is the escalation — what you reach for the moment web_extract comes back with a challenge page, a blank shell, or a 403.

What Scrapling does

Scrapling is a 100% local Python scraping library. Inside Hermes it gives your agent three things the built-in tools don’t:

• StealthyFetcher — anti-bot bypass. Solves Cloudflare Turnstile and Interstitial challenges automatically, backed by patchright (a stealth-patched Playwright). This is the reason most people install it.
• DynamicFetcher — JavaScript rendering. A full Playwright Chromium browser for single-page apps and JS-heavy pages, without the stealth overhead.
• CSS/XPath extraction. Pull exactly the element you want with page.css(...) — the key to keeping output small (more on that below).

It isn’t magic — every anti-bot evasion leaks eventually after the next browser update — but for a site that blocks a plain fetch, Scrapling is often the difference between real data and a challenge page. And because it runs locally, there are no API keys and nothing leaves your machine.

The "one command" that isn’t: installing Scrapling

The official route is advertised as a single command. In practice a working install takes one command, two prompts, and about five minutes.

Step 1 — Install the skill yourself

hermes skills install official/research/scrapling --force

This one stays manual on purpose. You’re installing third-party code, and Hermes’ security scan returns a CAUTION verdict here — it flags the skill’s pip install lines as a supply-chain risk. That’s a false positive for the official SDK, but the decision to override it should be yours, not something an agent does quietly on your behalf. Read the skill, then --force past the benign flags.

Step 2 — Let your agent install the rest

The skill alone doesn’t fetch anything: the package that ships with Hermes is missing curl_cffi, patchright, camoufox, browserforge, and msgspec, and the browsers aren’t installed either. That’s three more commands — but you don’t need to know which Python environment Hermes runs in, because your agent already does. Paste this:

Install the Scrapling extras in your own Python environment:
run pip install "scrapling[all]", then scrapling install,
then python -m patchright install chromium. Show me each result.

Letting the agent run these in its own environment also avoids the classic failure where pip installs into a different Python than the one Hermes actually uses.

Step 3 — Verify with a real fetch

This is the step people skip, and it’s the one that matters — because the skill list will claim everything is fine either way. Ask for an actual fetch:

Verify Scrapling works: use DynamicFetcher to fetch
https://example.com and show me the page title.

What comes back tells you exactly where you stand:

• "Example Domain" — you’re done.
• ModuleNotFoundError: curl_cffi — the extras didn’t install. Tell the agent to run pip install "scrapling[all]" and try again.
• A startup banner and then nothing — the stealth browser is missing. Tell the agent to run python -m patchright install chromium.

The useful part: your agent sees these errors too, so in most cases you can simply say "that failed — fix it and retry" and it will. One platform note: on Windows the commands use python, not python3.

Using Scrapling

Installing the skill doesn’t turn anything on by itself. A skill is a folder of instructions — a SKILL.md — and your agent only sees its name and one-line description until something matches. Ask for a page behind anti-bot protection and that description is what makes it open the file, read the instructions, and write the fetch code itself:

You: Get me the rating and title from this IMDb page:
     https://www.imdb.com/title/tt0111161/

The Python below is what your agent writes, not what you type — worth knowing so you can spot a wrong result and steer it:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch(
    "https://www.imdb.com/title/tt0111161/",
    solve_cloudflare=True,   # solves Cloudflare Turnstile automatically
    network_idle=True,       # wait until the page stops loading
)
title = page.css("h1 span::text").get()   # IMDb nests the title in a span; plain h1::text returns None

solve_cloudflare adds roughly 5–15 seconds per fetch, so it’s worth telling your agent to try the light route first — something like "use web_extract first, and only fall back to Scrapling if the page is blocked." For a JavaScript-heavy page that isn’t behind anti-bot protection, the lighter DynamicFetcher does the job:

from scrapling.fetchers import DynamicFetcher

page = DynamicFetcher.fetch("https://quotes.toscrape.com/")
quotes = page.css(".quote .text::text").getall()

If your agent picks the wrong route, name the skill outright: "use the Scrapling skill for this one."

Keeping output small: spillover and CSS selectors

A full-page fetch from a modern e-commerce site runs to hundreds of kilobytes of HTML, and pages start spilling somewhere around 50–100 KB. Hermes caps inline tool results and spills the full output to a file on disk instead of jamming it into the agent’s context window. When that happens, the rule is: never re-fetch the same URL. The data is already saved — read it back with Hermes’ read_file tool. Re-fetching wastes time, burns a browser instance, and risks a rate limiter.

Better still, prevent the spill. Apply a CSS selector before the content reaches your agent — ask for the element you want instead of the whole page:

page = StealthyFetcher.fetch(
    "https://www.amazon.com/dp/BXXXXXXXXX",
    solve_cloudflare=True,
)
price = page.css(".a-price .a-offscreen::text").get()
title = page.css("#productTitle::text").get()

That returns just the price and title — maybe 2 KB instead of 150 KB. The selector runs inside Scrapling before anything reaches your agent, saving both bandwidth and context tokens.

Site playbooks

• Amazon. Use StealthyFetcher with a selector like .a-price .a-offscreen and #productTitle, plus solve_cloudflare=True. The selector is non-negotiable — without one, a single Amazon product page easily spills to disk. But treat the price selector as best-effort: Amazon sometimes serves a soft-blocked variant where the page loads and the title comes through, yet the price block is missing — .a-price returns nothing, or you get a price-block-error-message. Always check you actually got a price, and re-fetch if not.
• Classifieds and marketplace search pages. Use Scrapling as the fallback when a lighter fetch hits a 403 or 429. Watch out for search URLs that quietly ignore filter parameters — Marktplaats’ /q/ endpoint drops condition and filter values, for instance — so filter on title keywords and price in your parse step rather than trusting the URL.
• IMDb. Worth being honest here: Hermes’ built-in web_extract usually pulls IMDb cleanly on its own, so try that first and reserve Scrapling for a section web_extract misses. Scrapling isn’t always the answer — it’s the escalation.

Best practices

• Always verify. An HTTP 200 means nothing — a Cloudflare challenge page also returns 200. Confirm you got real content, not a blank shell or a challenge screen.
• Light first, Scrapling only when needed. It’s a speed rule and an etiquette rule: don’t drive a stealth browser when a plain fetch would do. Respect robots.txt and rate limits.
• Anti-bot is an arms race. Patchright works today and will need a patch after the next Chromium release. Test your setup against your target sites before relying on it in an automated workflow.
• On Windows, use python, not python3.

Troubleshooting

ModuleNotFoundError: No module named 'curl_cffi'

The extras never installed. Ask your agent to run pip install "scrapling[all]" in its own environment and try the fetch again — the package Hermes ships is missing what Scrapling needs to fetch anything.

The stealth fetch hangs — a banner, then nothing

The stealth browser is missing. Ask your agent to run python -m patchright install chromium, then retry.

Cloudflare keeps blocking even with solve_cloudflare=True

Give it time — Turnstile challenges are slow. Try adding real_chrome=True to the fetch, then verify whether you’re getting the real page or a challenge. Cloudflare’s detection shifts, and what worked last week may need a Scrapling update today.

Fetch output is truncated

Hermes capped the inline result and spilled the rest to disk. Read the saved file with read_file instead of re-fetching, and add a CSS selector to keep future fetches under the threshold.

Frequently asked questions

Is Scrapling new?

No. It has been an optional Hermes skill since March 2026 (v1.0.0). What’s easy to miss is that shipping with Hermes doesn’t mean it’s ready to use — you still have to finish the install.

Do I need API keys?

No. Scrapling runs 100% locally in Python. That’s its main advantage over cloud fetchers — and its main limitation, since your own IP is the one making the request.

When should I use Scrapling instead of the built-in tools?

Try the built-in web_extract first — it handles a lot. Reach for Scrapling when a site returns a challenge page, a blank shell, or a 403, i.e. when the light tools hit a wall.

Is scraping with a stealth browser legal?

It depends on what you scrape, how you use it, and the site’s terms of service. Scrapling’s own docs say: check robots.txt, respect rate limits, read the ToS, respect copyright and privacy, and get permission for commercial use. This guide is for personal, non-commercial agent automation — fetching a price for yourself is different from scraping a catalog at scale. When in doubt, don’t.

Wrapping up

Scrapling is your local anti-bot escape hatch: StealthyFetcher for Cloudflare, DynamicFetcher for JavaScript, and CSS selectors to keep output lean. One command, two prompts, five minutes — and the "ready" status is a lie until that last verification fetch comes back with real content.

Two follow-ups build on this. If you’d rather run Scrapling as your own containerized service — to pin a version or reuse it outside Hermes — a companion guide covers running it as a standalone MCP server in Docker. And a later article ties all the scraping options together — the built-in web_extract, the browser tools, and Scrapling — into a single routing matrix, so you always know which one to reach for.

This article is part of the Agentic AI series — hands-on guides for building and extending your own local AI agent. If you haven’t installed Hermes yet, start with the Windows 11 install guide and work through the Local AI on Windows series.