yail259/frankensurf

One call that gets your agent through any site, stitched from 28 browsing tools

Summary

Frankensurf is a Python library that wraps 28 web-fetching tools behind a single read() call aimed at AI agents. It starts with the cheapest method and escalates to stronger ones only when a site blocks or challenges the request, returning raw page data plus a receipt of which tools ran. The project ships an MCP server, a CLI, and integrations for LangChain, the OpenAI Agents SDK, and CrewAI.

What it is useful for

Frankensurf addresses the fact that every web tool fails on a different set of sites: plain HTTP is blocked on some, headless Chrome on others, paid APIs on others. For an agent that needs to read arbitrary pages on the open web, it acts as a unified front that picks the right tool per URL. It fits retrieval-augmented agents, web research automations, and scraping pipelines where reliability across many sites matters more than peak speed on any one site.

The read() call returns text or markdown, JSON-LD and meta tags, images, and a quality grade (good, partial, poor) with flags like paywall or placeholder. A retry_of=<trace_id> argument lets a follow-up call skip tools that already failed, and --lightning races extra free tools when a read stalls past three seconds. A bundled local SearXNG (started by frankensurf install search, no Docker required) supports the free search tier.

How engineers can use it

Install with pip install "frankensurf[mcp]" (Python 3.12+ on Linux, macOS, or Windows via WSL), then run frankensurf install to pull in the free toolchain. The CLI is the fastest path:

frankensurf read https://example.com --markdown
frankensurf search "playwright python tutorial" --recency week

From Python:

from frankensurf import Runtime

async with Runtime("state") as web:
    page = await web.read("https://example.com", policy_overrides={"prefer_markdown": True})
    hits = await web.search("playwright python tutorial")

For agents, register the MCP server with claude mcp add frankensurf -- frankensurf-mcp, or import the framework helpers after pip install "frankensurf[langchain]" (and similar extras for the OpenAI Agents SDK and CrewAI):

from frankensurf.agent_tools import langchain_tools
tools = langchain_tools()  # frankensurf_read, frankensurf_search

The README flags the project as alpha. Etsy, Home Depot, Yelp, Crunchbase, and G2 blocked every free tool in spot checks; Instagram, TikTok, Facebook, and Pinterest show only the public shell without a login. Paid fallbacks such as Firecrawl and Zyte are opt-in via API keys and the --paid flag, with --max-cost USD capping spend. New tools slot in as plugins; design notes live in SPEC.rst and contributor rules in AGENTS.md.