SmartScrape

Problem

Most "AI scraping" tools either ship as opaque SaaS with your data on someone else's servers, or make you hand-write selectors for every site you want to track.

I wanted a tool that takes a URL and a plain-English description, produces a working scrape configuration, runs it on a schedule, detects what actually changed between runs, and notifies — all self-hosted, with the operator's own API keys.

SmartScrape screenshots

01 / 08 · Home — recent runs and at-a-glance status

Approach

An Express + TypeScript API, a React/Vite SPA and a Redis/BullMQ worker, deployed with Docker Compose alongside Postgres 16. The core loop is deliberately small:

• Configure. A wizard sends the target page and the user's brief to an LLM (OpenRouter / OpenAI / Anthropic — the user chooses) and proposes a JSON extraction schema, a comparison key, and notification rules. The user accepts, edits, or starts over manually.
• Fetch. Cheerio for static HTML where it works; Playwright when it doesn't. The choice is auto-detected per job.
• Diff. SHA-256 per extracted row, matched across runs by the user-chosen comparison key. The output is an added / removed / changed delta, not a full document diff.
• Notify. Rule types include any-change, new-items, removed-items, field threshold (price < 500), and field-value transitions. Delivery to email, Telegram, or a per-job HMAC-signed webhook with retry and backoff.
• Operate. Failures are classified, and a job pauses itself after three failures in a row. Schedules are rebuilt from Postgres on boot and a sweeper clears stale runs. A headless CLI and personal access tokens cover scripting.

Security work that mattered more than the AI surface: API keys and OAuth tokens encrypted with AES-256-GCM at rest, bcrypt-hashed passwords, hashed rotating refresh tokens in HttpOnly cookies, an SSRF guard on every user-supplied URL (proxy and webhook URLs included), prompt-injection hardening with a secret-leak guard on every extraction call, CORS locked to the configured frontend, Helmet headers, and zero outbound analytics.

Result

A working, deployable tool with a mobile-responsive UI and OAuth-backed Google Sheets export. CI gates every merge: format, lint, typecheck, server and client tests (about 260 server tests), build, npm audit, Postgres migrations up/down/up, and a Docker image build.

Honest status: the latest release is v0.9.0; the CLI, webhooks, access tokens and operational hardening landed on main after it. It hasn't been run in production yet, so there are no real run counts to report.