/PROJECT 05 · SCRAPPER - AUTOMATION - DATA API (FREELANCE CLIENT DELIVERY)
WHreek-Fuelbot-API
A production fuel-price API covering roughly 83,000 stations across 27 European countries, delivered to a freelance client who needed reliable programmatic access to fuel prices that only exist behind a Cloudflare-protected source with no public API. The service combines a persistent price store, a freshness window, stale-price fallback, per-station request coalescing and a global rate-limit circuit breaker, so the API keeps serving prices during upstream outages instead of failing. It is in production, deployed under PM2, and handled approximately 4,000 requests per day at about 99.2% uptime in its first week.
/GOAL
Serve reliable prices from a source that has no API and refuses plain requests.
/STACK
Technologies used
/ARCHITECTURE
How it is built
- Read layer — Public API v1Station search, station detail with live prices, per-station prices, and 'cheapest fuel nearby'. Versioned responses with a presenter layer, so internal storage can change without breaking clients.
- Freshness and fallback — Price pipelineStored prices are served while fresh, concurrent requests for the same station share one scrape, and a failed live fetch falls back to stored prices with stale: true rather than an error.
- Scrape and schedule — Background jobsA job manager with parameter schemas, a concurrency cap and cooperative cancellation, plus cron schedules with an overlap guard and a resumable rolling refresh that walks every station in paced, cursor-persisted batches.
- Global cooldown — Rate-limit circuit breakerAn upstream 429 or a stuck Cloudflare challenge pauses all scraping for Retry-After (or a configured cooldown). The API keeps serving stored prices during the pause; the cooldown clears itself automatically.
- Fuelbot Control — Operator dashboardLive traffic metrics, scrape health, cache state, job and schedule control, an IP/CIDR blocklist, DB-synced logs and an audit trail — behind HMAC-signed stateless sessions with timing-safe key comparison.
/SCOPE
What was delivered
- Serve station search, live prices, and 'cheapest nearby' from a versioned public API
- Scrape a Cloudflare-protected source with a real headless Chrome and store every result
- Serve stored prices while fresh, and stale prices when a live scrape fails
- Coalesce concurrent requests for the same station into one scrape
- Treat upstream rate limiting as a global cooldown instead of retrying into it
- Run a resumable, paced rolling refresh over all ~83,000 stations with cursor persistence
- Expose background jobs and cron schedules with an overlap guard
- Provide an operator dashboard with traffic, scrape health, jobs, blocklist and audit
- Keep the schema portable so the database provider is a one-file change
/NOTES
Scope boundaries
Fuelbot is a read-only price API. It does not process payments, place orders, or act on behalf of a station. Its data comes from a third-party aggregator and depends on that site's markup and availability; prices are informational and the service does not guarantee their accuracy for any commercial decision.
/GALLERY
Screens
/MORE_WORK