Shipping Free SEO Tools Without Melting Your API (ScrapeKey’s Approach)

Part of Building ScrapeKey: Product Engineering Notes. See also the ScrapeKey case study.

Introduction

The hook

You want a growth wedge: let people use something valuable before they pay. Smart move.

Then launch week arrives. One scraper’s IP hits your long-tail endpoint a thousand times. Your Suggest proxy gets throttled. Honest users see spinners. Marketing blames “the tools page.” (Fair, honestly.)

Free tools without rate limits aren’t a funnel — they’re a load test you forgot to schedule.

The objective

By the end of this post you’ll know how ScrapeKey ships free SEO tools that:

1. Sit in front of the paid keyword → affiliate workflow.
2. Require login (so abuse keys off accounts, not only IPs).
3. Share a quality pipeline with the product (intent, affiliate fit, monetization score).
4. Fail gracefully with 429 + Retry-After, caching, and URL fetch guards.

Prerequisites

• Basic FastAPI routing and request validation
• Awareness of rate limiting (token bucket / sliding window ideas)
• Optional: React marketing → app tools navigation

---

Body

Step 1: Decide what “free tool” means in the product

ScrapeKey’s free tier is not a toy. The tools map to the same problems paid users solve:

Tool	Job to be done	Why it funnels to Pro
Long-tail generator	Expand a seed into phrases that can pay	Monetization score + Est. metrics tease Pro depth
SERP / meta preview	See how a page might look in Google	Content briefs / matching come next
Keyword density	Check on-page focus	Feeds content workflow
Intent classifier	commercial vs informational	Same intent model as matching
Weekly digest / bookmarklet	Habit + acquisition	Keeps ScrapeKey in the week

Product callout: We label volume and difficulty as Est. Free tools are not “fake Ahrefs.” Trust is the conversion strategy. If you invent exact volumes on a free endpoint, you will refund forever.

---

Step 2: Put tools behind a flag and a login gate: on purpose

ScrapeKey’s tools router is feature-flagged:

ENABLE_FREE_TOOLS = os.getenv("ENABLE_FREE_TOOLS", "true").lower() == "true"
TOOLS_REQUIRE_LOGIN = os.getenv("TOOLS_REQUIRE_LOGIN", "true").lower() == "true"

When login is required, every tool handler depends on get_current_active_user. That choice has tradeoffs:

Approach	Pros	Cons
Fully anonymous	Higher top-of-funnel traffic	IP rotation abuse; hard quotas
Login required (ScrapeKey)	User-keyed limits; email for support	Friction before first delight

We chose login so rate limits can key on user:{id} first, and so Free-plan enrichments and tool hours stack with the same identity as Stripe later.

Product callout: Landing copy can still say “try free tools.” The conversion step is account creation, which is also when welcome email and onboarding fire. Friction is a feature if the tool is good.

---

Step 3: Rate-limit like two windows, not one magic number

ScrapeKey’s tools_rate_limit service uses:

1. Hourly fair-use: default Free 50/hour, Pro 500, Agency 2000 (env overrides).
2. Burst window: e.g. 10 requests / 60 seconds so a loop can’t empty the hour in one second.

RATE_LIMIT_REQUESTS = int(os.getenv("TOOLS_RATE_LIMIT_FREE", "50"))
RATE_LIMIT_REQUESTS_PRO = int(os.getenv("TOOLS_RATE_LIMIT_PRO", "500"))
RATE_LIMIT_WINDOW_SECONDS = int(os.getenv("TOOLS_RATE_LIMIT_WINDOW", "3600"))
BURST_LIMIT_REQUESTS = int(os.getenv("TOOLS_BURST_LIMIT", "10"))
BURST_WINDOW_SECONDS = int(os.getenv("TOOLS_BURST_WINDOW", "60"))

Client key resolution:

def get_client_key(request: Request, user_id: Optional[str] = None) -> str:
    if user_id:
        return f"user:{user_id}"
    # fallback: IP + short UA hash for anonymous mode
    ...

On exceed, return 429 with structured detail and Retry-After:

raise HTTPException(
    status_code=429,
    detail={
        "error": {
            "code": error_code,
            "message": "Rate limit exceeded. Please try again later.",
            "details": {
                "retry_after_seconds": reset_seconds,
                "limit": limit,
                "remaining": 0,
            },
        }
    },
    headers={"Retry-After": str(reset_seconds)},
)

Caveat for hiring conversations: ScrapeKey’s first implementation stores timestamps in memory. Fine for a single Railway instance. Multi-instance needs Redis (or equivalent). Call that out in design reviews: don’t pretend the dict is global truth.

---

Step 4: Share the keyword quality pipeline with the paid product

Long-tail is not a throwaway script. It calls the same discovery path used by richer workflows:

1. Sanitize seed (charset, length).
2. Pull Google Suggest (async) + pattern variants (questions, modifiers).
3. Drop blocklist / off-seed noise.
4. Classify intent and affiliate_fit.
5. Compute monetization_score (0–100) from fit + intent, not from invented volume.
6. Attach estimated metrics, clearly marked.

def _variant_row(keyword: str, source: str, ...) -> Dict[str, Any]:
    intent = classify_intent(keyword)
    fit = classify_affiliate_fit(keyword, intent)
    score = compute_monetization_score(affiliate_fit=fit, intent=intent)
    row = {
        "keyword": keyword,
        "intent": intent,
        "affiliate_fit": fit,
        "monetization_score": score,
        "monetization_tier": monetization_tier(score),
        ...
    }
    return attach_estimated_metrics(row, from_google_suggest=(source == "google_suggest"))

Product callout: Free tools teach the vocabulary of the paid matcher, “commercial,” “high fit,” “monetization score.” When users hit Pro matching, the UI language is already familiar. That is funnel design as API design.

---

Step 5: Guard the dangerous bits: URL fetch and HTML parse

SERP preview and density tools can accept a URL. That opens SSRF and huge download traps.

ScrapeKey centralizes fetch safety (validate_public_http_url, max bytes) before BeautifulSoup. Tutorial rule:

• Only http/https
• No private/link-local ranges
• Cap response size
• Time out aggressively

Never requests.get(user_url) “just for a demo tool.” Demo tools get abused first.

---

Step 6: Cache what you can; count what you must

Repeat seeds hammer Suggest. A short TTL cache (get_cached / set_cached on tools) cuts duplicate upstream work. Rate limits still increment on cache hits or misses depending on your product choice, ScrapeKey counts the API call (user-facing request), which keeps UX predictable: “50 tool runs/hour” means button presses, not unique seeds.

Enrichment quotas (monthly) are separate from tool hourly limits. A Free user can hit tool fair-use before they burn enrichment credits, or the reverse. Document both on Pricing (PRICING_LIMITS in the frontend).

---

Step 7: Don’t ship a blank screenshot (ops meets marketing)

Growth pages live or die on trust. ScrapeKey’s landing once showed a spinner-only mobile capture for tools, 99% white pixels. Users thought the product was empty.

Treat marketing captures as a release artifact:

1. Log in.
2. Load /tools or run long-tail with a seed.
3. Screenshot with real UI.
4. Check file size / near-white ratio before commit.

Product callout: If the API is down (e.g. host 404), tools and beta unlock both fail. Free-tool marketing should fail soft (“tools temporarily unavailable”), not hang on a spinner forever.

Step 8: Wire the funnel in the UI: not only in the backend

A free-tools strategy fails if navigation is a dead end. ScrapeKey’s pattern:

1. Marketing: landing band + pricing bullets that name the six tools and fair-use numbers.
2. App shell: /tools index cards that deep-link to each tool.
3. In-tool upgrade: Pro banners (“Try Pro for bulk intent analysis”) that explain what unlocks, not just “Upgrade.”
4. Shared vocabulary: monetization badges and Est. labels look the same on tools and project keywords.

When you measure the funnel later, instrument: tool run → project created → match job → checkout. Backend rate limits keep step one from lying about capacity; product copy keeps step three honest.

---

Architecture sketch

[ Landing CTA → Register / Login ]
              │
              ▼
[ /tools/*  FastAPI ]
   ├─ ENABLE_FREE_TOOLS
   ├─ auth (login required)
   ├─ hourly + burst rate limit → 429
   ├─ cache (seed → variants)
   └─ keyword_quality / SERP / density
              │
              ▼
[ Same signals: intent, affiliate_fit, monetization_score ]
              │
              ▼
[ Pro: projects, matcher, higher caps ]

---

Conclusion and next steps

Summary

• Free tools are a funnel, not an open proxy — login + dual-window rate limits saved us.
• Share scoring language with the paid product so tools educate buyers.
• Be honest about Est. metrics; guard URL fetch.
• In-memory limits are an MVP; plan Redis for scale.
• Marketing screenshots are part of quality (learned that one the hard way).

Call to action

Audit your own “free” endpoints: Are they keyed by user? Do they return Retry-After? Would you stake your brand on the metrics they display?

Next: How ScrapeKey scores and ranks affiliate matches. Earlier: plan limits in FastAPI.

References

• MDN: 429 Too Many Requests
• OWASP SSRF
• ScrapeKey: api/routes/tools.py, api/services/tools_rate_limit.py, api/services/keyword_quality.py, frontend/src/constants/pricingPlans.js