Shipping Free SEO Tools Without Melting Your API (ScrapeKey’s Approach)

Part of Building ScrapeKey — Product Engineering Notes. See also the ScrapeKey case study.

Introduction

The hook

You want a growth wedge: let people use something valuable before they pay.

Then launch week arrives. One scraper’s IP hits your long-tail endpoint a thousand times. Your Suggest proxy gets throttled. Honest users see spinners. Marketing blames “the tools page.”

Free tools without rate limits are not a funnel. They’re a load test you forgot to schedule.

The objective

By the end of this post you’ll know how ScrapeKey ships free SEO tools that:

1. Sit in front of the paid keyword → affiliate workflow.
2. Require login (so abuse keys off accounts, not only IPs).
3. Share a quality pipeline with the product (intent, affiliate fit, monetization score).
4. Fail gracefully with 429 + Retry-After, caching, and URL fetch guards.

Prerequisites

• Basic FastAPI routing and request validation
• Awareness of rate limiting (token bucket / sliding window ideas)
• Optional: React marketing → app tools navigation

---

Body

Step 1: Decide what “free tool” means in the product

ScrapeKey’s free tier is not a toy. The tools map to the same problems paid users solve:

Tool	Job to be done	Why it funnels to Pro
Long-tail generator	Expand a seed into phrases that can pay	Monetization score + Est. metrics tease Pro depth
SERP / meta preview	See how a page might look in Google	Content briefs / matching come next
Keyword density	Check on-page focus	Feeds content workflow
Intent classifier	commercial vs informational	Same intent model as matching
Weekly digest / bookmarklet	Habit + acquisition	Keeps ScrapeKey in the week

Product callout: We label volume and difficulty as Est. Free tools are not “fake Ahrefs.” Trust is the conversion strategy. If you invent exact volumes on a free endpoint, you will refund forever.

---

Step 2: Put tools behind a flag and a login gate — on purpose

ScrapeKey’s tools router is feature-flagged:

ENABLE_FREE_TOOLS = os.getenv("ENABLE_FREE_TOOLS", "true").lower() == "true"
TOOLS_REQUIRE_LOGIN = os.getenv("TOOLS_REQUIRE_LOGIN", "true").lower() == "true"

When login is required, every tool handler depends on get_current_active_user. That choice has tradeoffs:

Approach	Pros	Cons
Fully anonymous	Higher top-of-funnel traffic	IP rotation abuse; hard quotas
Login required (ScrapeKey)	User-keyed limits; email for support	Friction before first delight

We chose login so rate limits can key on user:{id} first, and so Free-plan enrichments and tool hours stack with the same identity as Stripe later.

Product callout: Landing copy can still say “try free tools.” The conversion step is account creation — which is also when welcome email and onboarding fire. Friction is a feature if the tool is good.

---

Step 3: Rate-limit like two windows, not one magic number

ScrapeKey’s tools_rate_limit service uses:

1. Hourly fair-use — default Free 50/hour, Pro 500, Agency 2000 (env overrides).
2. Burst window — e.g. 10 requests / 60 seconds so a loop can’t empty the hour in one second.

RATE_LIMIT_REQUESTS = int(os.getenv("TOOLS_RATE_LIMIT_FREE", "50"))
RATE_LIMIT_REQUESTS_PRO = int(os.getenv("TOOLS_RATE_LIMIT_PRO", "500"))
RATE_LIMIT_WINDOW_SECONDS = int(os.getenv("TOOLS_RATE_LIMIT_WINDOW", "3600"))
BURST_LIMIT_REQUESTS = int(os.getenv("TOOLS_BURST_LIMIT", "10"))
BURST_WINDOW_SECONDS = int(os.getenv("TOOLS_BURST_WINDOW", "60"))

Client key resolution:

def get_client_key(request: Request, user_id: Optional[str] = None) -> str:
    if user_id:
        return f"user:{user_id}"
    # fallback: IP + short UA hash for anonymous mode
    ...

On exceed, return 429 with structured detail and Retry-After:

raise HTTPException(
    status_code=429,
    detail={
        "error": {
            "code": error_code,
            "message": "Rate limit exceeded. Please try again later.",
            "details": {
                "retry_after_seconds": reset_seconds,
                "limit": limit,
                "remaining": 0,
            },
        }
    },
    headers={"Retry-After": str(reset_seconds)},
)

Caveat for hiring conversations: ScrapeKey’s first implementation stores timestamps in memory. Fine for a single Railway instance. Multi-instance needs Redis (or equivalent). Call that out in design reviews — don’t pretend the dict is global truth.

---

Step 4: Share the keyword quality pipeline with the paid product

Long-tail is not a throwaway script. It calls the same discovery path used by richer workflows:

1. Sanitize seed (charset, length).
2. Pull Google Suggest (async) + pattern variants (questions, modifiers).
3. Drop blocklist / off-seed noise.
4. Classify intent and affiliate_fit.
5. Compute monetization_score (0–100) from fit + intent — not from invented volume.
6. Attach estimated metrics, clearly marked.

def _variant_row(keyword: str, source: str, ...) -> Dict[str, Any]:
    intent = classify_intent(keyword)
    fit = classify_affiliate_fit(keyword, intent)
    score = compute_monetization_score(affiliate_fit=fit, intent=intent)
    row = {
        "keyword": keyword,
        "intent": intent,
        "affiliate_fit": fit,
        "monetization_score": score,
        "monetization_tier": monetization_tier(score),
        ...
    }
    return attach_estimated_metrics(row, from_google_suggest=(source == "google_suggest"))

Product callout: Free tools teach the vocabulary of the paid matcher — “commercial,” “high fit,” “monetization score.” When users hit Pro matching, the UI language is already familiar. That is funnel design as API design.

---

Step 5: Guard the dangerous bits — URL fetch and HTML parse

SERP preview and density tools can accept a URL. That opens SSRF and huge download traps.

ScrapeKey centralizes fetch safety (validate_public_http_url, max bytes) before BeautifulSoup. Tutorial rule:

• Only http/https
• No private/link-local ranges
• Cap response size
• Time out aggressively

Never requests.get(user_url) “just for a demo tool.” Demo tools get abused first.

---

Step 6: Cache what you can; count what you must

Repeat seeds hammer Suggest. A short TTL cache (get_cached / set_cached on tools) cuts duplicate upstream work. Rate limits still increment on cache hits or misses depending on your product choice — ScrapeKey counts the API call (user-facing request), which keeps UX predictable: “50 tool runs/hour” means button presses, not unique seeds.

Enrichment quotas (monthly) are separate from tool hourly limits. A Free user can hit tool fair-use before they burn enrichment credits, or the reverse. Document both on Pricing (PRICING_LIMITS in the frontend).

---

Step 7: Don’t ship a blank screenshot (ops meets marketing)

Growth pages live or die on trust. ScrapeKey’s landing once showed a spinner-only mobile capture for tools — 99% white pixels. Users thought the product was empty.

Treat marketing captures as a release artifact:

1. Log in.
2. Load /tools or run long-tail with a seed.
3. Screenshot with real UI.
4. Check file size / near-white ratio before commit.

Product callout: If the API is down (e.g. host 404), tools and beta unlock both fail. Free-tool marketing should fail soft (“tools temporarily unavailable”), not hang on a spinner forever.

Step 8: Wire the funnel in the UI — not only in the backend

A free-tools strategy fails if navigation is a dead end. ScrapeKey’s pattern:

1. Marketing — landing band + pricing bullets that name the six tools and fair-use numbers.
2. App shell — /tools index cards that deep-link to each tool.
3. In-tool upgrade — Pro banners (“Try Pro for bulk intent analysis”) that explain what unlocks, not just “Upgrade.”
4. Shared vocabulary — monetization badges and Est. labels look the same on tools and project keywords.

When you measure the funnel later, instrument: tool run → project created → match job → checkout. Backend rate limits keep step one from lying about capacity; product copy keeps step three honest.

---

Architecture sketch

[ Landing CTA → Register / Login ]
              │
              ▼
[ /tools/*  FastAPI ]
   ├─ ENABLE_FREE_TOOLS
   ├─ auth (login required)
   ├─ hourly + burst rate limit → 429
   ├─ cache (seed → variants)
   └─ keyword_quality / SERP / density
              │
              ▼
[ Same signals: intent, affiliate_fit, monetization_score ]
              │
              ▼
[ Pro: projects, matcher, higher caps ]

---

Conclusion and next steps

Summary

• Free tools are a funnel, not an open proxy — login + dual-window rate limits.
• Share scoring language with the paid product so tools educate buyers.
• Be honest about Est. metrics; guard URL fetch.
• In-memory limits are an MVP; plan Redis for scale.
• Marketing screenshots are part of quality.

Call to action

Audit your own “free” endpoints: Are they keyed by user? Do they return Retry-After? Would you stake your brand on the metrics they display?

Next: How ScrapeKey scores and ranks affiliate matches. Earlier: plan limits in FastAPI.

References

• MDN: 429 Too Many Requests
• OWASP SSRF
• ScrapeKey: api/routes/tools.py, api/services/tools_rate_limit.py, api/services/keyword_quality.py, frontend/src/constants/pricingPlans.js