How ScrapeKey Enforces Free, Pro, and Agency Limits in a FastAPI Backend | Darrell Kilo

ScrapeKey Pricing page showing Free, Pro, and Agency plan cards

Part of Building ScrapeKey — Product Engineering Notes. See also the ScrapeKey case study.

Introduction

The hook

You ship a pricing page that says “5 projects on Free” and “unlimited on Pro.”

A week later a free user has twelve projects.

The bug usually is not Stripe. It’s that marketing copy and the API never shared one source of truth. The frontend showed a lock icon. The backend still said yes.

The objective

By the end of this post you’ll know how ScrapeKey enforces plan limits so that:

Tier is resolved from the database (not only from the JWT).
Feature gates and usage caps use the same resolver.
Pricing UI and backend env defaults stay aligned on purpose.

You’ll see the patterns we use in production: a tier ladder, a project create guard, and monthly quotas for enrichments and API usage.

Prerequisites

Basic FastAPI (dependencies, HTTPException)
Familiarity with a document store (we use MongoDB via MongoEngine)
Comfort with the idea of subscription plan_name + status

Body

Step 1: Name the tiers like a product, enforce them like an API

ScrapeKey’s public plans are Free, Pro, and Agency.

Internally we treat them as an ordered ladder:

Tier	Level	Product idea (short)
`free`	0	Try the keyword → affiliate workflow
`pro`	1	Unlimited projects + paid-tool API access
`agency`	2	Higher caps (team features still rolling out)

Product callout: Agency can be “coming soon” on the marketing site and still exist in the backend ladder. That lets us raise caps and gate routes without rewriting every check later.

Feature checks ask: “Is this user’s level ≥ the required level?”
Usage caps ask: “Which tier are they on, and what is that tier’s monthly budget?”

Those are different questions. Don’t collapse them into one if plan == "pro" soup.

Step 2: Resolve tier from the database (JWT is a cache, not the truth)

JWTs in ScrapeKey can carry a tier claim, but plan changes (upgrade, cancel, admin override) must win immediately. So privilege checks re-read subscription state:

def get_user_subscription_tier(user_id: str) -> str:
    user = User.objects(id=user_id).first()
    if not user:
        return "free"
    subscription = Subscription.objects(user=user).first()
    if not subscription or subscription.status != "active":
        return "free"
    return subscription.plan_name

Rules that matter in interviews and production:

Missing user → free
Missing subscription → free
Non-active status → free (canceled/past_due do not keep Pro)

Then feature gating becomes a small ladder:

def validate_tier_access(required_tier: str, current_user: TokenData = Depends(...)):
    tier_levels = {"free": 0, "pro": 1, "agency": 2}
    user_tier = get_user_subscription_tier(current_user.user_id)
    if tier_levels.get(user_tier, 0) < tier_levels.get(required_tier, 0):
        raise HTTPException(
            status_code=403,
            detail=f"Access denied. {required_tier.capitalize()} subscription required.",
        )
    return current_user

On Pro-only routes we declare the dependency once:

@app.get("/some-pro-feature")
async def pro_feature(current_user = Depends(lambda: validate_tier_access("pro", ...))):
    ...

(In ScrapeKey this is validate_tier_access("pro", current_user) at the start of the handler.)

Product callout: Returning a clear 403 string (“Pro subscription required”) is better UX than a generic 404. Pricing and upgrade CTAs can key off that message.

Step 3: Separate “who they are” from “what they’ve used this month”

Role is not the same as plan. ScrapeKey operators (owner / admin) need Agency-level caps while testing without buying a seat every time. We wrap that in one resolver used by every quota:

FREE_MAX_PROJECTS = int(os.getenv("FREE_MAX_PROJECTS", "5"))

def resolve_plan_tier(user: Optional[User]) -> str:
    if not user:
        return "free"
    role = str(getattr(user, "role", "") or "").lower()
    if role in ("owner", "admin"):
        return "agency"
    return get_user_subscription_tier(str(user.id))

Then project creation is a hard gate for Free only:

def assert_can_create_project(user: User) -> None:
    tier = resolve_plan_tier(user)
    if tier != "free":
        return
    used = Project.objects(user=user).count()
    if used >= FREE_MAX_PROJECTS:
        raise HTTPException(
            status_code=403,
            detail=(
                f"Free plan project limit reached ({FREE_MAX_PROJECTS} projects). "
                "Upgrade to Pro for unlimited projects."
            ),
        )

Call this inside POST /projects (or equivalent) before save(). Never only disable the button on the client.

We also expose a small status payload for the UI (used / limit / remaining) so the subscriptions page can show progress without inventing numbers.

Step 4: Cap the expensive work — enrichments and API calls

Projects are storage. Enrichments (estimated volume/difficulty, monetization inputs) and paid-tool API usage are the meters that actually cost you.

ScrapeKey keeps env-tunable defaults, for example:

Cap	Free	Pro	Agency
Projects	5	unlimited	unlimited
Enrichments / month	~100	~1,000	~5,000
Tool requests / hour	50	500	2,000
Paid API requests / month	—	1,000	10,000

The enrichment path looks like this (simplified):

Resolve tier via resolve_plan_tier.
check_enrichment_quota(user, units=1).
If over quota: API → 429 with structured detail; background scrape → skip quietly (strict=False) so a batch job doesn’t die mid-run.

def enrich_keyword_document(keyword_doc, user, *, strict: True) -> bool:
    if not metrics_engine_enabled():
        return False
    allowed, quota_info = check_enrichment_quota(user, units=1)
    if not allowed:
        if strict:
            raise HTTPException(status_code=429, detail=quota_info)
        return False
    apply_metrics_to_keyword_document(keyword_doc)
    return True

Product callout: ScrapeKey labels volume/difficulty as Est. We sold honesty, not Ahrefs. Quota messages say “monthly enrichment limit” — not “your SEO data is broken.” That keeps support calm when Free users hit the ceiling.

API usage for paid tools follows the same shape: resolve tier → compare month-to-date count → allow or 429.

Step 5: Keep the frontend honest — one constants file that mirrors the backend

A surprising number of “plan bugs” are copy drift: pricing said 5, env said 10, toast said unlimited.

In ScrapeKey, frontend/src/constants/pricingPlans.js documents the alignment explicitly:

/**
 * Pricing copy aligned with backend plan caps
 * (api/services/plan_limits.py, api_usage.py).
 */
export const PRICING_LIMITS = {
  free: { maxProjects: 5, enrichmentsPerMonth: 100, toolRequestsPerHour: 50 },
  pro:  { apiRequestsPerMonth: 1000, enrichmentsPerMonth: 1000, toolRequestsPerHour: 500 },
  agency: { apiRequestsPerMonth: 10000, enrichmentsPerMonth: 5000, toolRequestsPerHour: 2000 },
};

Landing cards and the subscriptions page both import from here. When we change a cap, we change:

Backend env default (or Railway variable)
PRICING_LIMITS / feature bullets
Any admin/comp scripts that assume old numbers

There is no magical sync. The discipline is the product process.

Product callout: We redesigned the subscriptions page as stacked plan cards (no horizontal scroll) because Free users need to see the same five-project story the API enforces. Layout and limits are one narrative.

Step 6: Test the unpleasant paths

Tutorial value is thin if you only test “Pro can access Pro.”

Minimum checklist ScrapeKey cares about:

Free user at 5 projects → create project → 403 with upgrade text.
Free user with canceled Pro sub → treated as free.
Pro-only route as Free → 403.
Enrichment over quota (strict) → 429; background path → skip, no crash.
Owner/admin → Agency-level caps without a paid Agency subscription.

Automate what you can in pytest; the rest is a 10-minute smoke against staging.

Architecture sketch (for the cover / body figure)

[ React Pricing / Subscriptions ]
            │  copy from PRICING_LIMITS
            ▼
[ FastAPI routes ]
   ├─ validate_tier_access("pro"|"agency")  ← feature gates
   ├─ assert_can_create_project()         ← Free project cap
   └─ check_enrichment_quota / api_usage  ← monthly/hourly meters
            │
            ▼
[ resolve_plan_tier(user) ]
   ├─ owner/admin → agency
   └─ else Subscription.plan_name if status==active else free

Conclusion and next steps

Summary

One resolver (resolve_plan_tier / DB-backed tier) feeds both feature gates and usage caps.
Ladder levels make “Pro includes Free rights” trivial.
Hard checks live in the API; the UI only reflects them.
Marketing and backend share declared defaults — and you update both when caps change.
429 vs 403 matters: quota exhausted is not the same as “wrong plan.”

Call to action

If you’re building a SaaS with Free/Pro/Agency (or similar): audit one path this week — project create, export, or “AI enrichment.” Does the backend reject Free the same way the pricing page promises?

Next in this series: Shipping free SEO tools without melting your API, then affiliate match scoring.

References

FastAPI Dependencies
HTTP 403 vs 429 (plan vs quota)
ScrapeKey implementation (private/repo): api/services/plan_limits.py, api/auth.py (validate_tier_access), api/services/enrichment_usage.py, api/services/api_usage.py, frontend/src/constants/pricingPlans.js