How to Distinguish Legitimate Users from Malicious Bots

Correct the premise first: bots are not automatically bad

Type	Typical example	Should you block it?
Search engine crawlers	Major search engines, indexing AI fetchers	No — verify identity, then allow
Commercial API callers	Partners, customer integrations	No — separate tokens and quotas
Uptime and monitoring	Synthetic checks, health probes	No — allow-list them
Malicious scrapers	Bulk collection of content, catalogues or pricing	Yes
Abuse scripts	Fake engagement, free-tier farming, credential stuffing	Yes

Treating “automated” as malicious hurts your search visibility and your partners. Treating it as harmless burns your budget and your content rights. The real job is not telling humans from machines; it is telling authorised automation from unauthorised automation.

Three layers of evidence

Credential layer: who you claim to be. Signed tokens, mTLS certificates, API keys, OAuth claims. Strongest of the three, and the one engineering teams most often ignore — because most systems use credentials only for authorisation, never as a risk signal.

Environment layer: what you arrived in. TLS fingerprint (handshake parameter combination), HTTP/2 frame ordering, browser environment integrity, whether a JavaScript challenge was executed. Scripts issuing raw requests differ from real browsers here, and changing the User-Agent does not close the gap. Cloudflare Bot Management collects these signals at the edge and, combined with a model trained on global traffic, returns a Bot Score (1–99 automation confidence).

Behaviour layer: what you are doing. Is the cadence mechanically uniform? Do you only touch generation endpoints? Do sessions start but never heartbeat? Does the endpoint sequence match a real playback or browsing path? Hardest to forge, but it needs a long enough observation window.

Reliability of the three layers

Layer	Forgery difficulty	Typical bypass	Suggested weight
Credential	High (needs a real credential)	Leaked or shared credentials	High, but only on authenticated endpoints
Environment	High (needs protocol-stack changes)	Headless kernels, real browser drivers	High — the main scoring input
Behaviour	Highest (needs human mimicry)	Slow, distributed, human-solving farms	Medium-high, needs observation window
Origin (IP / UA / Referer)	Low	Proxy pools, UA spoofing	Low — supporting signal only

Origin signals are what every team reaches for first and what attackers forge most cheaply: the UA is one header, the Referer is client-controlled, and home broadband and mobile networks share egress. Make IP your primary test and you get high false positives and high misses at the same time.

A decision order that works

1. Release verified good bots first. Major search engines have verifiable reverse DNS and published fetcher declarations — confirm them and allow-list. Partners and health checks use separate tokens or fixed egress ranges. The point is removing noise from the sample so thresholds downstream can be tightened.

2. Credential checks. If a token is present, validate it. Requests without one fall through to the next layer rather than being rejected outright.

3. Environment fingerprint scoring. Use edge-collected signals to produce automation confidence. This is the workhorse, but it outputs a probability, not a verdict.

4. Behaviour analysis as backstop. Extend observation for the middle band: cadence, endpoint mix, session completeness. Let suspicious traffic expose itself.

5. Grade the response — never just allow or deny.

Graded response matrix

Verdict band	Response	Note
Verified good bot	Allow + separate metering	Still measure consumption so partners cannot run away
Low suspicion	Allow + log	Keep samples for review
Medium suspicion	Invisible challenge / degrade / queue	Prefer frictionless checks so real users are not interrupted
High suspicion	Explicit challenge	Confirm humanity, then remember the result briefly
Clearly malicious	Deny / rate-limit / block	Only ban sustained, attributable sources

The middle band is where the return is highest: denying it causes false positives, allowing it causes misses — handling it with invisible challenges or degradation avoids both errors.

Caveats and where this does not apply

• SEO that depends on long-tail crawlers should not blanket-block unknown user agents. Grade by behaviour and consumption first; killing a new fetcher means giving up a new indexing channel.
• Authorised but over-consuming partners are a quota problem, not a bot problem. Handle them with budgets and alerting, not challenges.
• Where privacy compliance limits fingerprinting, raise the weight of credential and behaviour layers rather than leaning entirely on environment signals.

FAQ

What Bot Score should we block at? Do not pick one global threshold. Group by endpoint and look at the distribution: login, credit-claiming and generation endpoints each deserve their own threshold, because the cost of a false positive differs completely.

We hurt a search engine crawler. What now? Add it to the allow-list instead of lowering the overall threshold. Lowering the threshold to let crawlers in also lets malicious traffic in — a textbook case of the wrong fix.

Why are we still being farmed after adding a JS challenge? Because advanced automation runs real browser engines that execute JavaScript. JS challenges stop direct scripts, not headless browsers; this layer only pays off when combined with environment fingerprints and behaviour analysis.

Can headless browsers be fully detected? No. The goal of detection is raising the attacker’s engineering cost until scale stops being profitable. Accept a small miss rate and invest in quotas, degradation and cost circuit-breakers — that is more practical than chasing perfect detection.