The bot invasion: why we need smarter defenses

TL;DR

• AI bot traffic now makes up over 50% of all web traffic (and climbing fast)
• Traditional defenses fail for three critical reasons: contextual blindness, overwhelming false positives and unsustainable maintenance
• Most security teams abandon proper firewall tuning because it’s too time-consuming
• This means that we need new, smarter defenses
• The future is about inside-out protection: embedding security directly in your application

Welcome to the bot apocalypse

The web is being overrun by AI bots, and they’re becoming impossible to stop with traditional methods. Last month, Meta’s AI crawlers caused chaos by hammering websites with automated traffic that mimicked distributed denial of service (DDoS) attacks and made the sites unavailable to legitimate users.

A real-world example of the impact of this bot apocalypse came from Gergely Orosz of The Pragmatic Engineer, who watched his infrastructure nearly collapse under the weight of the bots. His bandwidth usage jumped from 100GB to over 700GB monthly— from Meta’s crawlers and other bots mindlessly scraping his content. His hosting costs skyrocketed, meaning he literally paid for these bots’ training data. And he wasn’t alone.

What makes all of this particularly dangerous is that modern bots can now mimic human behaviour with remarkable accuracy, making traditional perimeter defenses nearly useless. And the problem is only getting worse.

Today’s bots are smarter than your defenses

Modern bots execute sophisticated objectives that go way beyond basic scraping:

• Credential stuffing: Executing distributed authentication attempts using harvested credentials
• Account takeover campaigns: Hammering authentication endpoints with brute-force attempts
• API abuse vectors: Exploiting rate-limit weaknesses and implementation gaps
• Resource exhaustion attacks: Overwhelming infrastructure with legitimate-appearing requests
• Targeted data harvesting: Extracting sensitive information while bypassing standard protections

Why your firewall is basically useless now

Existing defensive architecture is fundamentally unprepared for these types of threats. Current firewalls sit at your network’s edge, attempting to filter traffic through increasingly inadequate pattern matching and outdated blocklists. These approaches fail for three critical reasons:

• Contextual blindness: Firewalls see the requests but have zero insight into your application’s execution context
• False positive overload: When nearly half your traffic looks potentially suspicious, how can you accurately identify real threats?
• Unsustainable maintenance: Most security teams abandon proper firewall tuning because it takes too much time, leading to either disabled protections or ignored alerts

The case for inside-out protection

So what’s the solution? We need to flip the model completely.

Rather than trying to detect bots at the perimeter, the most effective approach embeds security directly inside your application. Like an inside bodyguard stopping threats at the gates – rather than an external wall – this approach:

• Provides deeper context awareness, seeing how requests interact with your actual code
• Reduces false positives by 85% through understanding the true impact of each request
• Works immediately without constant rule updates or complex tuning
• Blocks malicious bots based on behaviour, not just signatures

This makes the entire management process dramatically easier for developers and security teams alike.

The future: dual-layer defense framework

The most effective strategy combines community-driven intelligence with in-app protection:

1. Leverage collective threat intelligence: When a malicious bot hits one application, that signature can immediately protect all others.
2. Deploy embedded runtime protection: Use in-app solutions that understand code context to catch what perimeter defenses miss.
3. Focus on behaviour, not identity: Monitor what bots actually do, not just their identification signals.

Final word: a checklist for securing your infrastructure

• Audit perimeter defenses against modern bot threats
• Implement app-level monitoring for request context
• Add in-app protection to complement existing security
• Use community-driven threat intelligence
• Deploy dual-layer defense (perimeter + in-app)
• Measure detection quality and false positive rates