Meet the bots that made half the web’s traffic in 2014

Bots, those software programs that automate web activity, accounted for 56% of Internet traffic in 2014, according to content delivery platform Incapsula. (This was actually down from 61.5% of traffic in 2013.)

Though you might not notice them, bots are everywhere online—and used for both good and evil. Here are the main types of bots you’ll run across:

• Search engine crawlers index websites so humans can easily find them. They’re harmless.


• RSS bots, also benign, check for and pull in updates from news sites and blogs. (These have been in decline since Google shut down its Reader service in July of 2013.)


• Scrapers scour the web to steal and duplicate content. This is why you might see articles from respectable news organizations on sketchy sites you’ve never heard of. They can also find email addresses on the web and sell them to spammers, another type of bot that distributes malware and phishing links.


• Spy bots scan for website changes and report this information back to competitors.


• Impersonator bots now make up almost one-fourth of bot traffic. They are by far the most sophisticated–and scary. They masquerade as someone or something else in order to gain access to websites. Impersonators can pretend to be search crawlers so they can poke around for vulnerabilities, or disguise themselves as browsers to avoid being blocked by websites. They can also carry out denial of service attacks to overwhelm and take down websites.

Though bots are used by organized cybergangs, it’s typically lone actors who write the malicious software programs. They contribute to the ecosystem by selling and renting bots to others looking to distribute malware or attack websites.

“We are essentially at an arms race [against hackers] trying to protect websites,” Incapsula CEO Marc Gaffan tells Quartz.

Why do bots matter? Beyond the obvious harm done by malicious attacks, even friendly bots can cost us money. The bandwidth they consume loading photos and videos costs just as much as when a real person accesses them.

Bots also contribute to online ad fraud. It’s estimated that 15% of impressions on the web actually come from bots—and when it comes to video ad views, that number’s closer to 25%. While some publishers try to combat the problem, not all are so inclined, fearing a drop in their traffic statistics.

Perhaps they’re looking at the wrong stats? Though bots artificially inflate impressions, they also severely depress engagement rates, considered a more telling metric.

Meet the bots that made half the web’s traffic in 2014