How can you detect and prevent malicious bot traffic? In this article, we have you covered. Read on to learn more.
Key statistics
- In 2022, bot traffic accounted for more than 40% of the total internet traffic.
- The cost of bot traffic, as measured through digital ad fraud, is projected to reach $100 billion in 2023.
- Businesses are leaving bot attacks unaddressed for an average of nearly four months.
The latest bot traffic trends
Last year, amidst record-breaking Thanksgiving weekend sales, there was an uninvited guest – bot traffic.
The National Retail Federation reported that a total of 196.7 million Americans shopped from Thanksgiving Day through Cyber Monday, representing an increase of 17 million shoppers compared to the previous year. Cyber Monday revenue hit a new record and the business growth looked promising.
However, one cyber security firm estimates that a third of Black Friday shoppers were fake. And on Cyber Monday, one in five site visits likely came from bot traffic. In case you’re unfamiliar, bots are automated scripts designed to make you think that they are real people.
“Within digital marketing and ecommerce, it is evident that a rise in adversaries’ cyber capabilities is resulting in direct and evident revenue loss to businesses around the globe,” says Lior Frenkel, Chairman of the Israel Cybersecurity Forum, in reference to fake online traffic.
Bot traffic, bad for business
Malicious bot traffic can have financial implications for enterprises, even if website performance is unaffected. For instance, sites that depend on advertising and that sell merchandise with limited inventory are particularly vulnerable to bot traffic issues.
- For websites that depend on ad revenue, bots that access the site and that click on page elements can trigger fake ad clicks. This is commonly known as click-fraud. While this may initially result in an ad revenue increase, over the long term, online advertising networks may detect the bots, and take action. In turn, an organization may not be able to continue advertising across that network. On this account, it’s important that site owners remain aware of and take action around bot click fraud.
- E-commerce sites that sell limited inventory or services are also threatened by bots. For instance, scalpers have famously used bots to snap up high-value items, from gaming consoles to event tickets. In other cases, bot owners have simply directed their minions to dump a vendor’s merchandise into their online shopping carts, making the inventory temporarily unavailable to legitimate shoppers.
More on why you should block bots
When you block bots, you can:
- Reduce your IT spend. Malicious bots take up bandwidth and increase the bills from your server, API and CDN providers. When you block these types of bots, costs should decline.
- Protect the user experience. An increase in bot traffic can reduce the speed of your website, or in some cases, it may crash the website altogether. This results in a negative user-interface experience, which tarnishes your brand’s reputation.
- Stay ahead of competition. Your competitors may rely on bots that scrape your business’s prices and content for their own gain. Effective bot prevention and detection renders it challenging for competitors to obtain these insights.
- Spend less time on business emergencies. A bot attack can potentially affect all of your enterprise’s departments; from IT, to customer support, to the marketing team. Preventing harmful bot traffic will mean that your departments won’t have to spend precious time in crisis mode due to a bot attack.
- Remain in compliance. Organizations that prevent bots can more easily remain in compliance with data protection frameworks than otherwise. In turn, organizations are less likely to experience data compromise, reducing the probability of receiving a heavy fine from a regulatory body.
How to detect bots
Identifying potentially harmful bot traffic often requires significant analysis and sophistication. However, the following tips can help. You might suspect bot traffic if:
- You see irregular spikes in website traffic.
- A single channel is contributing to a large number of sessions or users.
- You observe an increase in activity on your website that comes from a remote location.
- You see a large number of hits from a single IP address within a short time span.
- You observe a surge in phony-looking conversions, including accounts created using gibberish email addresses or contact forms that clearly display fake names and fake phone numbers.
Prevent harmful bots
- Block or require CAPTCHA for outdated browsers. For many tools and scripts, the default configurations contain user-agent string lists that are outdated. Blocking or requiring CAPTCHA for outdated browsers will not prevent the more advanced bot operators, but it may discourage a few. The risk involved in blocking outdated browsers is fairly low. And the majority of modern browsers force auto-updates on users, making it difficult for users to surf the web with outdated browsers.
- Block known hosting providers and proxy services. While the most advanced cyber adversaries may switch to the more difficult-to-block networks, less sophisticated adversaries may use easily accessible hosting and proxy services. Disallowing access to your site from these types of sources may discourage attackers from weaponizing bots against your API, your website and your mobile apps.
- Protect all access points. Ensure that your organization protects exposed APIs and mobile apps across your systems. Implementing protection for your website alone is like blocking the front door while leaving the side door and the back door open.
- Investigate traffic spikes. An organization may accidentally mistake high volumes of traffic for a win. However, the traffic might actually just be bots. If you see a clear spike in traffic, ensure that you and your IT department can find an explanation for it – hopefully, one that is not related to bot activity.
- Keep an eye on public data breaches. In the immediate aftermath of a data breach, stolen credentials may still be active. In turn, cyber criminals may attempt to run them against your login set-up.
Bot blocking tools
In the past, blocking bots often depended on rule-based measures, such as blocking IP ranges, countries or data centers known to host bots. Web Application Firewalls (WAF) and Access Control Lists (ACL) have also been leveraged for the purpose of detecting and blocking harmful bots. Yet, the aforementioned methodologies are not always as effective as dedicated endpoint management solutions, ad-fraud solutions, and/or specific bot-blocking technologies.
Good bots vs. bad bots
While bot traffic can be harmful, it’s worth noting that there are ‘good’ bots. As the name implies, these bots do not result in harm to your website or server. Rather, they announce themselves and inform website owners about how they will interact with a website. It is normal for a website to have a small percentage of traffic that stems from ‘good’ bots.
Here’s a short list of ‘good bot’ varieties:
- Search engine bots can crawl the web and assist website owners in ensuring that their domains are appropriately listed on Google, Yahoo and Bing.
- SEO crawlers are software that obtains and indexes a website and its competitors, ultimately providing data and analytics on page views, users and content.
- Copyright bots can scour the internet for copyrighted images, ensuring that copyrighted content is not used inappropriately.
- Monitoring bots assist publishers in ensuring that a website is performing at peak and that it’s easily accessible to all users. Should any aspect of the website break or go offline, the publisher will receive an automatic notification, rendering these bots particularly useful to website owners.
- Feed/aggregator bots collect and aggregate newsworthy content to send to your website visitors or email newsletter subscribers.
The bots that you don’t want on your site include:
- Email scraper bots. These bots harvest email addresses and deploy malicious emails.
- Comment spam bots. These bots spam your website with comments and links that redirect users to malicious sites. These bots may also spam your website in order to advertise or to try to acquire backlinks to their sites.
- Scrapers bots. These bots will bombard a website and download everything available. That includes text, images, HTML files, and videos. The bot operators then reuse your content in violation of copyright laws.
- Bots for credential stuffing or brute force attacks. These types of bots try to gain access to a website in order to pinch sensitive information. They do so by attempting to login as a legitimate website user.
- Botnets. These are linked networks of infected machines that perform distributed denial-of-service attacks. During a distributed denial of service attack, an adversary will flood a website with bot traffic. In turn, the web server becomes overwhelmed with requests, resulting in a lethargic website.
- Inventory and ticket bots. As mentioned earlier, these bots scalp goods and services. Operators may then resell the goods or services at a higher price than initially offered, in order to achieve significant financial gain.
In conclusion
Bots can place a costly strain on IT staff and operational resources, as they imitate human behavior and can sneak past traditional security tools. Protect your organization from automated threats that can negatively affect your business outcomes.
For insights into detecting and blocking advanced bot traffic, see this AWS blog post. Lastly, to receive cutting-edge cyber security news, exclusive interviews, expert analyses and security resources, please sign up for the CyberTalk.org newsletter.