About the LinkOpportunityEngine crawler
LinkOpportunityEngine is a crawler run by ATBI Software. If you've seen this in your server logs, here is what it does and how to stop it.
User agent
LinkOpportunityEngine/1.0 (+https://www.atbisoftware.com/bot)
What it fetches and why
It reads a small number of public web pages, plus each site's robots.txt, to find pages where a link to ATBI Software's own websites might be relevant. It only reads pages: it doesn't fill in forms, sign in, or send or keep cookies.
How it behaves
- It follows
robots.txt, includingCrawl-delay. It looks for rules under the nameLinkOpportunityEngine, and otherwise follows the rules for all crawlers (*). - It makes one request at a time to each site, at least 5 seconds apart, or longer if your
Crawl-delayasks for it. - It reads at most 25 pages from a site in one run, and doesn't fetch the same page again within 7 days.
- If your server answers 429 or 503, it waits as long as
Retry-Afterasks, or backs off further each time, and gives up after a couple of tries.
How to opt out
To block it from your whole site, add this to your robots.txt:
User-agent: LinkOpportunityEngine
Disallow: /
Or email info@atbisoftware.com with the domain. We'll stop fetching it, including its subdomains, and won't fetch it again.