About the LinkOpportunityEngine crawler

LinkOpportunityEngine is a crawler run by ATBI Software. If you've seen this in your server logs, here is what it does and how to stop it.

User agent

LinkOpportunityEngine/1.0 (+https://www.atbisoftware.com/bot)

What it fetches and why

It reads a small number of public web pages, plus each site's robots.txt, to find pages where a link to ATBI Software's own websites might be relevant. It only reads pages: it doesn't fill in forms, sign in, or send or keep cookies.

How it behaves

  • It follows robots.txt, including Crawl-delay. It looks for rules under the name LinkOpportunityEngine, and otherwise follows the rules for all crawlers (*).
  • It makes one request at a time to each site, at least 5 seconds apart, or longer if your Crawl-delay asks for it.
  • It reads at most 25 pages from a site in one run, and doesn't fetch the same page again within 7 days.
  • If your server answers 429 or 503, it waits as long as Retry-After asks, or backs off further each time, and gives up after a couple of tries.

How to opt out

To block it from your whole site, add this to your robots.txt:

User-agent: LinkOpportunityEngine
Disallow: /

Or email info@atbisoftware.com with the domain. We'll stop fetching it, including its subdomains, and won't fetch it again.