FactgetBot
Last updated 18 October 2026
FactgetBot is the web crawler operated by Factget. It collects publicly available business information, such as regulatory filings, company registry records, sanctions lists and company web pages, and turns it into structured datasets for Factget's customers.
| Item | Detail |
|---|---|
| Operator | Factget (Turtlelogix LLP), Bengaluru, India |
| Website | factget.ai |
| Contact | bot@factget.ai |
| Purpose | Scheduled collection of public business data (filings, registries, regulatory publications, product and price pages) on behalf of Factget customers |
Every request includes the token FactgetBot in its user agent. Simple page fetches use:
Mozilla/5.0 (compatible; FactgetBot/1.0; +https://factget.ai/bot)
Pages that need JavaScript are loaded with a headless browser, which uses:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; FactgetBot/1.0; +https://factget.ai/bot) Chrome/W.X.Y.Z Safari/537.36
Requests also carry a From: bot@factget.ai header.
A user agent alone can be copied, so FactgetBot can also be verified in three ways:
- Signed requests. Requests are signed with HTTP Message Signatures (RFC 9421) using the Web Bot Auth profile. Our public keys are published at
https://factget.ai/.well-known/http-message-signatures-directory. - IP addresses. FactgetBot only sends traffic from the addresses listed at
https://factget.ai/bot/ips.json. These addresses are used by FactgetBot and nothing else. - Reverse DNS. Each FactgetBot IP resolves to a hostname under
crawl.factget.ai, and that hostname resolves back to the same IP.
Requests that claim to be FactgetBot but fail both checks are not from us. Please report them to bot@factget.ai.
- Follows
robots.txt, includingCrawl-delay, and rules addressed toFactgetBotor*. - Limits its request rate and parallel connections per site, honours
Retry-After, and slows down when a site responds with errors or rate-limit responses. - Uses conditional requests (
If-Modified-Since,ETag) so unchanged pages are not downloaded again on scheduled reruns. - Honours opt-out requests sent to bot@factget.ai, even where
robots.txthas not been changed. - Collects only publicly accessible pages. It does not log in, does not solve CAPTCHAs and does not try to get around access controls.
- Does not hide its identity, rotate through third-party proxy networks or disguise itself as a browser.
To block FactgetBot from your whole site, add this to your robots.txt:
User-agent: FactgetBot Disallow: /
To block only part of your site:
User-agent: FactgetBot Disallow: /private/
To slow it down:
User-agent: FactgetBot Crawl-delay: 10
Changes to robots.txt are picked up within 24 hours. For anything else, including access questions or a request to stop crawling, email bot@factget.ai. We aim to reply within two business days.
FactgetBot is being registered with the Cloudflare Verified Bots program and the Akamai Bot Directory, so that site operators using those services can identify it and set their own policy for it.