HexCoreScanner

HexCoreScanner is the crawler HexCore uses to look at public storefront pages. If you found it in your server logs, this page explains what it did and how to make it stop.

What it identifies as

Its user-agent string ends with HexCoreScanner/0.1 (+https://hexcore.us/scanner).

What it does

• Opens the home page in a headless browser, follows its links, and visits at most 8 public pages per run: up to two each of category, product, cart, checkout and content pages. A page that fails or is blocked is recorded as such and requested once, not retried.
• Reads your robots.txt first. Paths you disallow are not requested, and a Crawl-delay is honoured up to 30 seconds. If you set none, requests are spaced only by page load time.
• Does not log in, submit forms or enter payment details. It follows links only.
• Because it renders pages in a real browser, your page scripts run as they would for a visitor. It records which scripts and third-party hosts load, and runs automated accessibility checks on the rendered page.

Stop it

Add this to your robots.txt and it will not request any page:

User-agent: HexCoreScanner
Disallow: /

Or email [email protected] with your domain and the word "never". We add it to a do-not-scan list and do not ask why.

Who runs it

HexCore Corp. dba HexCore, San Jose, California. Back to the main site.