Crawler

PageCop fetches product URLs to compare them with a Shopify catalog. This page describes that crawler.

User agent

The default user agent is:

``` Mozilla/5.0 (compatible; PageCop/1.0; +https://pagecop.com/bot) ```

The token names this app and points here. It is not Googlebot, AdsBot, or any other Google crawler. Google asks other clients not to use the Googlebot token, and a shop that checks reverse DNS will not treat this request as Google.

What a fetch assumes

The crawler does not run a browser and does not execute JavaScript. It reads the server-rendered HTML: status, redirects, JSON-LD, microdata, and a product JSON object when the theme includes one.

The assumption is the same one Google's smartphone crawler uses when the offer is already in the HTML. One product URL is fetched. There is no second desktop pass. A theme that builds the price only after JavaScript runs will not match what Google renders. That gap is filed as possible, not as a confirmed mismatch.

robots.txt is still read with Googlebot's rules, because Merchant Center uses that token to decide if a product URL can be crawled.

Optional Googlebot fetch

A merchant can turn on a second request that sends Google's smartphone user-agent string. The setting is off by default. It applies only to that merchant's own store, not to the public URL checker. The scan still sends the PageCop token first. If status, the final URL, the offer, or the add-to-cart control differ, the scan files Googlebot page differs and says the second request was not from Google. Some themes and apps serve Googlebot a different page from the one shoppers get. That is why the switch exists. Leave it off unless you have seen that split.

Politeness

Catalog scans wait about 400 milliseconds between starts and keep only a few requests in flight. A long scan is split into batches so one shop does not hold the worker. The public checker allows 8 checks per 10 minutes per client.

Contact

To ask about a fetch, email support@pagecop.com. Include the URL and the time. Do not send storefront passwords or API secrets.