Skip to content

SmartSitePlanBot

SmartSitePlanBot is the crawler behind the public map data search in Smart Site Plan. It reads the public directories of ArcGIS REST servers so people can find public map data by place. Smart Site Plan is operated by Juno Maps, LLC.

User-Agent

Mozilla/5.0 (compatible; SmartSitePlanBot/1.0; +https://smartsiteplan.com/contact)

What it does

An ArcGIS REST server publishes a directory of its folders and services, and each service describes itself and its layers. SmartSitePlanBot reads those descriptions, and Smart Site Plan builds a search from them, so a person can ask for sewer mains, parcels or flood zones at a place and find the public layers that cover it. Every result names the server that publishes it.

What it asks a server for:

  • The directory: the list of folders and services at the server's root, and the list inside each folder.
  • Each service's description, and the descriptions of its layers: name, extent, geometry type and fields.
  • Feature counts: how many features each layer holds. Where a layer's extent cannot be worked out from its description, one more request asks that layer for its extent. Neither request returns the features themselves.
  • A later check that a service it already knows still answers: the service's description again.
  • Where a layer's data lies: a count of the layer's features and, for a smaller layer, a page of its feature outlines, so a search can tell a layer that covers a place from one that only says it does.
  • A small preview image of a layer. It asks the server to draw the layer as a map image, or for map tiles the server has already drawn, or for a limited sample of the layer's features in the pictured area so the preview can be drawn from them. A stored preview is redrawn only once it is more than 30 days old.

What it does not do

  • It does not log in. It sends no user name, password or sign-in token, and a service that asks for one is treated as not public: after repeated checks it drops out of the search.
  • It does not use what it reads to train AI models.

How it behaves

  • On a server run by one publisher, each crawl job (walking the directory, or checking that services still answer) keeps no more than two requests open at a time. The shared ArcGIS Online hosting servers, where one host name serves many organizations, are allowed more.
  • A service usually costs two or three requests per visit, however many layers it has.
  • Every request has a time limit, so a slow server is never left holding open connections.
  • When a server does not answer at all, it leaves that server alone: for about 15 minutes after the first failure, then twice as long after each further failure in a row, up to one day.
  • When a server answers that it is busy (HTTP 429, 502, 503 or 504), it waits about an hour before asking for that service again, and twice as long each time the answer repeats.
  • When a person searches for map data at a place, Smart Site Plan asks the matching layers what they hold there and for a preview of it, with the same User-Agent. Those requests are limited per server too.

robots.txt

SmartSitePlanBot obeys robots.txt on every server it crawls. It follows the rules addressed to SmartSitePlanBot, or the rules for * when the file has none for it. It keeps a copy of the file for up to 24 hours, so a change can take a day to reach it. A path the file closes is never requested, and that alone does not remove a layer from the search: the layer stays listed, unchecked.

A Crawl-delay of up to two seconds is kept between its requests to a server. A server that asks for a longer pause is left alone, as if it had closed its paths.

To learn where public servers are, it also queries a few public APIs as an ordinary client, under each API's own usage rules: the ArcGIS Online and ArcGIS Hub search APIs, the Wikidata query service and Certificate Transparency logs.

To keep it off a server entirely:

User-agent: SmartSitePlanBot
Disallow: /

To keep it out of one folder only:

User-agent: SmartSitePlanBot
Disallow: /arcgis/rest/services/Draft/

Checking that a request is really ours

Anyone can copy a User-Agent, so SmartSitePlanBot signs its requests with HTTP Message Signatures (RFC 9421), the method Web Bot Auth uses. Each signature is made with an Ed25519 key, covers the host the request is sent to, and expires after one minute.

The request's Signature-Agent header names this website, https://smartsiteplan.com, and the public keys that verify the signature are published in its key directory:

https://smartsiteplan.com/.well-known/http-message-signatures-directory

Reaching us, or asking to be left out

If SmartSitePlanBot is causing trouble for your server, or you want your server left out, write to us through the contact page or at support@smartsiteplan.com and name the host. We can block a server on our side, and the crawler then stops contacting it. You can also turn it away yourself with robots.txt, as shown above.