Large custom map_layers with tiles

[Mintz]

Something that comes up occasionally is a request to add a custom map_layer to Looker that is very large. Most commonly the questions start when a user realizes that the TopoJSON file for their custom map is hundreds of MB or even GBs and that loading the whole thing into the browser at once isn’t going to work.

Luckily, there are great tools available that are low-cost or even free that make this process reasonably simple. It does require a little bit of patience and some developer tools (like the command line), but if you’ve used Github before, it’s probably well within your skillset.

For the purposes of these instructions, I’m going to assume that you already have a single file that represents your map layer. When you’re uploading the file directly to Looker, that file has to be TopoJSON. But for our purposes, we actually need it to be GeoJSON, KML, or a shapefile. If you have TopoJSON, Mapshaper is a super easy tool for transforming it to GeoJSON.

For this tutorial, I’ll use a GeoJSON file that outlines all 73,057 tracts from the 2010 U.S. Census. This GeoJSON weighs in at just over 1 Gb–definitely not something I want users to load into the browser every time they access it.

But I can take advantage of the same technology that drives Google Maps–tiles–to make the file tractable (pun intended). The basic idea is that if I’m zoomed way in, only a few tracts will be in my viewport, so I only need to load those. Then as I pan, I load the adjacent ones and so on.

If, on the other hand, I’m zoomed way out and can see big regions of the country with thousands of tracts, I don’t actually need all the detail that is in my GeoJSON file. Very rough outlines of the tracts (which weigh far less) will suffice.

Tiling the map at various zoom levels gives me all the raw materials I need to only download the pieces of the map I need right now, at the appropriate level of detail. And Mapbox provides a simple set of tools to do it.

1. So first, go create a free account at Mapbox. Once you’ve done that, create an Access Token and save it for later. (The default public scopes work fine.)

2. Then go into Mapbox’s Studio to upload your data. You want to create a new tileset.

3. Once the tileset is uploaded, it’ll take a bit to process (gotta make all those tiles). After it’s done, click in and you’ll see something like this:
    

   

4. Now comes the trickiest (but not really that tricky) part. We need to calculate the bounding box for each tile region in our shapefile. This is because in order to know which regions to load at any given time, we need to know what the furthest SW and NE points are that encompass the region.

5. We have a little script that’ll do this for you, which you can download here. The instructions are in the script, but you’re going to do the following:

• Install Yarn by opening a command line and typing ( brew install yarn )
• Run Yarn by typing yarn
• Generate the extents.json file by typing yarn run extents /my/input/geojson.json extents.json ZCTA5. Except you’re going to replace /my/input/geojson.json with the actual location of your file and ZCTA5 with the ID field in your shapefile that uniquely identifies each region.

6. Now that you have the extents.json file generated, the last thing you need is to put that somewhere where Looker can access it. You can upload it to an object store like S3 or Google Cloud Storage, but you’ll have to configure the access-control-allow-origin headers properly. You also can’t use Github directly, because it’ll set the wrong “Content-Type Headers.” Luckily, you can upload the file to Github or as a Github gist, and then use this nifty service to fix the headers.

7. Now we have all the pieces we need to properly define our new map layer in Looker! Go to your model in Looker and as part of the model file, define your new map_layer as follows.

map_layer: tract {

  format: "vector_tile_region"

  url: "https://a.tiles.mapbox.com/v4/[YOUR_MAPBOX_TILESET_ID]/{z}/{x}/{y}.mvt?access_token=[YOUR_MAPBOX_ACCESS_TOKEN]"

  feature_key: "[YOUR_MAPBOX_TILESET_NAME]"

  extents_json_url: "https://rawcdn.githack.com/[YOUR_EXTENTS.JSON_FILE_LOCATION]"

  min_zoom_level: [FROM_MAPBOX]

  max_zoom_level: [FROM_MAPBOX]

  property_key: "[THE_UNIQUE_IDENTIFIER_FOR_REGIONS_IN_YOUR_SHAPEFILE]"

}

To illustrate where you’re going to get all this info, see below:

And:

And that’s it! So, not a 5-second process. But it’s totally worth it