30 Dec 2016 Generating high-res maps with Mapnik and Docker

I have been working on a project that is a real-estate web portal. One of the main features of the portal is maps. You can search for a property on the map using certain criteria. You can see what other objects – supermarkets, hospitals, kindergartens – are located near it, or you can drill into information about specific properties.

Initially I choose to use OpenStreetMap to render maps, the main reason being the level of detail that it provides. For instance, you can see things like factories, industrial zones, suburb borders, schools, etc. This information is quite important when you are looking for a new place to live.

But (and there’s always a ‘but’), I ran into a problem when I got my first Retina MacBook Pro. Maps looked a bit blurry on it. It turned out that the problem was the Retina (hi-res) display.

What I needed was special tiles that rendered with a bigger scale factor to make them sharper. I was looking for a free, ready-to-use solution, but couldn’t find one. There are some commercial projects that you can use, but they cost money, which we didn’t really have for that project.

Furthermore, I needed to keep tile format the same so I don’t need to make any modifications on front end.

Fortunately, OpenStreetMap not only serves up map tiles, but gives you access to the huge geo database underneath as well. So, I decided to render my own tiles from this database.

In this article I’ll present my solution. Importantly, this solution will use Docker to build and run. Using Docker helps avoid problems running shared libraries across different platforms, because most of the tools in that area written with C++. The hope is that you’ll be able to easily setup this solution in your own environment.

Let’s get started

To support Retina image display, we need to generate tiles with 2x resolution. The web client can then request these images and render them into the same sized area on a web page that was previously being used for the non-Retina images.

For instance, if we have an image ‘foo.png’ with a size 128px x 128px that is being rendered by an tag,  the client will implicitly give that tag an area of 128px x 128px. For a Retina-grade image, we would now have the client request an image with size 256px x 256px, but explicitly keep the display area at 128px x 128px.

So instead of having the client request pre-rendered tiles from the OpenStreetMap.org service, we’ll have it request them from a custom back-end that can render high-resolution tiles. Also, rendering our own tiles can be handy if:

• We want to customise how map looks, i.e. create our own look and feel.
• We want to have control of what objects we want to show on the map (roads/rivers/building/etc).

Generally speaking, if you need flexibility and you have compute resources, then rendering tiles yourself is the way to go. Let’s take a look at the solution!

Solution Overview

We will setup our own render server with a custom map style. That renderer will generate map tiles as png images with scale factor 2. We need a custom style mainly to change the font size, because it’s too big for the new tiles. Otherwise, objects would be rendered closer together, but the font size would remain the same.  Also, it’s handy to be able to change a style to use our own colour scheme or drop some object type that we don’t need.

As an example, our solution will render Melbourne city. It will render a map that will look like this:

Which we can contrast to the original one from openstreetmap.org:

Here are projects that the solution will use to render our tiles:

• Mapnik: It’s our base tool that does the actual rendering. We use it as a library to call from golang to render tiles.
• Postgis + Postgresql: We store the maps in Postgresql DB using the Postgis plugin. Mapnik  requests data from that DB.
• Gopnik:This is a frontend server that calls libmapnik to render tiles. This project is using a master-slave pattern and allows you to create cluster of renderers. Also, it’s written with Golang 😉  – I couldn’t miss that opportunity to hack with it a bit!
• Map Style: A fork of the original OSM style project that I optimised for hi-res displays by supporting 2x size tiles and using different font sizes.

Building It

NOTE: For this to work on your machine, you should have ports 8888,8080,8090 open and free on your host machine.

$ git clone https://github.com/dooman87/melbourne-map.git

$ ./download.sh

Now let’s have a look at docker-compose.yml:

1. Postgis Service

This service runs a database that contains imported data for Melbourne city. This data has been extracted from OpenStreetMap by the guys at the mapzen project, and is also open source (you can find extracts for all big cities there). The Docker image itself is based on a postgis image that was created by mdillon. I added the osm2pgsql package because we depend on it to import map data into the database. The entire Dockerfile is quite small and simple:

We clone the openstreetmap-carto project because it defines what objects we will render, as osm2pgsql will need to know what objects should be imported into the DB. The project contains a file called openstreetmap-carto.style that defines objects and has simple structure of things that will be in DB. For example, here’s a fragment of the file:

Also, I had to customise the Docker entrypoint script to import map data on first start of the container, and create a flag-file to let the renderer know that the database has been successfully imported (last two lines):

2. Renderer Service

This is the renderer that generates tiles using Gopnik. The image below is showing how exactly rendering process is working. “Dispatcher” and “Render” are Gopnik’s components. We are using normal filesystem as a cache. Mapnik is used as a library that “Render” component calls:

Image was created by gopnik team. Original address: http://sputnik-maps.github.io/gopnik/img/render.png

• It installs the Adobe Source Pro fonts that get used by map style
• It clones the map style repo
• It creates indexes from shape files that were downloaded in the first step
• It defines a volume called ‘map_data’. That volume contains a ‘config.json’ file that we pass to gopnik. Most importantly, config.json defines the scale factor (line 72) that we want to use to generate tiles:

"Cmd": ["/gopnik/bin/gopnikslave", 
"-stylesheet", "/openstreetmap-carto/stylesheet.xml", 
"-pluginsPath", "/usr/lib/mapnik/2.2/input", 
"-fontsPath", "/usr/share/fonts", 
"-scaleFactor", "2"]

• It copies scripts to compile the style and run the Gopnik dispatcher and render slaves

3. Web Service

This is a docker container that uses a ready-made nginx image to serve up a simple example web page. It uses the leaflet library to show a map and position at Melbourne:

$ docker-compose up

postgis_1   | LOG:  autovacuum launcher started
renderer_1  | Starting renderer
renderer_1  | 2016/12/27 23:34:30 app.go:266: [INFO] Serving debug data (/debug/vars) on %s... :9080
renderer_1  | 2016/12/27 23:34:30 app.go:267: [INFO] Serving monitoring xml data on %s... :9080
renderer_1  | 2016/12/27 23:34:30 renderselector.go:209: [DEBUG] ping error %v dial tcp 127.0.0.1:8090: getsockopt: connection refused
renderer_1  | 2016/12/27 23:34:30 renderselector.go:117: [DEBUG] '%v' is %v localhost:8090 Offline
renderer_1  | 2016/12/27 23:34:30 main.go:118: [INFO] Starting on %s... :8080
renderer_1  | 2016/12/27 23:34:30 app.go:266: [INFO] Serving debug data (/debug/vars) on %s... :9090
renderer_1  | 2016/12/27 23:34:30 app.go:267: [INFO] Serving monitoring xml data on %s... :9090
renderer_1  | 2016/12/27 23:34:35 main.go:95: [INFO] Done in %v seconds 4.84147165
renderer_1  | 2016/12/27 23:34:35 main.go:103: [INFO] Starting on %s... :8090

http://localhost:8888

Shipping It

How you ship this solution is really up to you. I’m using this in a project that has about 300 hits per day (what a big boy!). I’m running all the containers on one EC2 instance. Also, if you cache all tiles that are already rendered, then you won’t need huge CPU resources and – up to a point – may even have some resilience if the Gopnik renderers go down. You can even get the Gopnik renderers to prerender tiles if you want.

That said, strictly speaking this  is not a production-ready solution. From gopnic’s docs:

Code is unstable. Current status of Gopnik is something like developer preview. Everything may change. Everything may be broken.

Regardless, it’s probably worth mentioning that uptime of my EC2 instance is currently 5412 hours (was launched on May 18, 2016) and it’s still serving tiles.

Where is NEW YORK, PARIS, MOSCOW, MILAN?

It’s quite easy to render other cities. Just update the Postgis Dockerfile to download a different pbf file and update its name in the entrypoint script.

wget https://s3.amazonaws.com/metro-extracts.mapzen.com/melbourne_australia.osm.pbf

osm2pgsql --style /openstreetmap-carto/openstreetmap-carto.style -d gis 
-U postgres -k --slim /melbourne_australia.osm.pbf

$ ./clean.sh
$ docker-compose build
$ docker-compose up

What’s Next?

So that’s my solution for serving up Retina-quality map tiles, packaged up with Docker for easy distribution.

In the ideal world it would be nice to use vector format like svg to render tiles. That way you wouldn’t have to worry about serving up particular resolutions. There are some things going on (http://wiki.openstreetmap.org/wiki/Vector_tiles) in that space, along with commercial solutions.

There is one project that I’d like to mention is osm2vectortiles. You can generate vector tiles from data extracts using it. So, the next step for my project would be to evaluate current solution to generate vector tiles instead of raster.