Generating high-res maps with Mapnik and Docker

30 Dec 2016 Generating high-res maps with Mapnik and Docker

Posted by

I have been working on a project that is a real-estate web portal. One of the main features of the portal is maps. You can search for a property on the map using certain criteria. You can see what other objects – supermarkets, hospitals, kindergartens – are located near it, or you can drill into information about specific properties.

Initially I choose to use OpenStreetMap to render maps, the main reason being the level of detail that it provides. For instance, you can see things like factories, industrial zones, suburb borders, schools, etc. This information is quite important when you are looking for a new place to live.

But (and there’s always a ‘but’), I ran into a problem when I got my first Retina MacBook Pro. Maps looked a bit blurry on it. It turned out that the problem was the Retina (hi-res) display.

What I needed was special tiles that rendered with a bigger scale factor to make them sharper. I was looking for a free, ready-to-use solution, but couldn’t find one. There are some commercial projects that you can use, but they cost money, which we didn’t really have for that project.

Furthermore, I needed to keep tile format the same so I don’t need to make any modifications on front end.

Fortunately, OpenStreetMap not only serves up map tiles, but gives you access to the huge geo database underneath as well. So, I decided to render my own tiles from this database.

In this article I’ll present my solution. Importantly, this solution will use Docker to build and run. Using Docker helps avoid problems running shared libraries across different platforms, because most of the tools in that area written with C++. The hope is that you’ll be able to easily setup this solution in your own environment.

Let’s get started

To support Retina image display, we need to generate tiles with 2x resolution. The web client can then request these images and render them into the same sized area on a web page that was previously being used for the non-Retina images.

For instance, if we have an image ‘foo.png’ with a size 128px x 128px that is being rendered by an tag,  the client will implicitly give that tag an area of 128px x 128px. For a Retina-grade image, we would now have the client request an image with size 256px x 256px, but explicitly keep the display area at 128px x 128px.

So instead of having the client request pre-rendered tiles from the OpenStreetMap.org service, we’ll have it request them from a custom back-end that can render high-resolution tiles. Also, rendering our own tiles can be handy if:

  • We want to customise how map looks, i.e. create our own look and feel.
  • We want to have control of what objects we want to show on the map (roads/rivers/building/etc).

Generally speaking, if you need flexibility and you have compute resources, then rendering tiles yourself is the way to go. Let’s take a look at the solution!

Solution Overview

We will setup our own render server with a custom map style. That renderer will generate map tiles as png images with scale factor 2. We need a custom style mainly to change the font size, because it’s too big for the new tiles. Otherwise, objects would be rendered closer together, but the font size would remain the same.  Also, it’s handy to be able to change a style to use our own colour scheme or drop some object type that we don’t need.

As an example, our solution will render Melbourne city. It will render a map that will look like this:screen-shot-2016-12-28-at-12-06-36-pm

Which we can contrast to the original one from openstreetmap.org:screen-shot-2016-12-28-at-12-06-27-pm

Here are projects that the solution will use to render our tiles:

  • Mapnik: It’s our base tool that does the actual rendering. We use it as a library to call from golang to render tiles.
  • Postgis + Postgresql: We store the maps in Postgresql DB using the Postgis plugin. Mapnik  requests data from that DB.
  • Gopnik:This is a frontend server that calls libmapnik to render tiles. This project is using a master-slave pattern and allows you to create cluster of renderers. Also, it’s written with Golang 😉  – I couldn’t miss that opportunity to hack with it a bit!
  • Map Style: A fork of the original OSM style project that I optimised for hi-res displays by supporting 2x size tiles and using different font sizes.
Best of all,  everything is packaged up inside Docker so we’ll be able to reuse it in any environment.

Building It

NOTE: For this to work on your machine, you should have ports 8888,8080,8090 open and free on your host machine.

The project is hosted on GitHub and contains a Docker Compose environment that will run the Database + Gopnik cluster + Web example. First of all let’s clone the repo:
$ git clone https://github.com/dooman87/melbourne-map.git
 We also need some static files to download because they will be required to render some basic stuff (e.g. continent shapes) on low zoom. The files are quite big (total is about 1 Gb) which is why we don’t download them as part of the Docker image-building step. To download these static files, go to your repository clone and run:
$ ./download.sh

Now let’s have a look at docker-compose.yml:

version: '2'
services:
postgis:
build: ./postgis
volumes:
- ./data:/var/lib/postgresql/data
environment:
- POSTGRES_DB=gis
renderer:
build: ./renderer
volumes:
- ./renderer/map_data:/map_data
- ./renderer/scripts:/scripts
volumes_from:
- postgis:ro
ports:
- "8080:8080"
- "9090:9090"
depends_on:
- postgis
web:
image: nginx:1.11-alpine
volumes:
- ./web:/usr/share/nginx/html
ports:
- "8888:80"
view raw docker-compose.yml hosted with ❤ by GitHub

You can see that we have 3 services. Let’s have a look at each a bit closer:

1. Postgis Service

This service runs a database that contains imported data for Melbourne city. This data has been extracted from OpenStreetMap by the guys at the mapzen project, and is also open source (you can find extracts for all big cities there). The Docker image itself is based on a postgis image that was created by mdillon. I added the osm2pgsql package because we depend on it to import map data into the database. The entire Dockerfile is quite small and simple:

FROM mdillon/postgis:9.5
WORKDIR /
RUN apt-get update && \
apt-get install -y osm2pgsql git wget && \
rm -rf /var/lib/apt/lists/*
RUN git clone https://github.com/dooman87/openstreetmap-carto.git && \
wget https://s3.amazonaws.com/metro-extracts.mapzen.com/melbourne_australia.osm.pbf
#Overriding init script to add hstore extension that osm2pgsql requires
COPY ./initdb-postgis.sh /docker-entrypoint-initdb.d/postgis.sh
view raw Dockerfile hosted with ❤ by GitHub

We clone the openstreetmap-carto project because it defines what objects we will render, as osm2pgsql will need to know what objects should be imported into the DB. The project contains a file called openstreetmap-carto.style that defines objects and has simple structure of things that will be in DB. For example, here’s a fragment of the file:

# OsmType Tag DataType Flags
node,way access text linear
node,way addr:housename text linear
node,way addr:housenumber text linear
node,way addr:interpolation text linear
node,way admin_level text linear
node,way aerialway text linear
node,way aeroway text polygon
node,way amenity text polygon
node capital text linear
...
...
view raw openstreetmap-carto.style hosted with ❤ by GitHub

So, for example if we don’t want to have labels on capitals, then we can simply remove the line with that type from the file.

Also, I had to customise the Docker entrypoint script to import map data on first start of the container, and create a flag-file to let the renderer know that the database has been successfully imported (last two lines):

#!/bin/sh
set -e
# Perform all actions as $POSTGRES_USER
export PGUSER="$POSTGRES_USER"
# Create the 'template_postgis' template db
psql --dbname="$POSTGRES_DB" <<- 'EOSQL'
CREATE DATABASE template_postgis;
UPDATE pg_database SET datistemplate = TRUE WHERE datname = 'template_postgis';
EOSQL
# Load PostGIS into both template_database and $POSTGRES_DB
for DB in template_postgis "$POSTGRES_DB"; do
echo "Loading PostGIS extensions into $DB"
psql --dbname="$DB" <<-'EOSQL'
CREATE EXTENSION postgis;
CREATE EXTENSION postgis_topology;
CREATE EXTENSION fuzzystrmatch;
CREATE EXTENSION postgis_tiger_geocoder;
CREATE EXTENSION hstore;
EOSQL
done
#import Melbourne city
osm2pgsql --style /openstreetmap-carto/openstreetmap-carto.style -d gis -U postgres -k --slim /melbourne_australia.osm.pbf
touch /var/lib/postgresql/data/DB_INITED
view raw initdb-postgis.sh hosted with ❤ by GitHub

2. Renderer Service

This is the renderer that generates tiles using Gopnik. The image below is showing how exactly rendering process is working. “Dispatcher” and “Render” are Gopnik’s components. We are using normal filesystem as a cache. Mapnik is used as a library that “Render” component calls:

render
Image was created by gopnik team. Original address: http://sputnik-maps.github.io/gopnik/img/render.png
The Docker image for the renderer is based on an original gopnik image that I created and contributed back to the project. Here’s the Docker file:
FROM dpokidov/gopnik
RUN apt-get update && \
apt-get install -y git curl unzip node-carto mapnik-utils fontconfig && \
rm -rf /var/lib/apt/lists/*
WORKDIR /
RUN wget https://github.com/adobe-fonts/source-sans-pro/archive/2.020R-ro/1.075R-it.tar.gz && \
tar -xzvf 1.075R-it.tar.gz && \
mkdir /usr/share/fonts/truetype/sourcepro-ttf/ && \
cp source-sans-pro-2.020R-ro-1.075R-it/TTF/*.ttf /usr/share/fonts/truetype/sourcepro-ttf/ && \
fc-cache && \
rm 1.075R-it.tar.gz && \
rm -rf source-sans-pro-2.020R-ro-1.075R-it
RUN git clone https://github.com/dooman87/openstreetmap-carto.git
WORKDIR /openstreetmap-carto
ADD shapes/ data/
RUN shapeindex --shape_files \
data/simplified-water-polygons-complete-3857/simplified_water_polygons.shp \
data/water-polygons-split-3857/water_polygons.shp \
data/antarctica-icesheet-polygons-3857/icesheet_polygons.shp \
data/antarctica-icesheet-outlines-3857/icesheet_outlines.shp \
data/ne_110m_admin_0_boundary_lines_land/ne_110m_admin_0_boundary_lines_land.shp
RUN mkdir /map_data
VOLUME /map_data
RUN mkdir /scripts
COPY scripts/compile_style.sh /scripts
COPY scripts/run_render.sh /scripts
COPY entrypoint.sh /
RUN chmod 755 /scripts/*.sh && \
chmod 755 /entrypoint.sh
WORKDIR /
ENTRYPOINT /entrypoint.sh
view raw Dockerfile hosted with ❤ by GitHub
Some key points about this file:
  • It installs the Adobe Source Pro fonts that get used by map style
  • It clones the map style repo
  • It creates indexes from shape files that were downloaded in the first step
  • It defines a volume called ‘map_data’. That volume contains a ‘config.json’ file that we pass to gopnik. Most importantly, config.json defines the scale factor (line 72) that we want to use to generate tiles:
"Cmd": ["/gopnik/bin/gopnikslave", 
"-stylesheet", "/openstreetmap-carto/stylesheet.xml", 
"-pluginsPath", "/usr/lib/mapnik/2.2/input", 
"-fontsPath", "/usr/share/fonts", 
"-scaleFactor", "2"]
  • It copies scripts to compile the style and run the Gopnik dispatcher and render slaves
The entrypoint script of the Docker container for the renderer service compiles the map style and waits until the DB becomes accessible. We kick-off the renderer process once the DB is ready:
#!/usr/bin/env bash
sh /scripts/compile_style.sh
while [ ! -e /var/lib/postgresql/data/DB_INITED ]
do
sleep 5
echo "Waiting while database is initializing..."
done
#Have to wait because once DB created then osm2pgsql restarting postgres.
#TODO: Using pg_isready
echo "DB successfully created, waiting for restart"
sleep 60
echo "Starting renderer"
sh /scripts/run_render.sh
view raw entrypoint.sh hosted with ❤ by GitHub

3. Web Service

This is a docker container that uses a ready-made nginx image to serve up a simple example web page. It uses the leaflet library to show a map and position at Melbourne:

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8"/>
<link rel="stylesheet" href="http://cdn.leafletjs.com/leaflet-0.7.1/leaflet.css" />
<script type="text/javascript" src="http://cdn.leafletjs.com/leaflet-0.7.1/leaflet.js"></script>
<style type="text/css">
html, body, #map {
width: 100%;
height: 100%;
margin: 0 !important;
overflow: hidden;
}
</style>
</head>
<body>
<div id="map"></div>
</body>
<script type="text/javascript">
var map = L.map('map').setView([-37.8130, 144.9484], 14);
L.tileLayer('http://localhost:8080/{z}/{x}/{y}.png', {
attribution: 'Map data &copy; <a href="http://openstreetmap.org">OpenStreetMap</a&gt; contributors, <a href="http://creativecommons.org/licenses/by-sa/2.0/">CC-BY-SA</a&gt;, Imagery © <a href="http://cloudmade.com">CloudMade</a>&#39;,
maxZoom: 18
}).addTo(map);
</script>
</html>
view raw index.html hosted with ❤ by GitHub

Starting It

To run things, you need to run the following command in your repository clone:

$ docker-compose up
That’s really it. The first time it will take a while to build images and start up (up to 10 mins) because we need to import the database. You can find out when all containers are started and ready to go by looking for log messages that look like this:
postgis_1   | LOG:  autovacuum launcher started
renderer_1  | Starting renderer
renderer_1  | 2016/12/27 23:34:30 app.go:266: [INFO] Serving debug data (/debug/vars) on %s... :9080
renderer_1  | 2016/12/27 23:34:30 app.go:267: [INFO] Serving monitoring xml data on %s... :9080
renderer_1  | 2016/12/27 23:34:30 renderselector.go:209: [DEBUG] ping error %v dial tcp 127.0.0.1:8090: getsockopt: connection refused
renderer_1  | 2016/12/27 23:34:30 renderselector.go:117: [DEBUG] '%v' is %v localhost:8090 Offline
renderer_1  | 2016/12/27 23:34:30 main.go:118: [INFO] Starting on %s... :8080
renderer_1  | 2016/12/27 23:34:30 app.go:266: [INFO] Serving debug data (/debug/vars) on %s... :9090
renderer_1  | 2016/12/27 23:34:30 app.go:267: [INFO] Serving monitoring xml data on %s... :9090
renderer_1  | 2016/12/27 23:34:35 main.go:95: [INFO] Done in %v seconds 4.84147165
renderer_1  | 2016/12/27 23:34:35 main.go:103: [INFO] Starting on %s... :8090

Testing it

Now you can open your favourite browser and go to:
http://localhost:8888
The initial rendering may take 10-30 seconds, after which you will see a fabulous map of Melbourne city.

Shipping It

How you ship this solution is really up to you. I’m using this in a project that has about 300 hits per day (what a big boy!). I’m running all the containers on one EC2 instance. Also, if you cache all tiles that are already rendered, then you won’t need huge CPU resources and – up to a point – may even have some resilience if the Gopnik renderers go down. You can even get the Gopnik renderers to prerender tiles if you want.

That said, strictly speaking this  is not a production-ready solution. From gopnic’s docs:

Code is unstable. Current status of Gopnik is something like developer preview. Everything may change. Everything may be broken.

Regardless, it’s probably worth mentioning that uptime of my EC2 instance is currently 5412 hours (was launched on May 18, 2016) and it’s still serving tiles.

Where is NEW YORK, PARIS, MOSCOW, MILAN?

It’s quite easy to render other cities. Just update the Postgis Dockerfile to download a different pbf file and update its name in the entrypoint script.

Here’s the line that you would change in the Dockerfile:
wget https://s3.amazonaws.com/metro-extracts.mapzen.com/melbourne_australia.osm.pbf
and here’s the line you would change in initdb-postgis.sh:
osm2pgsql --style /openstreetmap-carto/openstreetmap-carto.style -d gis 
-U postgres -k --slim /melbourne_australia.osm.pbf
After you’ve changed these, you then clean and rebuild everything:
$ ./clean.sh
$ docker-compose build
$ docker-compose up

What’s Next?

So that’s my solution for serving up Retina-quality map tiles, packaged up with Docker for easy distribution.

In the ideal world it would be nice to use vector format like svg to render tiles. That way you wouldn’t have to worry about serving up particular resolutions. There are some things going on (http://wiki.openstreetmap.org/wiki/Vector_tiles) in that space, along with commercial solutions.

There is one project that I’d like to mention is osm2vectortiles. You can generate vector tiles from data extracts using it. So, the next step for my project would be to evaluate current solution to generate vector tiles instead of raster.

Tags:
, , ,
4 Comments

Leave a Reply

%d bloggers like this: