Marble/OSMNaturalEarth

Natural Earth Dataset derived Zoom Levels

Introduction

While we use the actual data provided by OpenStreetMap for all higher levels (starting from level 9/11) we want to use the Natural Earth Dataset for all the lower zoom levels. The style is supposed to mimic the default styling of the OpenStreetMap dataset as close as possible (to guarantee a seemless user experience).

Data Import

Sources

Natural Earth provides data in three levels of detail.

Large Scale Data (1:10m)

Cultural [1] Physical [2]

Medium Scale Data (1:10m)

Cultural [3] Physical [4]

Small Scale Data (1:100m)

Cultural [5] Physical [6]

Conversion from SHP to OSM

The vector data provided by Natural Earth is in SHP format. In order to create vector tiles, this data needs to be converted into OSM format. Two mechanisms were being considered for converting files from SHP to OSM. The first mechanism involves usage of polyshp2osm[7] tool. The second mechanism involves building a custom converter on the lines of the existing shp2pn2 tool of marble.

polyshp2osm is built upon the GDAL/OGR python bindings. GDAL/OGR is a library for reading/writing raster and vector geospatial formats. It presents a single abstract data model, for all the formats, to the calling application. polyshp2osm reads the SHP file into this abstract data model and then loops over all the possible geometries to figure out the nodes, ways and relations of the corresponding OSM file. In this regard it is quite similar to the shp2pn2 tool which reads the SHP file into a GeoDataDocument. However, polyshp2osm is limited in the sense that it provides support only for polygons and not for linestrings and other geometries. In fact, it throws an error if we try to convert SHP files containing linestrings like ne_10m_roads and ne_10m_rivers_lake_centerlines. Hence in order to make the tool work, it had to extended so that it is able to process geometries other than polygons like linestrings, points etc. This makeshift extension although works well in most cases , lags in a few areas. Thus polyshp2osm lags in geometry conversion and effort must be put in order to perform this conversion properly.

The other area of polyshp2osm is its metadata extraction and tag-mapping. The GDAL/OGR library, on which the converter is built provides direct methods for extracting the metadata for a particular geometry present in SHP file. This metadata is available in the form of key-value pairs. These key-value pairs obtained from the SHP files can be directly mapped to corresponding OSM key-value pairs. However all such mapping from the Natural Earth metadata to the OSM key-value pair has to be put into the polyshp2osm program manually since the polyshp2osm was built keeping in mind the MassGIS OpenSpace dataset and has mappings corresponding to this dataset only. Though the tool(polyshp2osm) provides a framework for tag-mapping, still effort had to be put in analyzing the metadata provided by different datasets of Natural Earth and finding the corresponding OSM key-value mappings.

In spite of its shortcomings, polyshp2osm was chosen over building a new converter on the line of shp2pn2 since building a new converter will involve adding support for creating nodes, ways and relations on the existing shp2pn2 procedures which will take more time as compared to adding support for 1-2 more geometries in polyshp2osm. Also we will have to build a metadata extraction and tag-mapping feature for the new converter which are already present in the polyshp2osm tool.

Status of dataset

Admin 0 – Countries

Features present:

• Admin 0 – Countries
  • Geometry : Polygon
  • Standard OSM Tag : No
  • Custom OSM tag used : Yes
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

Admin 0 – Details

Features present:

• Admin-0 sovereignty
  • Geometry : Polygon
  • Standard OSM Tag : No
  • Custom OSM tag used : Yes
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

• Admin-0 map
  • Geometry : Polygon
  • Standard OSM Tag : No
  • Custom OSM tag used : Yes
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

• Adm-0 scale ranks
  • Geometry : Polygon
  • Standrad OSM Tag : No
  • Custom OSM tag introduced : Yes
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

• Admin-0 Tiny Countries
  • Geometry : Point
  • Standard OSM Tag : No
  • Custom OSM tag used : No
  • Rendered : No
  • Feature mapped to OSM tag: No

• Admin-0 Tiny Countries
  • Geometry : Point
  • Standard OSM Tag : No
  • Custom OSM tag used : No
  • Rendered : No
  • Feature mapped to OSM tag: No

• Admin-0 Tiny GeoUnit
  • Geometry : Point
  • Standard OSM Tag : No
  • Custom OSM tag used : No
  • Rendered : No
  • Feature mapped to OSM tag: No

• Admin-0 Tiny GeoSubunit
  • Geometry : Point
  • Standard OSM Tag : No
  • Custom OSM tag used : No
  • Rendered : No
  • Feature mapped to OSM tag: No

• Admin-0 Tiny Countries Pacific
  • Geometry : Point
  • Standard OSM Tag : No
  • Custom OSM tag used : No
  • Rendered : No
  • Feature mapped to OSM tag: No

Admin 0 – Boundary Lines

Features present:

• Indefinite (please verify)
  • Geometry : Linestring
  • Standard OSM Tag : Yes
  • Custom OSM tag used : No
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

• Line of control (please verify)
  • Geometry : Linestring
  • Standard OSM Tag : Yes
  • Custom OSM tag used : No
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

• International boundary (verify)
  • Geometry : Linestring
  • Standard OSM Tag : Yes
  • Custom OSM tag used : No
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

• Disputed (please verify)
  • Geometry : Linestring
  • Standard OSM Tag : Yes
  • Custom OSM tag used : No
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

• Pacific Groupings
  • Geometry : Linestring
  • Standard OSM Tag : No
  • Custom OSM tag used : No
  • Rendered : No
  • Feature mapped to OSM tag: No

• Map unit boundary
  • Geometry : Linestring
  • Standard OSM Tag : No
  • Custom OSM tag used : No
  • Rendered : No
  • Feature mapped to OSM tag: No

• Marine Indicator Treaty
  • Geometry : Linestring
  • Standard OSM Tag : No
  • Custom OSM tag used : No
  • Rendered : No
  • Feature mapped to OSM tag: No

• Marine Indicator Median
  • Geometry : Linestring
  • Standard OSM Tag : No
  • Custom OSM tag used : No
  • Rendered : No
  • Feature mapped to OSM tag: No

• Boundary Treaty
  • Geometry : Linestring
  • Standard OSM Tag : No
  • Custom OSM tag used : No
  • Rendered : No
  • Feature mapped to OSM tag: No

• Marine Indicator 200 mi nl
  • Geometry : Linestring
  • Standard OSM Tag : No
  • Custom OSM tag used : No
  • Rendered : No
  • Feature mapped to OSM tag: No

• Marine Indicator Disputed
  • Geometry : Linestring
  • Standard OSM Tag : No
  • Custom OSM tag used : No
  • Rendered : No
  • Feature mapped to OSM tag: No

Admin 1 – States, Provinces

Features present:

• Admin-1 scale rank
  • Geometry : Polygon
  • Standard OSM Tag : No
  • Custom OSM tag used : Yes
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

• Admin-1 boundary
  • Geometry : Linestring
  • Standard OSM Tag : Yes
  • Custom OSM tag used : No
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

Populated Places

Features present:

• Historic place
  • Geometry : Point
  • Standard OSM Tag : Yes
  • Custom OSM tag used : No
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

• Populated place
  • Geometry : Point
  • Standard OSM Tag : Yes
  • Custom OSM tag used : No
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

• Scientific station
  • Geometry : Point
  • Standard OSM Tag : Yes
  • Custom OSM tag used : No
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

• Admin-0 capital
  • Geometry : Point
  • Standard OSM Tag : Yes
  • Custom OSM tag used : No
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

• Admin-1 capital
  • Geometry : Point
  • Standard OSM Tag : Yes
  • Custom OSM tag used : No
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

• Admin-0 region capital
  • Geometry : Point
  • Standard OSM Tag : Yes
  • Custom OSM tag used : No
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

• Admin-1 region capital
  • Geometry : Point
  • Standard OSM Tag : Yes
  • Custom OSM tag used : No
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

• Admin-0 capital alt
  • Geometry : Point
  • Standard OSM Tag : Yes
  • Custom OSM tag used : No
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

• Meteorological Station
  • Geometry : Point
  • Standard OSM Tag : Yes
  • Custom OSM tag used : No
  • Rendered : Yes
  • Feature mapped to OSM tag: Yes

TBD

Combining datasets

TBD

Mapping different resolutions (1:10, 1:50, 1:110) to zoom levels

TBD