​​DC Data Engineering Meetup: Open Geodata Workshop

​
​bit.ly/dc-data-3​

  • ​​60 minute hands on workshop to extract raw spatial data from OSM, filter, transform and visualize it on the browser
​​

​
​OpenStreetMap www.osm.org​

  • ​​Globally crowdsourced street map of the world
  • ​​5+ billion points
  • ​​Data license: ODbl (CC-by-sa like)
​​
​​
​​Change explorer: https://osmcha.mapbox.com
​​
​​

​
​OpenStreetMap Data ​📦​ 

​​
  • ​​All objects are versioned, full history: 
​​
​​Formats: protobuf, XML
​​
  • ​​Geographic extracts
  • ​​Filtered data may not have all the OSM objects
​​
​​Formats: GeoJSON, Shapefiles
​​
​
​Building an upto date OSM data mirror
  1. ​​Get latest weekly planet
  1. ​​Apply latest day, hour, minute diffs since the last weekly dump
​​
​​
​​

​
​OSM data primitives

​
​Geometry:Node

​​

​
​Geometry:Way

​​
​​Unlike traditional GIS data both lines and areas are treated as way geometries in OSM.
​​
​​Lines:
​​
​​Areas: An enclosed line can be used to define areas. Depending on the exact tags, it might also be interpreted as a linear feature, eg. a wall
​​

​
​Geometry:Relation

​​Ordered collection of nodes and ways
​​

​
​Metadata:Tags

​​Key value pairs describe the properties of the physical object that is being mapped. These are freeform text and an object can have any number of tags to qualify it.
​​
​​

​
​:Example​ 

​
​OSM feature layer: Places

​​
​​
​​
​​

​
​Data editing

  1. ​​Create geometry
  1. ​​Add metadata
​​

​
​Popular editors:

​​iD - Simple web editor: Edit button on https://www.osm.org
​​JOSM - Advanced desktop editor:  https://josm.openstreetmap.de/
​​
​​

​
​Query some data using tags

​​Traffic signals in DC: http://overpass-turbo.eu/s/P6k
​​
​​Overpass API is useful for extracting a subset of data using tags.
​​
​​
​​

​
​​⚙️​ Lets process some map data ​⚙️​ 

​​
​​

​
​1. Get the data

​
​OSM API

​​Download .osm xml data directly from the API. Requests are limited by geographic area, usually the size of a city.
  1. ​​Hit export to download the OSM XML file
​​
​​

​
​2. Filter 

​
​OSMOSIS

​​
  • ​​Read OSM XML and PBF files
  • ​​Filters and extracts data
  • ​​Work with replication
​​
​
​Filter an osm file using tags
​​osmosis --read-xml map.osm \
​​  --tf accept-ways 'highway=*' \
​​  --write-xml highway.osm
​​

​
​Other tools:

​​Convert between OSM and intermediate processing format
​​

​
​OSMIUM

​​Similiar to OSMOSIS with slightly different features
​​

​
​3. Transform

​​

​
​OGR2OGR 

​​Convert and transform between geospatial formats
​​
​
​Use the ogrinfo tool to inspect the data
​​ogrinfo map.osm
​​
​​ogrinfo map.osm lines
​​
​​
​
​Extract the lines from the filtered osm file as a geojson
​​ogr2ogr -f GeoJSON highway-lines.geojson highway.osm \
​​    -dialect sqlite -sql "SELECT * from lines"
​​

​
​QGIS Desktop GIS

​​Full featured open source desktop GIS application
​​
​​

​
​4. Load and visualize

​​

​
​Kepler.gl Online spatial data visualization engine

​​

​
​Mapbox Studio Online map data hosting and map designer

​​

​
​PostGIS spatial postgres database

​​
​​

​
​5. Build spatial applications in the browser

​
​turf.js

​​Geospatial analysis JS libraries for the browser
​​WebGL renderer for interactive maps in the browser

​
​Examples apps

​​
  • ​​Text: CC-0
  • ​​arun@mapbox.com
​​
​​
​​
​​