World Back Then

Posted on May 7, 2020

Website

Github repo with the data pipeline to extract and clean the data

Github repo with the webapp

 

Like many people, when Covid hit I had some free time on my hands and decided to work on a project that combined two passions of mine: coding and learning about History.

World Back Then is an interactive world history map that lets you explore different times in History as if you were on Google maps. The contextual data when you click anywhere has been extracted semi-automatically from Wikipedia through a custom data pipeline and the state boundaries to build the maps are from an open-source GIS dataset.

Technical details:

  • the data pipeline was implemented in Scala and Spark running on a Wikipedia dump to extract and clean relevant information about states/countries, peoples and historical inventions
  • the frontend was written with the Play! framework and Scala.js and deployed as S3 static website served through a Cloudfront distribution
  • the backend was deployed on AWS with Lamdba functions and the API Gateway, with data stored on an RDS MySQL instance
  • the map visualization was built on top of MapTiler’s free low resolution satellite tiles plus open-source GIS data for the shapefiles representing country borders (which were developed on a best-effort basis, I actually had to fix some evident mistakes here and there)