Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Unlike traditional apps where all the data is stored on a single server (in the cloud or in an office), the database is integrated into the app on the device, and every device has a copy of all the data. We made this choice so that Mapeo can work entirely offline and does not need the user to set up any database or machine for storing the data. If any device is lost, synchronizing a new device with any other device in the project will restore the data. This peer-to-peer database also means that users can transfer data to a computer in the field without internet, and several users with computers can all work on the same dataset and share edits.
The way to synchronize data between two Mapeo devices is currently to connect them to a local wifi network. They will discover each other automatically (if they are part of the same project) and the user can initiate a sync between them. The wifi network does not need to be connected to the internet - we currently use cheap ($25) travel routers to create a wifi network in the field. In the future we plan to do this over bluetooth. Currently every user ends up with a copy of all data from all users, but on mobile they can only edit or delete their own observations- others' observations are read-only. In the future we plan to give more control over this (one option is to continue to synchronize all data everywhere, but to encrypt each users data for specific admin users, so other users cannot read it even though they have a copy).
The main data type in Mapeo is an "Observation". An observation has lat and lon coordinates; GPS metadata; one or more attachments (currently only photos, but audio and video in the future); and any number of key-value tags with attributes about the observation. The full schema is here.
When the user creates an observation with Mapeo they are asked to fill in a description and, optionally, additional questions. The description is always written as a "note" tag. Which additional questions are shown and how the answers translate to key-value tags is defined by "presets". Presets define how to show questions to the user and the tag key that the answer should be stored under. The schema of presets and fields is here (presets) and here (fields). This is very close to the presets model used by the OpenStreetMap editor, iD.
The presets define "Categories", which can have an icon. When a user creates an Observation they select a category by pressing an icon, and the category they choose defines which additional questions are shown. We chose this approach because it gives us a very flexible data model - Observations can have any structure of attributes, and how they are presented to the user can change over time and is defined by the presets and fields. In Mapeo we also have a "Place" data type which can be a point, line or area. In the future we will support multiple "Observations" to be linked to a single "Place", but for now there is a one-to-one relationship between an observation and a place.
Internally, all the data is stored as an immutable ledger for each device that is cryptographically signed to prove it has not been tampered with. The main "database" is a plain text file for each device of JSON documents for each creation, edit and deletion of observations, and an accompanying file of cyrptographic signatures. When devices sync they share the latest ledgers (every device has a ledger, and it is only appended to, it never changes). After sync we create an index of all the data from all the devices, and if there are any conflicts then we choose to display the most recent edit, but the other versions are not deleted.
The user interface and front-end logic of Mapeo Mobile is written in React Native. This means that all the code is in JavaScript, but the UI is displayed using native code by the React Native framework. This means that it runs well on slower devices.
We chose React Native because it means we can share code between desktop and mobile and it is cross-platform. Currently we only support Android, but we have a working iOS prototype with minimal changes to the code.
The database is also written in JavaScript. We run the database in a separate process in the app in Node.js compiled for Android. The database and the front-end speak to each other via a local http (web) **server and RPC channels.
We use mapbox-gl for displaying maps. It supports both raster (e.g. satellite) and vector tiles. Vector tiles are helpful because they take up less space online. We use Mapbox Studio for designing maps.
We chose this approach because it gives us a lot of flexibility about styling maps, we can download map tiles and show maps offline, and the vector maps provide a really good user experience for zooming and moving the maps, even on slower devices.
Download the latest version on Google Play.
You can also download as an APK and load manually on the phone. Please note that to install Android application APK from outside of Google Play Store you need to enable installing unknown apps.
If you are in place with no Internet connection, you can share the previously downloaded Mapeo APK from your mobile device with others via Bluetooth using third party tools such as Bluetooth APK sender or using your Mobile phone WiFi with Easy Share : WiFi File Transfer
Find the downloaded Mapeo Installer (e.g. Installar_Mapeo_Windows.exe
) and double click
Wait for installer to finish, which will open the Mapeo on completion
You may need to click Allow Access if you have Windows Defender enabled
Find the downloaded Mapeo Installer and double click
Drag the Mapeo icon into your Applications folder
Find Mapeo in your Applications folder and right click to open
Click Open—this verification only happens the first time you use Mapeo
Pick the release that works for you. We recommend using the .deb
package for most systems.
Double-click the .deb
installer.
Mapeo is an open source, local-first map editor. Mapeo makes it easy for teams to create maps and organize stories and knowledge.
Mapeo is an open source, local-first map editor. Mapeo makes it easy for individuals or teams to create maps and organize stories and knowledge.
Mapeo leverages the same easy-to-use editor and flexible data structure as OpenStreetMap. It is powered by a peer-to-peer database that enables offline collaboration and control over data sharing.
We are building Mapeo with indigenous communities in the Amazon, who asked for an easier way to create and edit their own maps so as to defend their lands and cultures.
We are building Mapeo in close collaboration with our partners Amazon Frontlines and Alianza Ceibo in the Ecuadorian Amazon, working specifically with the Waorani, Cofan and Siekopai peoples. Technical partners include Development Seed, the Dat Project and more.
The development of Mapeo has been made possible due to generous grants from the Knight Foundation, Abundance Foundation and Leonardo DiCaprio Foundation.
Mapeo is built with JavaScript programming language. To get started, you'll need to install Node.js development environment.. If you already have Node.js installed you can skip this section.
You need to be at least on Node.js version 8 (or higher) for the mapeo-settings-builder to work properly. Head over to Node.js download page and select installer for your operating system.
Alternatively you can also use NVM (Node Version Manager) to install and manage multiple versions of Node.js on your computer.
Then close terminal and open again
https://docs.npmjs.com/cli/install
You'll see output on the terminal, but this is OK
If your computer is ready to create configurations, type
You should see output that looks something like
Now you're ready to move to putting your icons and questions together!
Create a custom .mapeosettings
package to install in devices using MAPEO Desktop and Mobile.
Every project has different needs. And users involved in a project can make use of a security feature, project keys, to keep data collected with intended users.
In anticipation of using your customized configurations of MAPEO, with enough time to test and edit before the official activity. Prep time depends on the author, the project and the process for approvals and revisions.
Anywhere online and offline*. (*installers required before building presets offline).
Project planner(s): They are, or are in direct dialogue with, decision makers and can consult people in the project area
Designers, artists or facilitators: Responsible for creating or facilitating the creation of icons appropriate to the project.
Configurations author: Usually a programmer who can run scripts and debug errors. A novice coder can also do this by following steps closely and reaching out for help when needed.
Lone wolves: You do it all yourself. But remember, you are not alone.
Provide feedback about instructions here.
Time it takes: (30 min - 30 days. Estimate: 30 min per category)
Each of the presets you create needs to be assigned an icon, and these all need to be saved as .svg
files. You can assign the same icon to various presets, or they can each have their own individual one.
See example icons created for our default 'jungle' configuration.
Design icons as 100x100 pixel graphics that are clear when viewing at 100%.
From Illustrator, File > Export
Select SVG
With the artboard box ticked
Additional settings as below. These are how Mapbox says to export icons for svgs to use on their maps too.
If you have trouble viewing your icons, check this troubleshooting guide created by Mapbox or ask us for support.
The icons need to read by MAPEO in two sizes: 100 and 24 pixels. For that reason there is a specific file naming convention:
name**-100px**.svg
name**-24px**.svg
Each icon can be duplicated and renamed so that there is one of each. They are opened and read by the .json files in the "preset" folder. Verify that "name" is correctly entered where needed (more on this in the next section). The build script will process the pixel size suffix.
Create a directory named icons
and put all icons in there according to the above naming convention.
The directory should look like this:
We're here to help!
Slack: https://slack.digital-democracy.org
IRC #ddem on freenode https://webchat.freenode.net/
Support is available 9:00-0:00 GMT, which covers 9:00-17:00 GMT and 9am-5pm during US West-Pacific Time. Unfortunately, we are unlikely to respond between midnight and 9:00 GMT.
Sorry about that! We'll get to you as soon as possible. In the meantime, please open an issue on GitHub for Mapeo Desktop or Mapeo Mobile
Great! we're always interested in hearing from you and making MAPEO better
Open an issue on GitHub for Mapeo Desktop or Mapeo Mobile
Time it takes: (10 min)
It's possible to export a data file or a piece of data from Mapeo and upload them to a map on the Internet. The map can be shared and used in campaigns or for other more interactive formats.
Inside Mapeo Desktop, filter data until you have the data that you want to appear in the map on the Internet.
Choose "Export Webmap" from the menu in the top right corner of the window.
Write a title and description of the map, these will appear on the map online.
Name and save the file.
Enter your account details or create a new account if it's your first time.
The window shows a list of maps that you've already uploaded to the Internet in your account.
It's possible to edit the map data, add new maps, delete maps, and share public links to the maps to other people.
To add a new map, make a click on ADD MAP
Choose a file that was exported from Mapeo.
The map will be uploaded with your data and media.
View the map in a website.
Click the symbol to the side of the map which says Public Link.
The map will open in a new window.
To share with others, copy the web address (URL) of the map and share with your networks or website.
You can zoom in and out of the map.
Click on a photo to the left to open details.
To view larger, click the photo in the map.
Time it takes: (20 min - 20 days. Estimate: 20 min per category)
Download the example configuration.
Unzip the contents to a new folder using a program like 7zip.
Rename the folder from "mapeo-default-settings-2.1.0" to "mapeo-config-projectname".
These are the directories that live in the top level folder
fields
icons
presets
defaults.json
metadata.json
package.json
style.css
fields
directory customize the .json
filesIn the fields directory, each .json file needs a key
, type
, label
, and placeholder.
type
can be one of select_multiple
, select_one
, text
, or textarea
presets
directory, customize the .json
filesIn the presets directory, each .json file needs:
a list of fields
which should match the key
created in the fields directory.
icon
must mach the name of an icon in the icons
folder
name
will be the human-readable label shown to the user
color
determines the color of observation dots on the map. (Dots fall back to orange if no color is defined.) Value can be a hex code, CSS color name or any string supported by validate-color.
geometery
must be a list of point
,area
, and/or line
The icons
folder should have all of the icons you created from the previous section
Type, 'cd`, then a space, then drag and drop the folder where the prepared assets are and press enter. It will look something like this
You will then be ready to run scripts directly in the folder.
This -s tells npm to be silent, so that you only see errors that are meaningful to you.
You will see something like the following output. Errors will be highlighted in RED with hopefully some helpful description so that you can remedy the issue.
You'll also see a .mapeosettings
file inside of the build
directory.
A .mapeosettings
file is a tar file, similar to a zip file. You can see the contents of the file by changing the file extension to .tar
and using any application that can extract tar files (such as 7zip, mentioned above).
Type the following into the terminal
You need to be at least on Node version 8 for the mapeo-settings-builder to work properly. If you need help, review the 'Preparing Computer' section and ensure you're on the latest version of mapeo-settings-builder.
You also may want to delete node_modules and install updated versions of the dependencies.
In Mac or Linux, in the terminal:
If you're having more issues, please open an issue on the GitHub repository or e-mail our support hotline.
An overview of the key components in the Mapeo core infrastructure.
Behind the scenes of the user interfaces for working with mapping and monitoring data, you'll find Mapeo's core infrastructure. This document is a work in progress and if you find anything missing or unclear, please feel free to submit a pull request!
Schema
Observation. Mapeo extends the OpenStreetMap specification to add 'Observations' -- a new data type that is like a node or point, but with extra metadata and media.
Field. Called a 'question' (ODK) or 'tag' (OSM), a Field is how we structure metadata about Observations and OSM Elements.
Preset. Called a 'Category' in the Mapeo interface, but in code we call this a 'preset.' This type defines a set of Fields and an Icon.
Filter. An expression that can be applied on the values in one or more Field across a set of observations.
Database
KappaDB is one of the fundamental primitives of Mapeo Core that handles indexing of data that isn't media (i.e., it does not handle images). KappaDB is a flexible database for peer-to-peer applications that we developed in 2018, and is considered stable. If you haven't yet, go ahead and try the workshop to familiarize yourself with how it works.
Configuration
Each 'project' has a set of configuration files. It is identified by a project key which is a shared secret amongst all devices. It authorizes devices to read and write to the database. There is no way at this time to revoke a configuration from a device remotely. Read more about configurations in the Preparing Mapeo Configurations page.
Discovery
Each device announces itself on the local network using MDNS. It announces itself using a hash of the project key, which is also called a discovery key. This means that 3rd-party observers will not be able to discern the project key.
Replication
Replication requires advance knowledge of the project key. Replication will be refused if a peer's project key is not the same.
Multifeed is the module that manages replication across multiple Mapeo devices. It depends upon hypercore, a distributed append-only log. Hypercore only allows one writer; multifeed syncronizes a set of hypercores -- by a variety of authors -- across all devices with the same project key. This forms the basis for creating a view of a single dataset that incorporates changes from multiple writers across multiple feeds.
Media
A project also contains media files which are handled separately from the multifeed module, which concerns itself primarily with structured data (e.g., json, protobuf, etc). Media files are replicated using blob-store-replication-stream. This is a simple module that announces the list of files available on both sides of a duplex stream, and then replicates the files that are missing.
Clients can provide filters for which files they want to accept on either end; by default, Mobile to Mobile sync does not send original images, only preview and thumbnail sizes. Original images are always syncronized from Mobile to Desktop.
Mapeo is built with long-term sustainability as a focus. This goes beyond financial sustainability -- we ensure that no single entity or technology is responsible for data storage or access.
Mapeo does not lock your data into a particular data format. You can easily move your data around, and it's kept in it's original form. Mapeo can export data to SMART, Shapefile, CSV, and GeoJSON. You'll never be locked into using Mapeo software.
If you are seeing a blank or beige screen, ensure that you are zoomed out. Use the Minus (-) sign on the top right hand corner of the map to zoom out as far as possible. When you've zoomed out to see the entire world, you'll be able to see your data.
To more quickly get access to your data, you can also use the Zoom to Data feature found in the application menu bar, under View->Zoom to data.
If you see an error message in Mapeo, please open a bug report on GitHub, or contact us in our support channels.
Make sure your Mapeo applications are on the latest stable versions, downloaded from the Mapeo website or Google Play.
Ensure that all devices are using the same configuration and project key. You can determine this by looking at the bottom of the syncronization screen in Desktop;
And the Observations->Settings->Project Configuration screen on mobile.
If you continue to have issues, please let us know and get support from our technical team.
Mapeo uses secure peer-to-peer technology with a distributed ledger, which does not include a public blockchain or consensus.
Public blockchains are designed for a scenario where public transactions must be mediated between participants which are all potentially malicious. These “trustless” transactions are the key assumption of a blockchain that distinguishes it from peer-to-peer technology.
Applications like Mapeo assume that data is managed by the community generating it -- and some of that data may never be publicly accessible. Mapeo's approach, in contrast, creates a closed group, where data creators are also data stewards, managing their own data and controlling who has access. We take security seriously and maintain protections from third-party attacks such as targeted hacks and surveillance by third-parties. For users of Mapeo, privacy is critical for protecting their ancestral knowledge and environmental monitoring information until it’s ready to be shared with trusted partners.
For more information, read this blog post and this paper from Article 19 about why blockchain is not a recommended tool for human rights and freedom of expression.
In the metadata.json file in your Mapeo Configuration, you can include a projectKey
which is a random cryptographic string of characters to prevent unwanted devices from getting access to the data.
If MM or MD has this inside the presets it has loaded, then it can only sync with another MM or MD that has the same project key.
You can edit the project key. For example, if want to make first 4 characters identifiable to a project, but can only contain letters a-f and numbers 0-9.
It can also only be 64 characters long -- no more, no less.
To create a projectKey, first open the Terminal.
Copy and paste the following command into the terminal
You'll see something like this (but with x replaced with real characters and numbers)
Copy this string and add it to the metadata.json
file so it looks like this:
Notice that there are double quotes "
around each value.
Time it takes: (10 min)
A .mapeosettings
file is a tar file, similar to a zip file. You can see the contents of the file by changing the file extension to .tar
and using any application that can extract tar files.
Before loading your preset configuration file into the phone, use mapeo-settings lint
to test your files to make sure they will compile on the phone.
Run the following command if you are using the default template.
Now, you are ready to load your settings to Mapeo Mobile.
Rename the .mapeosettings
file to .tar
Then expand .tar file by double clicking to open it
Copy that folder onto the phone
See https://github.com/digidem/mapeo-mobile#usage for latest published documentation
adb
Plug in the device to your computer using a USB cable.
Set the mode for USB debugging on the phone to 'Transfer Files'.
In the terminal, typing adb devices
should give you the following output:
Now, you'll need to place the presets folder in the appropriate directory on the Android device using adb push
Did it work? Test that the files are available on the device by using the shell.
If it worked correctly, you should see the following output
Mapeo Desktop supports importing .mapeosettings
files in the user interface.
After you've created your settings file, you can import it by navigating to File->Import Configuration....
You can also extract the .mapeosettings
file as a tar file into the following directory:
USERDATA is the per-user application data directory, which by default points to:
%APPDATA%
on Windows
$XDG_CONFIG_HOME
or ~/.config on Linux
~/Library/Application Support
on macOS
This folder (default
) expects these files directly in under this default folder (i.e. no sub-folder with a different name):
Here is an overview of the Mapeo ecosystem, including various modules and utilities
mapeo-desktop: Source code for Linux, MacOS, and Windows
mapeo-mobile: Source code for Mobile
mapeo-settings-builder: Build presets and sprites for the Mapeo apps
mapeo-migrate: Commandline tool for mapeo migrating databases from older formats to the latest.
mapeo-core: Library for creating custom geo data and synchronizing via a peer to peer network
mapeo-server: Mapping web server for managing observations, media, tiles, and various static files.
mapeo-settings: Manage settings files for Mapeo
iD-mapeo: Fork of OpenStreetMap editor, for offline use with Mapeo
mapeo-schema: Data schemas for mapeo data types
mapeo-styles: Default styles & tiles for mapeo backgrounds
mapeo-openmaptiles: Lightweight map tiles for mapeo-mobile for global offline map
osm-p2p: High-level p2p OpenStreetMap database for node and the browser
osm-p2p-server: Peer-to-peer OpenStreetMap API v0.6 Server for osm-p2p-db
mapeo-print-preview: Demo of web vector tiles for osm-p2p data.
mapeo-docs: Docs site for mapeo
Viewing map imagery while offline or in remote environments with limited connectivity.
Offline maps must be placed in this folder:
This folder should contain these files directly under this default
folder:
To Create an Asar file
Mapeo has a built-in tile importer. Go to File->Import Offline Map Tiles...
and point Mapeo to the tiles you want to use. It accepts a directory of tiles organized by /{zoom}/{x}/{y}
. You can change these parameters when you launch Mapeo desktop in the background imagery layers menu.
The automatic importer attempts to use .jpg
, .png
, and .jpeg
as the format for each tile.
You can put styles and tiles into the following directory:
USERDATA is the per-user application data directory, which by default points to:
%APPDATA%
on Windows
$XDG_CONFIG_HOME
or ~/.config on Linux
~/Library/Application Support
on macOS
Similar to Mapeo Mobile, this expects these files directly under this default
folder:
After moving files to the directory (if you aren't using automatic import), try manually refreshing Mapeo to see your new tiles.
Once you've imported your tiles, in the Map Editor, you need to follow some more steps to get the OpenStreetMap iD editor to recognize your new tiles.
Press 'b' to open the imagery layers menu.
Choose 'Custom' from the bottom of the list (you need to scroll quite a lot).
Input the appropriate URL (continue reading)
If you used a tar
file, you can use the default setting:
If you used an .asar
file or used Manual Import, you need to specify the name of your asar file that contains the tiles you want to view:
Map Filter requires a style.json
file in the default
directory
TODO: document how to get this file, or perhaps Mapeo should generate a default one for the user!
tile-dl
needs to be told the latitude, longitude, zoom level, and radius of the area to download locally.
To find the latitude and longitude of the area, open Mapeo Desktop and navigate to the rough centre of the area you're interested in. In the bottom right hand corner of the screen you'll see two numbers separated by a comma. These are your current longitude and latitude (in that order). Note them.
Let's store the template for a map tile provider for use by tile-dl:
On Windows
Now you can invoke the tile-dl
program:
On Windows
This example downloads the area around Oakland, California. You can tweak the parameters to meet your needs:
lat
: The latitude at the centre of your download area.
lon
: The longitude at the centre of your download area.
radius
: The size of the area to download, in kilometres.
zoom
: The zoom level to use. 9 is a very wide area; 11 is a large area; 13 is the size of a village; 16 is the size of a small road.
The above zoom level figures are very rough. Experiment with small radii and see how the results look: you can open the resulting JPGs or PNGs with an image viewer and see if they look appropriate to your needs.
Map tile providers offer these satellite images for free; please don't abuse their generosity by downloading more than what you need.
It will accept .tar
files. It does not currently support the .mbtiles
format, .
These instructions were written with systems in mind (Linux, macOS). Windows users may have to infer some of the differences in the commands shown.
If you don't have tile data already, you can use a command line tool that we created to do this. First, make sure you have installed.
Next, install :