Simon Willison's Weblog: mapping

OpenStreetMap vector tiles demo

2024-11-19T23:39:18+00:00

Long-time OpenStreetMap developer Paul Norman has been working on adding vector tile support to OpenStreetMap for quite a while. Paul recently announced that vector.openstreetmap.org is now serving vector tiles (in Mapbox Vector Tiles (MVT) format) - here's his interactive demo for seeing what they look like.

Via Mark Litwintschik

Tags: geospatial, mapping, openstreetmap

Visualizing local election results with Datasette, Observable and MapLibre GL

2024-11-09T23:32:06+00:00

Alex Garcia and myself hosted the first Datasette Open Office Hours on Friday - a live-streamed video session where we hacked on a project together and took questions and tips from community members on Discord.

We didn't record this one (surprisingly not a feature that Discord offers) but we hope to do more of these and record them in the future.

This post is a detailed write-up of what we built during the session.

San Mateo County election results

I live in El Granada, a tiny town just north of Half Moon Bay in San Mateo County, California.

Every county appears to handle counting and publishing election results differently. For San Mateo County the results are published on this page, and detailed per-precinct and per-candidate breakdowns are made available as a CSV file.

(I optimistically set up a Git scraper for these results in simonw/scrape-san-mateo-county-election-results-2024 only to learn that the CSV is updated just once a day, not continually as the ballots are counted.)

I'm particularly invested in the results of the Granada Community Services District board member elections. Our little town of El Granada is in "unincorporated San Mateo County" which means we don't have a mayor or any local officials, so the closest we get to hyper-local government is the officials that run our local sewage and parks organization! My partner Natalie ran the candidate forum event (effectively the debate) featuring three of the four candidates running for the two open places on the board.

Let's explore the data for that race using Datasette.

Importing CSV data into Datasette

I ran my part of the demo using Datasette Cloud, the beta of my new hosted Datasette service.

I started by using the pre-configured datasette-import plugin to import the data from the CSV file into a fresh table:

Modifying the schema

The table imported cleanly, but all of the columns from the CSV were still being treated as text. I used the datasette-edit-schema plugin to switch the relevant columns to integers so that we could run sums and sorts against them.

(I also noted that I really should add a "detect column types" feature to that plugin!)

The resulting 15,589 rows represent counts from individual precincts around the county for each of the races and measures on the ballot, with a row per precinct per candidate/choice per race.

Faceting and filtering the table

Since I'm interested in the Granada Community Services District election, I applied a facet on "Contest_title" and then used that to select that specific race.

I applied additional facets on "candidate_name" and "Precinct name".

This looks right to me: we have 7 precincts and 4 candidates for 28 rows in total.

Importing geospatial precinct shapes

Those precinct names are pretty non-descriptive! What does 33001 mean?

To answer that question, I added a new table.

San Mateo County offers precinct maps in the form of 23 PDF files. Our precincts are in the "Unincorporated Coastside" file:

Thankfully the county also makes that data available as geospatial data, hosted using Socrata with an option to export as GeoJSON.

The datasette-import plugin can handle JSON files... and if a JSON file contains a top-level object with a key that is an array of objects, it will import those objects as a table.

Dragging that file into Datasette is enough to import it as a table with a properties JSON column containing properties and a geometry JSON columnn with the GeoJSON geometry.

Here's where another plugin kicks in: datasette-leaflet-geojson looks for columns that contain valid GeoJSON geometries and... draws them on a map!

So now we can see the shape of the individual geometries.

Enriching that data to extract the precinct IDs

The precinctid is present in the data, but it's tucked away in a JSON object in that properties JSON blob. It would be more convenient if it was a top-level column.

Datasette's enrichments feature provides tools for running operations against every row in a table and adding new columns based on the results.

My Datasette Cloud instance was missing the datasette-enrichments-quickjs plugin that would let me run JavaScript code against the data. I used my privileged access on Datasette Cloud to add that plugin to my requirements and restarted the instance to install it.

I used that to run this JavaScript code against every row in the table and saved the output in a new precinct_id column:

function enrich(row) {
  return JSON.parse(row.properties).precinctid;
}

This took less than a second to run, adding and populating a new precinct_id column for the table.

Running a join

I demonstrated how to run a join between the election results and the precincts table using the Datasette SQL query editor.

I tried a few different things, but the most interesting query was this one:

select
  Precinct_name,
  precincts.geometry,
  total_ballots,
  json_group_object(
    candidate_name,
    total_votes
  ) as votes_by_candidate
from
  election_results 
  join precincts on election_results.Precinct_name = precincts.precinct_id
where 
  Contest_title = "Granada Community Services District Members, Board of Directors"
group by 
  Precinct_name,
  precincts.geometry,
  total_ballots;

Creating an API token to access the data

I was nearly ready to hand over to Alex for the second half of our demo, where he would use Observable Notebooks to build some custom visualizations on top of the data.

A great pattern for this is to host the data in Datasette and then fetch it into Observable via the Datasette JSON API.

Since Datasette Cloud instances are private by default we would need to create an API token that could do this.

I used this interface (from the datasette-auth-tokens plugin) to create a new token with read-only access to all databases and tables in the instance:

Since we're running a dedicated instance just for Datasette Public Office Hours there's no reason not to distribute that read-only token in publically accessible code.

Getting CORS working

Embarrassingly, I had forgotten that we would need CORS headers in order to access the data from an Observable notebook. Thankfully we have another plugin for that: datasette-cors. I installed that quickly and we confirmed that it granted access to the API from Observable as intended.

I handed over to Alex for the next section of the demo.

Working with Datasette in Observable

Alex started by running a SQL query from client-side JavaScript to pull in the joined data for our specific El Granada race:

sql = `
select
  Precinct_name,
  precincts.geometry,
  Split_name,
  Reporting_flag,
  Update_count,
  Pct_Id,
  Pct_seq_nbr,
  Reg_voters,
  Turn_Out,
  Contest_Id,
  Contest_seq_nbr,
  Contest_title,
  Contest_party_name,
  Selectable_Options,
  candidate_id,
  candidate_name,
  Candidate_Type,
  cand_seq_nbr,
  Party_Code,
  total_ballots,
  total_votes,
  total_under_votes,
  total_over_votes,
  [Vote Centers_ballots],
  [Vote Centers_votes],
  [Vote Centers_under_votes],
  [Vote Centers_over_votes],
  [Vote by Mail_ballots],
  [Vote by Mail_votes],
  [Vote by Mail_under_votes],
  [Vote by Mail_over_votes]
from
  election_results join precincts on election_results.Precinct_name = precincts.precinct_id
where "Contest_title" = "Granada Community Services District Members, Board of Directors"
limit 101;`

And in the next cell:

raw_data = fetch(
  `https://datasette-public-office-hours.datasette.cloud/data/-/query.json?_shape=array&sql=${encodeURIComponent(
    sql
  )}`,
  {
    headers: {
      Authorization: `Bearer ${secret}`
    }
  }
).then((r) => r.json())

Note the ?_shape=array parameter there, which causes Datasette to output the results directly as a JSON array of objects.

That's all it takes to get the data into Observable. Adding another cell like this confirms that the data is now available:

Inputs.table(raw_data)

Visualizing those with MapLibre GL

There are plenty of good options for visualizing GeoJSON data using JavaScript in an Observable notebook.

Alex started with MapLibre GL, using the excellent OpenFreeMap 3D tiles:

viewof map = {
  const container = html`<div style="height:800px;">`;
  yield container;
  const map = (container.value = new maplibregl.Map({
    container,
    zoom: 2,
    //style: "https://basemaps.cartocdn.com/gl/voyager-gl-style/style.json",
    style: "https://tiles.openfreemap.org/styles/liberty",
    scrollZoom: true
  }));
  yield container;

  map.on("load", function () {
    map.fitBounds(d3.geoBounds(data), { duration: 0 });
    map.addSource("precincts", {
      type: "geojson",
      data: data
    });
    map.addLayer({
      id: "precincts",
      type: "fill",
      source: "precincts",
      paint: {
        "fill-opacity": 0.4,
        "fill-color": [
          "case",
          ["==", ["get", "ratio"], null], "#000000",
          [
            "interpolate",
            ["linear"],
            ["get", "ratio"],
            0.0, "#0000ff",
            0.5, "#d3d3d3",
            1.0, "#ff0000"
          ]
        ]
      }
    });
    map.on("click", "precincts", (e) => {
      const { precinct, ratio } = e.features[0].properties;
      const description = JSON.stringify();

      new maplibregl.Popup()
        .setLngLat(e.lngLat)
        .setHTML(description)
        .addTo(map);
    });
  });
  invalidation.then(() => map.remove());
}

(This is just one of several iterations, I didn't capture detailed notes of every change Alex made to the code.)

Observable Plot

Observable notebooks come pre-loaded with the excellent Observable Plot charting library - Mike Bostock's high-level charting tool built on top of D3.

Alex used that to first render the shapes of the precincts directly, without even needing a tiled basemap:

Plot.plot({
  width,
  height: 600,
  legend: true,
  projection: {
    type: "conic-conformal",
    parallels: [37 + 4 / 60, 38 + 26 / 60],
    rotate: [120 + 30 / 60, 0],
    domain: data
  },
  marks: [
    Plot.geo(data, {
      strokeOpacity: 0.1,
      fill: "total_votes",
      title: (d) => JSON.stringify(d.properties),
      tip: true
    })
  ]
})

The parallels and rotate options there come from the handy veltman/d3-stateplane repo, which lists recommended settings for the State Plane Coordinate System used with projections in D3. Those values are for California Zone 3.

Bringing it all together

For the grand finale, Alex combined everything learned so far to build an interactive map allowing a user to select any of the 110 races on the ballot and see a heatmap of results for any selected candidate and option:

You can try this out in Alex's notebook. Here's the relevant code (Observable cells are divided by // --- comments). Note that Observable notebooks are reactive and allow variables to be referenced out of order.

// Select the contest
viewof contest = Inputs.select(contests, { label: "Choose a contest" })

// ---

// And the candidate
viewof candidate = Inputs.radio(
  candidates,
  {
    label: "Choose a candidate",
    value: candidates[0]
  }
)

// ---

// Show the map itself
Plot.plot({
  width,
  height: 600,
  legend: true,
  color: { scheme: "blues", legend: true },
  projection: {
    type: "mercator",
    domain: data2
  },
  marks: [
    Plot.geo(data2, {
      strokeOpacity: 0.1,
      fill: "ratio",
      tip: true
    })
  ]
})
# ---
data2 = ({
  type: "FeatureCollection",
  features: raw_data2.map((d) => ({
    type: "Feature",
    properties: {
      precinct: d.Precinct_name,
      total_ballots: d.total_ballots,
      ratio: JSON.parse(d.votes_by_candidate)[candidate] / d.total_ballots
    },
    geometry: JSON.parse(d.geometry)
  }))
})

// ---

raw_data2 = query(
  `select
  Precinct_name,
  precincts.geometry,
  total_ballots,
  json_grop_object(
    candidate_name,
    total_votes
  ) as votes_by_candidate
from
  election_results 
  join precincts on election_results.Precinct_name = precincts.precinct_id
where Contest_title = :contest
group by 
  Precinct_name,
  precincts.geometry,
  total_ballots;`,
  { contest }
)

// ---

raw_data2 = query(
  `select
  Precinct_name,
  precincts.geometry,
  total_ballots,
  json_group_object(
    candidate_name,
    total_votes
  ) as votes_by_candidate
from
  election_results 
  join precincts on election_results.Precinct_name = precincts.precinct_id
where Contest_title = :contest
group by 
  Precinct_name,
  precincts.geometry,
  total_ballots;`,
  { contest }
)

// ---

// Fetch the available contests
contests = query("select distinct Contest_title from election_results").then(
  (d) => d.map((d) => d.Contest_title)
)

// ---

// Extract available candidates for selected contest

candidates = Object.keys(
  JSON.parse(raw_data2[0].votes_by_candidate)
)

// ---

function query(sql, params = {}) {
  return fetch(
    `https://datasette-public-office-hours.datasette.cloud/data/-/query.json?${new URLSearchParams(
      { sql, _shape: "array", ...params }
    ).toString()}`,
    {
      headers: {
        Authorization: `Bearer ${secret}`
      }
    }
  ).then((r) => r.json());
}

We'll be doing this again

This was our first time trying something like this and I think it worked really well. We're already thinking about ways to improve it next time:

I want to record these sessions and make them available on YouTube for people who couldn't be there live
It would be fun to mix up the format. I'm particularly keen on getting more people involved giving demos - maybe having 5-10 minute lightning demo slots so we can see what other people are working on

Keep an eye on this blog or on the Datasette Discord for news about future sessions.

Tags: geospatial, mapping, politics, projects, datasette, datasette-cloud, alex-garcia, datasette-public-office-hours, leaflet

tiny-world-map

2024-04-21T22:11:14+00:00

tiny-world-map

I love this project. It’s a JavaScript file (694K uncompressed, 283KB compressed) which can be used with the Leaflet mapping library and provides a SVG base map of the world with country borders and labels for every world city with a population more than 48,000—10,000 cities total.

This means you can bundle an offline map of the world as part of any application that doesn’t need a higher level of detail. A lot of smaller island nations are missing entirely though, so this may not be right for every project.

It even includes a service worker to help implement offline mapping support, plus several variants of the map with less cities that are even smaller.

Via Hacker News

Tags: geospatial, javascript, mapping, svg, service-workers, leaflet

DALL-E 3, GPT4All, PMTiles, sqlite-migrate, datasette-edit-schema

2023-10-30T23:59:12+00:00

I wrote a lot this week. I also did some fun research into new options for self-hosting vector maps and pushed out several new releases of plugins.

On the blog

Now add a walrus: Prompt engineering in DALL-E 3 talked about my explorations of the new DALL-E 3 image generation model, including some reverse engineering showing how OpenAI prompt engineered ChatGPT to pass generate its own prompts for DALL-E 3. And a lot of pictures of pelicans. I also wrote a TIL about the CSS grids I used in that post.
In Execute Jina embeddings with a CLI using llm-embed-jina I released a new plugin to run the new Jina AI 8K text embedding model using my LLM command-line tool.
Embeddings: What they are and why they matter is the big write-up of my talk about embeddings from PyBay this year. This has received a lot of traffic, presumably because it provides one of the more accessible answers to the question "what are embeddings?".

PMTiles and MapLibre GL

I saw a post about Protomaps on Hacker News. It's absolutely fantastic technology.

The Protomaps PMTiles file format lets you bundle together vector tiles in a single file which is designed to be queried using HTTP range header requests.

This means you can drop a single 107GB file on cloud hosting and use it to efficiently serve vector maps to clients, fetching just the data they need for the current map area.

Even better than that, you can create your own subset of the larger map covering just the area you care about.

I tried this out against my hometown of Half Moon Bay ond get a building-outline-level vector map for the whole town in just a 2MB file!

You can see the result (which also includes business listing markers from Overture maps) at simonw.github.io/hmb-map.

Lots more details of how I built this, including using Vite as a build tool and the MapLibre GL JavaScript library to serve the map, in my TIL Serving a custom vector web map using PMTiles and maplibre-gl.

I'm so excited about this: we now have the ability to entirely self-host vector maps of any location in the world, using openly licensed data, without depending on anything other than our own static file hosting web server.

llm-gpt4all

This was a tiny release - literally a one line code change - with a huge potential impact.

Nomic AI's GPT4All is a really cool project. They describe their focus as "a free-to-use, locally running, privacy-aware chatbot. No GPU or internet required." - they've taken llama.cpp (and other libraries) and wrapped them in a much nicer experience, complete with Windows, macOS and Ubuntu installers.

Under the hood it's mostly Python, and Nomic have done a fantastic job releasing that Python core as an installable Python package - meaning you can literally pip install gpt4all to get almost everything you need to run a local language model!

Unlike alternative Python libraries MLC and llama-cpp-python, Nomic have done the work to publish compiled binary wheels to PyPI... which means pip install gpt4all works without needing a compiler toolchain or any extra steps!

My LLM tool has had a llm-gpt4all plugin since I first added alternative model backends via plugins in July. Unfortunately, it spat out weird debugging information that I had been unable to hide (a problem that still affects llm-llama-cpp).

Nomic have fixed this!

As a result, llm-gpt4all is now my recommended plugin for getting started running local LLMs:

pipx install llm
llm install llm-gpt4all
llm -m mistral-7b-instruct-v0 "ten facts about pelicans"

The latest plugin can also now use the GPU on macOS, a key feature of Nomic's big release in September.

sqlite-migrate

sqlite-migrate is my plugin that adds a simple migration system to sqlite-utils, for applying changes to a database schema in a controlled, repeatable way.

Alex Garcia spotted a bug in the way it handled multiple migration sets with overlapping migration names, which is now fixed in sqlite-migrate 0.1b0.

Ironically the fix involved changing the schema of the _sqlite_migrations table used to track which migrations have been applied... which is the one part of the system that isn't itself managed by its own migration system! I had to implement a conditional check instead that checks if the table needs to be updated.

A recent thread about SQLite on Hacker News included a surprising number of complaints about the difficulty of running migrations, due to the lack of features of the core ALTER TABLE implementation.

The combination sqlite-migrate and the table.transform() method in sqlite-utils offers a pretty robust solution to this problem. Clearly I need to put more work into promoting it!

Homebrew trouble for LLM

I started getting confusing bug reports for my various LLM projects, all of which boiled down to a failure to install plugins that depended on PyTorch.

It turns out the LLM package for Homebrew upgraded to Python 3.12 last week... but PyTorch isn't yet available for Python 3.12.

This means that while base LLM installed from Homebrew works fine, attempts to install things like my new llm-embed-jina plugin fail with weird errors.

I'm not sure the best way to address this. For the moment I've removed the recommendation to install using Homebrew and replaced it with pipx in a few places. I have an open issue to find a better solution for this.

The difficulty of debugging this issue prompted me to ship a new plugin that I've been contemplating for a while: llm-python.

Installing this plugin adds a new llm python command, which runs a Python interpreter in same virtual environment as LLM - useful for if you installed LLM via pipx or Homebrew and don't know where that virtual environment is located.

It's great for debugging: I can ask people to run llm python -c 'import sys; print(sys.path)' for example to figure out what their Python path looks like.

It's also promising as a tool for future tutorials about the LLM Python library. I can tell people to pipx install llm and then run llm python to get a Python interpreter with the library already installed, without them having to mess around with virtual environments directly.

Add and remove indexes in datasette-edit-schema

We're iterating on Datasette Cloud based on feedback from people using the preview. One request was the ability to add and remove indexes from larger tables, to help speed up faceting.

datasette-edit-schema 0.7 adds that feature.

That plugin includes this script for automatically updating the screenshot in the README using shot-scraper. Here's the latest result:

Releases this week

sqlite-migrate 0.1b0 - 2023-10-27
A simple database migration system for SQLite, based on sqlite-utils
llm-python 0.1 - 2023-10-27
"llm python" is a command to run a Python interpreter in the LLM virtual environment
llm-embed-jina 0.1.2 - 2023-10-26
Embedding models from Jina AI
datasette-edit-schema 0.7 - 2023-10-26
Datasette plugin for modifying table schemas
datasette-ripgrep 0.8.2 - 2023-10-25
Web interface for searching your code using ripgrep, built as a Datasette plugin
llm-gpt4all 0.2 - 2023-10-24
Plugin for LLM adding support for the GPT4All collection of models

TIL this week

A simple two column CSS grid - 2023-10-27
Serving a custom vector web map using PMTiles and maplibre-gl - 2023-10-24
Serving a JavaScript project built using Vite from GitHub Pages - 2023-10-24

Tags: mapping, projects, weeknotes, sqlite-utils, llm, overture

GPSJam

2022-07-30T19:51:31+00:00

GPSJam

John Wiseman’s “Daily maps of GPS interference” —a beautiful interactive globe (powered by Mapbox GL) which you can use to see points of heaviest GPS interference over a 24 hour period, using data collected from commercial airline radios by ADS-B Exchange. “From what I can tell the most common reason for aircraft GPS systems to have degraded accuracy is jamming by military systems. At least, the vast majority of aircraft that I see with bad GPS accuracy are flying near conflict zones where GPS jamming is known to occur.”

Via @lemonodor

Tags: geospatial, gps, mapping, john-wiseman

Serving map tiles from SQLite with MBTiles and datasette-tiles

2021-02-04T01:09:30+00:00

Working on datasette-leaflet last week re-kindled my interest in using Datasette as a GIS (Geographic Information System) platform. SQLite already has strong GIS functionality in the form of SpatiaLite and datasette-cluster-map is currently the most downloaded plugin. Most importantly, maps are fun!

MBTiles

I was talking to Tom MacWright on Monday and I mentioned that I'd been thinking about how SQLite might make a good mechanism for distributing tile images for use with libraries like Leaflet. "I might be able to save you some time there" he said... and he showed me MBTiles, a specification he started developing ten years ago at Mapbox which does exactly that - bundles tile images up in SQLite databases.

(My best guess is I read about MBTiles a while ago, then managed to forget about the spec entirely while the idea of using SQLite for tile distribution wedged itself in my head somewhere.)

The new datasette-tiles plugin

I found some example MBTiles files on the internet and started playing around with them. My first prototype used the datasette-media plugin, described here previously in Fun with binary data and SQLite. I used some convoluted SQL to teach it that hits to /-/media/tiles/{z},{x},{y} should serve up content from the tiles table in my MBTiles database - you can see details of that prototype in this TIL: Serving MBTiles with datasette-media.

The obvious next step was to write a dedicated plugin: datasette-tiles. Install it and run Datasette against any MBTiles database file and the plugin will set up a /-/tiles/db-name/z/x/y.png endpoint that serves the specified tiles.

It also adds a tile explorer view with a pre-configured Leaflet map. Here's a live demo serving up a subset of Stamen's toner map - just zoom levels 6 and 7 for the country of Japan.

Here's how to run this on your own computer:

# Install Datasette
brew install datasette
# Install the plugin
datasette install datasette-tiles
# Download the japan-toner.db database
curl -O https://datasette-tiles-demo.datasette.io/japan-toner.db
# Launch Datasette and open a browser
datasette japan-toner.db -o
# Use the cog menu to access the tile explorer
# Or visit http://127.0.0.1:8001/-/tiles/japan-toner

Creating MBTiles files with my download-tiles tool

A sticking point when I started playing with MBTiles was finding example files to work with.

After some digging, I came across the amazing HOT Export Tool. It's a project by the Humanitarian OpenStreetMap Team that allows anyone to export subsets of data from OpenStreetMap in a wide variety of formats, including MBTiles.

I filed a minor bug report against it, and in doing so took a look at the source code (it's all open source)... and found the code that assembles MBTiles files. It uses another open source library called Landez, which provides functions for downloading tiles from existing providers and bundling those up as an MBTiles SQLite file.

I prefer command-line tools for this kind of thing over using Python libraries directly, so I fired up my click-app cookiecutter template and built a thin command-line interface over the top of the library.

The new tool is called download-tiles and it does exactly that: downloads tiles from a tile server and creates an MBTiles SQLite database on disk containing those tiles.

Please use this tool responsibly. Downloading large numbers of tiles is bad manners. Be sure to familiarize yourself with the OpenStreetMap Tile Usage Policy, and use the tool politely when pointing it at other tile servers.

Basic usage is as follows:

download-tiles world.mbtiles

By default the tool pulls tiles from OpenStreetMap. The above command will fetch zoom levels 0-3 of the entire world - 85 tiles total, well within acceptable usage limits.

Various options (described in the README) can be used to customize the tiles that are downloaded. Here's how I created the japan-toner.db demo database, linked to above:

download-tiles japan-toner.mbtiles \ 
    --zoom-levels 6-7 \
    --country Japan \
    --tiles-url "http://{s}.tile.stamen.com/toner/{z}/{x}/{y}.png" \
    --tiles-subdomains "a,b,c,d" \
    --attribution 'Map tiles by Stamen Design, under CC BY 3.0. Data by OpenStreetMap, under CC BY SA.'

The --country Japan option here looks up the bounding box for Japan using Nominatim. --zoom-levels 6-7 fetches zoom levels 6 and 7 (in this case that makes for 193 tiles total). --tiles-url and --tiles-subdomain configure the tile server to fetch them from. The --attribution option bakes that string into the metadata table for the database - which is then used to display it correctly in the tile explorer (and eventually in other Datasette plugins).

datasette-basemap

Out of the box, Datasette's current Leaflet plugins (datasette-cluster-map, datasette-leaflet-geojson and so on) serve tiles directly from the OpenStreetMap tile server.

I've never felt particularly comfortable about this. Users can configure the plugins to run against other tile servers, but pointing to OpenStreetMap as a default was the easiest way to ensure these plugins would work for people who just wanted to try them out.

Now that I have the tooling for bundling map subsets, maybe I can do better.

datasette-basemap offers an alternative: it's a plugin that bundles a 22.7MB SQLite file containing zoom levels 0-6 of OpenStreetMap - 5,461 tiles total.

Running pip install datasette-basemap (or datasette install datasette-basemap) will install the plugin, complete with that database - and register it with Datasette.

Start Datasette with the plugin installed and /basemap will expose the bundled database. Install datasette-tiles and you'll be able to browse it as a tile server: here's a demo.

(I recommend also installing datasette-render-images so you can see the tile images themselves in the regular table view, like this.)

Zoom level 6 is close enough that major cities and the roads between them are visible, for all of the countries in the world. Not bad for 22.7MB!

This is the first time I've built a Datasette plugin that bundles a full SQLite database as part of the Python package. The pattern seems to work well - I'm excited to explore it further with other projects.

Bonus feature: tile stacks

I added one last feature to datasette-tiles before writing everything up for my blog. I'm calling this feature tile stacks - it lets you serve tiles from multiple MBTiles files, falling back to other files if a tile is missing.

Imagine you had a low-zoom-level world map (similar to datasette-basemap) and a number of other databases providing packages of tiles for specific countries or cities. You could run Datasette like this:

datasette basemap.mbtiles japan.mbtiles london.mbtiles tokyo.mbtiles

Hitting /-/tiles-stack/1/1/1.png would seek out the specified tile in the tokyo.mbtiles file, then fall back to london.mbtiles and then japan.mbtiles and finally basemap.mbtiles if it couldn't find it.

For a demo, visit https://datasette-tiles-demo.datasette.io/-/tiles-stack and zoom in on Japan. It should start to display the Stamen toner map once you get to zoom levels 6 and 7.

Next steps

I've been having a lot of fun exploring MBTiles - it's such a natural fit for Datasette, and it's exciting to be able to build new things on top of nearly a decade of innovation by other geo-hackers.

There are plenty of features missing from datasette-tiles.

It currently only handles .png image data, but the MBTiles 1.3 specification also defines .jpg and .webp tiles, plus vector tiles using Mapbox's .pbf gzip-compressed protocol buffers.

UTFGrid is a related specification for including "rasterized interaction data" in MBTiles databases - it helps efficiently provide maps with millions of embedded objects.

As a newcomer to the MBTiles world I'd love to hear suggestions for new features and feedback on how I can improve what I've got so far in the datasette-tiles issues.

Being able to serve your own map tiles like this feels very much in the spirit of the OpenStreetMap project. I'm looking forward to using my own tile subsets for any future projects that fit within a sensible tile subset.

Tags: geospatial, mapping, openstreetmap, projects, sqlite, datasette, tom-macwright, leaflet

Polymaps

2010-08-20T18:46:00+00:00

Polymaps

Absurdly classy: “a JavaScript library for image- and vector-tiled maps using SVG”. It can pull in image tiles from sources such as OpenStreetMap, then overlay SVG paths specified using GeoJSON. The demos make use of GeoJSON tiles for US states and counties hosted on AppEngine. The library is developed by Stamen and SimpleGeo, and released under a BSD license. SVG support in the browser is required.

Tags: google-app-engine, javascript, mapping, openstreetmap, stamen-design, svg, recovered, geojson, polymaps, simplegeo

MapOSMatic

2010-07-11T12:15:00+00:00

MapOSMatic

Clever service built on top of OpenStreetMap, which renders double sided city maps with a map and grid on one size and an A-Z street name index on the other. Runs on top of Mapnik, PostGIS and Cairo, with a few thousand additional lines of Python and Django.

Tags: cairo, django, mapping, openstreetmap, postgis, postgresql, python, recovered, maposmatic

Doing things with Ordnance Survey OpenData

2010-05-20T15:22:00+00:00

Doing things with Ordnance Survey OpenData

Jo Walsh’s guide to processing Ordnance Survey OpenData using PostgreSQL and PostGIS.

Tags: mapping, open-data, ordnancesurvey, postgis, postgresql, recovered, jo-walsh

ClearMaps: A Mapping Framework for Data Visualization

2010-02-28T15:52:04+00:00

ClearMaps: A Mapping Framework for Data Visualization

An open source library for map visualisations using ActionScript, with an Adobe AIR based encoding tool for translating data from shapefiles in to vector data suitable for use with the library.

Tags: clearmaps, flash, mapping, visualisations

OSM the default map in Haiti

2010-01-25T21:26:57+00:00

OSM the default map in Haiti

A search and rescue team member in Haiti sends word that digital maps constructed by the OpenStreetMap community are spreading by word of mouth and being loaded on to GPS units on the ground.

Tags: gps, haiti, inspiring, mapping, openstreetmap

The View from Above

2009-12-11T09:32:10+00:00

The View from Above

Andy Allan’s notes on three different projects that aerial imagery with OpenStreetMap. Andy and friends hired a small plane and took their own aerial photographs of Stratford-upon-Avon as a demo for a GIS conference. Aid agencies in the Philippines benefitted from OSM and a donation of high quality satellite imagery. Rural Georgia now has hiqh quality images from 2007 thanks to the Department of Agriculture.

Tags: aerialimagery, andy-allan, mapping, openstreetmap, satellites

GeoPlanet data available again

2009-12-11T08:17:27+00:00

GeoPlanet data available again

Good news: the Yahoo! GeoPlanet data dump is available again. An issue with one of their data providers meant they had to remove that supplier’s data from the dump, but it’s now been separated and the dataset is live gain. By the end of 2010 they intend to derive all of the data from completely open sources.

Via Importing GeoPlanet data into MySQL

Tags: geoplanet, geospatial, mapping, yahoo

Quoting Stephen Timms

2009-11-17T18:10:38+00:00

About 80 per cent of public sector data mentions a place. Making Ordnance Survey data more freely available will encourage more effective exploitation of public data by businesses, individuals and community organisations.

— Stephen Timms, Minister for Digital Britain

Tags: datagov, mapping, ordnancesurvey, stephen-timms

Re-mapping the future for Ordnance Survey - making public data public

2009-11-17T18:09:29+00:00

Re-mapping the future for Ordnance Survey - making public data public

“The Prime Minister and Communities Secretary John Denham will today announce that the public will have more access to Ordnance Survey maps from next year, as part of a Government drive to open up data to improve transparency.”

Tags: datagov, mapping, ordnancesurvey

Quoting Ivan Sanchez

2009-11-12T10:52:31+00:00

A set of geodata, or a map, is libre only if somebody can give you a cake with that map on top, as a present.

— Ivan Sanchez

Tags: caketest, geospatial, ivansanchez, mapping, openstreetmap

How to Make a US County Thematic Map Using Free Tools

2009-11-12T10:49:23+00:00

How to Make a US County Thematic Map Using Free Tools

This is the trick I’ve been using to generate choropleths at the Guardian for the past year: figure out the preferred colours for a set of data in a Python script and then rewrite an SVG file to colour in the areas. I use ElementTree rather than BeautifulSoup but the technique is exactly the same. The best thing about SVG is that our graphics department can export them directly out of Illustrator, with named layers and paths automatically becoming SVG ID attributes. Bonus tip: sometimes you don’t have to rewrite the SVG XML at all, instead you can generate CSS to colour areas by ID selector and inject it in to the top of the file.

Tags: beautifulsoup, choropleths, css, elementtree, infographics, mapping, python, svg

Cartographer.js

2009-11-01T13:20:05+00:00

Cartographer.js

“Thematic mapping for Google Maps”—which means an easy way of adding heat maps (aka chloropleths), pie charts and point clusters as a layer over a Google map.

Tags: catography, chloropleths, google, google-maps, graphs, heatmaps, infographics, mapping, maps, piecharts, visualisation

Temporary Mapping: Solar Decathlon

2009-10-13T15:18:13+00:00

Temporary Mapping: Solar Decathlon

The OpenStreetMap default renderer supports start_date and end_date tags, meaning you can map temporary installations (in this case the 2009 Solar Decathlon on the DC National Mall) and have them automatically appear and disappear at the correct times.

Tags: andrew-turner, mapping, openstreetmap, solardecathlon, tagging

OSM static map api

2009-10-12T13:37:42+00:00

OSM static map api

A very welcome addition to the OpenStreetMap world (with plenty of options for overlaying points, polygons etc) slightly marred by the size and relative ugliness of the OpenStreetMap watermark.

Tags: apis, mapping, openstreetmap, staticmaps, watermarks

OpenStreetMap Rendering Database

2009-10-10T13:05:43+00:00

OpenStreetMap Rendering Database

Amazon have added an OpenStreetMap snapshot as a public data set, thanks to some smart prompting by Jeremy Dunck.

Tags: amazon, ec2, jeremy-dunck, mapping, openstreetmap, publicdatasets, s3

openstreetmap genuine advantage

2009-09-29T09:49:52+00:00

openstreetmap genuine advantage

The OpenStreetMap data model (points, ways and relations, all allowing arbitrary key/value tags) is a real thing of beauty—simple to understand but almost infinitely extensible. Mike Migurski’s latest project adds PGP signing to OpenStreetMap, allowing organisations (such as local government) to add a signature to a way (a sequence of points) and a subset of its tags, then write that signature in to a new tag on the object.

Tags: cryptography, geospatial, mapping, michal-migurski, openstreetmap, pgp

Tile Drawer

2009-08-26T09:32:56+00:00

Tile Drawer

The most inspired use of EC2 I’ve seen yet: center a map on an area, pick a Cascadenik stylesheet URL (or write and link to your own) and Tile Drawer gives you an Amazon EC2 AMI and a short JSON snippet. Launch the AMI with the JSON as the “user data” parameter and you get your own OpenStreetMap tile rendering server, which self-configures on startup and starts rendering and serving tiles using your custom design.

Via Mike Migurski

Tags: amazon, cascadenik, cloud-computing, ec2, json, mapnik, mapping, michal-migurski, openstreetmap, userdata

Static Maps API v2

2009-08-26T09:01:43+00:00

Static Maps API v2

The new version of the Google Static Maps API (static images generated using arguments in a URL, no JavaScript required) adds support for paths, areas and automatically geocoding addresses to specify locations of markers and the centre of the map.

Tags: google, google-maps, mapping, staticmapsapi

Best of OpenStreetMap

2009-08-13T12:30:05+00:00

Best of OpenStreetMap

I keep on telling people OpenStreetMap is this year’s Wikipedia—at its best, it beats commercially available maps. This “best of” site highlights the areas where OSM really shines (the yellow stars)—the German mapping community in particular have produced some outstanding cartography.

Via OpenGeoData

Tags: cartography, mapping, maps, openstreetmap, wikipedia

Hack Day tools for non-developers

2009-07-28T14:23:53+00:00

We're about to run our second internal hack day at the Guardian. The first was an enormous amount of fun and the second one looks set to be even more productive.

There's only one rule at hack day: build something you can demonstrate at the end of the event (Powerpoint slides don't count). Importantly though, our hack days are not restricted to just our development team: anyone from the technology department can get involved, and we extend the invitation to other parts of the organisation as well. At the Guardian, this includes journalists.

For our first hack day, I put together a list of "tools for non-developers" - sites, services and software that could be used for hacking without programming knowledge as a pre-requisite. I'm now updating that list with recommendations from elsewhere. Here's the list so far:

Freebase

Originally a kind of structured version of Wikipedia, Freebase changed its focus last year towards being a "social database about things you know and love". In other words, it's the most powerful OCD-enabler in the history of the world. Create your own "Base" on any subject you like, set up your own types and start gathering together topics from the millions already available in Freebase - or add your own. Examples include the Battlestar Galactica base, the Tall Ships base and the fabulous Database base. If you are a developer the tools in the Make Things with Freebase section are top notch.

Dabble DB

Dabble is a weird combination of a spreadsheet, an online database and a set of visualisation tools. Watch the 8 minute demo to get an idea of how powerful this is - you can start off by loading in an existing spreadsheet and take it from there. You'll need to sign up for the free 30 day trial.

Google Docs

You can always build a hack in Excel, but Google Spreadsheets is surprisingly powerful and means that you can collaborate with others on your hack (including developers, who can use the Google Docs API to get at the data in your spreadsheet). Check out the following tutorials, which describe ways of using Google Spreadsheets to scrape in data from other webpages and output it in interesting formats:

There's also a simple way to create a form that submits data in to a Google Spreadsheet.

Yahoo! Pipes

Visual tools for combining, filtering and modifying RSS feeds. Combine with the large number of full-content feeds on guardian.co.uk for all sorts of interesting possibilities. Here's a tutorial that incorporates Google Docs as well.

Google My Maps

Google provide a really neat interface for adding your own points, lines and areas to a Google Map. Outputs KML, a handy file format for carting geographic data around between different tools.

If you already have a KML or GeoRSS feed URL from somewhere (e.g. the output of a Yahoo! Pipe), you can paste it directly in to the Google Maps search box to see the points rendered on a map.

Google SketchUp

A simple to use 3D drawing package that lets you create 3D models of real-world buildings and then import them in to Google Earth.

OpenStreetMap

Try your hand at some open source cartography on OpenStreetMap, the geographic world's answer to Wikipedia. If you have the equipment you can contribute GPS traces, otherwise there's a clever online editor that will let you trace out roads from satellite photos - or you could just make sure your favourite pub is included on the map. The export tools can provide vector or static maps, and if you export as SVG you can further edit your map in Illustrator or Inkscape.

CloudMade Maps

Commercial tools built on top of OpenStreetMap, the most exciting of which allows you to create your own map theme by setting your preferred colours and line widths for various types of map feature.

Many Eyes

IBM Research's suite of data visualisation tools, with a wiki-style collaboration platform for publishing data and creating visualisations.

Dapper

Dapper provides a powerful tool for screen scraping websites, without needing to write any code. Output formats include RSS, iCalendar and Google Maps.

TiddlyWiki

TiddlyWiki is a complete wiki in a single HTML file, which you can save locally and use as a notebook, collaboration tool and much more. There's a large ecosystem of plugins and macros which can be used to extend it with new features - see TiddlyVault for an index.

WolframAlpha

The "computational knowledge engine" with the hubristic search-based interface, potentially useful as a source of data and a tool for processing and visualising that data.

Tumblr

Useful as both an input and an output for feeds processed using other tools, and with a smart bookmarklet for collecting bits and pieces from around the web.

The UCSB Toy Chest

An outstanding list of tools that people "without programming skills (but with basic computer and Internet literacy) can use to create interesting projects", compiled by the English department at UC Santa Barbara.

Your help needed

There must be dozens, if not hundreds of useful tools missing from the above. Tell me in the comments and I'll add them to the list.

Tags: freebase, google, google-maps, guardian, hackday, mapping, nondevelopers, openstreetmap, pipes, sketchup, tools, yahoo-pipes, computer-literacy

walking papers lives

2009-06-07T13:47:30+00:00

walking papers lives

Round trip mapping: print out a map from OpenStreetMap, walk around annotating it with a pen, then scan the result back in (a QR code ensures the area and orientation is recognised) . Specifically targeted at eye-level stuff which can’t be collected using GPS or aerial imagery alone. When I grow up, I want to be Mike Migurski.

Tags: mapping, michal-migurski, openstreetmap, papernet, walkingpapers

Mapstraction API Sandbox

2009-06-07T11:41:20+00:00

Mapstraction API Sandbox

Andrew Turner’s new tool for exploring the Mapstraction JavaScript library, which provides a unified code interface to 12 different mapping services

Tags: andrew-turner, javascript, mapping, mapstraction

Announcing Google Maps API v3

2009-05-28T01:22:45+00:00

Announcing Google Maps API v3

Sounds like a complete rewrite, with performance as the key goal. Only a developer preview at the moment, but my favourite feature is that API keys are no longer required.

Tags: api-keys, google, google-maps, googlemaps3, mapping

slippy faumaxion, take two

2009-03-15T15:40:57+00:00

slippy faumaxion, take two

Mike Migurski made a slippy map using triangular tiles, based on the same principle as Buckminster Fuller’s famous Dymaxion World Map.

Tags: buckminsterfuller, faumaxion, mapping, michal-migurski