Simon Willison's Weblog: foursquare

Foursquare Open Source Places: A new foundational dataset for the geospatial community

2024-11-20T05:52:38+00:00

Foursquare Open Source Places: A new foundational dataset for the geospatial community

I did not expect this!

[...] we are announcing today the general availability of a foundational open data set, Foursquare Open Source Places ("FSQ OS Places"). This base layer of 100mm+ global places of interest ("POI") includes 22 core attributes (see schema here) that will be updated monthly and available for commercial use under the Apache 2.0 license framework.

The data is available as Parquet files hosted on Amazon S3.

Here's how to list the available files:

aws s3 ls s3://fsq-os-places-us-east-1/release/dt=2024-11-19/places/parquet/

I got back places-00000.snappy.parquet through places-00024.snappy.parquet, each file around 455MB for a total of 10.6GB of data.

I ran duckdb and then used DuckDB's ability to remotely query Parquet on S3 to explore the data a bit more without downloading it to my laptop first:

select count(*) from 's3://fsq-os-places-us-east-1/release/dt=2024-11-19/places/parquet/places-00000.snappy.parquet';

This got back 4,180,424 - that number is similar for each file, suggesting around 104,000,000 records total.

Update: DuckDB can use wildcards in S3 paths (thanks, Paul) so this query provides an exact count:

select count(*) from 's3://fsq-os-places-us-east-1/release/dt=2024-11-19/places/parquet/places-*.snappy.parquet';

That returned 104,511,073 - and Activity Monitor on my Mac confirmed that DuckDB only needed to fetch 1.2MB of data to answer that query.

I ran this query to retrieve 1,000 places from that first file as newline-delimited JSON:

copy (
    select * from 's3://fsq-os-places-us-east-1/release/dt=2024-11-19/places/parquet/places-00000.snappy.parquet'
    limit 1000
) to '/tmp/places.json';

Here's that places.json file, and here it is imported into Datasette Lite.

Finally, I got ChatGPT Code Interpreter to convert that file to GeoJSON and pasted the result into this Gist, giving me a map of those thousand places (because Gists automatically render GeoJSON):

Via Andy Baio

Tags: geospatial, open-source, foursquare, geojson, parquet, duckdb, datasette-lite, ai-assisted-programming, code-interpreter, coding-agents

Which is the most complete and up to date API for restaurants/nightlife?

2013-10-19T18:48:00+00:00

My answer to Which is the most complete and up to date API for restaurants/nightlife? on Quora

The foursquare API is pretty great for restaurants and nightlife these days. No chance if revenue share though - how would you envisage revenue share working?

Tags: apis, yelp, quora, foursquare

Events (leisure): What platforms based on sharing/creating users' future plans with friends have been successful?

2013-07-21T12:35:00+00:00

My answer to Events (leisure): What platforms based on sharing/creating users' future plans with friends have been successful? on Quora

The founder of Plancast wrote an extensive post mortem discussing the challenges involved here:

http://www.techcrunch.com/2012/0...

A key challenge is that people don't tend to announce their plans more than a few days in advance, which means that the information they share has a very short shelf life during which it is useful.

We've managed to avoid that particular trap at http://lanyrd.com/ because people (in particular speakers) announce their participation in a conference months in advance - unlike social or entertainment events.

Tags: events, social-networks, quora, foursquare, plancast