http://simonwillison.net/2021-12-06T01:41:54+00:00Simon Willison2021-12-06T01:41:54+00:002021-12-06T01:41:54+00:00https://simonwillison.net/2021/Dec/6/centrifue/#atom-tag

<p><strong><a href="https://segment.com/blog/introducing-centrifuge/">Centrifuge: a reliable system for delivering billions of events per day</a></strong></p> From 2018, a write-up from Segment explaining how they solved the problem of delivering webhooks from thousands of different producers to hundreds of potentially unreliable endpoints. They started with Kafka and ended up on a custom system written in Go against RDS MySQL that was specifically tuned to their write-heavy requirements. <p><small></small>Via <a href="https://twitter.com/jkakar/status/1467653837917863936">@jkakar</a></small></p> <p>Tags: <a href="https://simonwillison.net/tags/scaling">scaling</a>, <a href="https://simonwillison.net/tags/webhooks">webhooks</a>, <a href="https://simonwillison.net/tags/segment">segment</a></p>

2020-01-10T05:15:49+00:002020-01-10T05:15:49+00:00https://simonwillison.net/2020/Jan/10/serving-100s-reads-with-100-availability/#atom-tag

<p><strong><a href="https://segment.com/blog/separating-our-data-and-control-planes-with-ctlstore/">Serving 100µs reads with 100% availability</a></strong></p> Fascinating use-case for SQLite from Segment: they needed a massively replicated configuration database across all of their instances that process streaming data. They chose to make the configuration available as a ~50GB SQLite database file mirrored to every instance, meaning lookups against that data could complete in microseconds. Changes to the central MySQL configuration store are pulled every 2-3 seconds, resulting in a trade-off of consistency for availability which fits their use-case just fine. <p><small></small>Via <a href="https://medium.com/@rbranson/sharing-sqlite-databases-across-containers-is-surprisingly-brilliant-bacb8d753054">Sharing an SQLite database across containers is surprisingly brilliant</a></small></p> <p>Tags: <a href="https://simonwillison.net/tags/scaling">scaling</a>, <a href="https://simonwillison.net/tags/sqlite">sqlite</a>, <a href="https://simonwillison.net/tags/segment">segment</a></p>