Simon Willison's Weblog: npm

Supply Chain Attack on Axios Pulls Malicious Dependency from npm

2026-03-31T23:28:40+00:00

Supply Chain Attack on Axios Pulls Malicious Dependency from npm

Useful writeup of today's supply chain attack against Axios, the HTTP client NPM package with 101 million weekly downloads. Versions 1.14.1 and 0.30.4 both included a new dependency called plain-crypto-js which was freshly published malware, stealing credentials and installing a remote access trojan (RAT).

It looks like the attack came from a leaked long-lived npm token. Axios have an open issue to adopt trusted publishing, which would ensure that only their GitHub Actions workflows are able to publish to npm. The malware packages were published without an accompanying GitHub release, which strikes me as a useful heuristic for spotting potentially malicious releases - the same pattern was present for LiteLLM last week as well.

Via lobste.rs

Tags: javascript, security, npm, supply-chain

Package Managers Need to Cool Down

2026-03-24T21:11:38+00:00

Package Managers Need to Cool Down

Today's LiteLLM supply chain attack inspired me to revisit the idea of dependency cooldowns, the practice of only installing updated dependencies once they've been out in the wild for a few days to give the community a chance to spot if they've been subverted in some way.

This recent piece (March 4th) piece by Andrew Nesbitt reviews the current state of dependency cooldown mechanisms across different packaging tools. It's surprisingly well supported! There's been a flurry of activity across major packaging tools, including:

pnpm 10.16 (September 2025) — minimumReleaseAge with minimumReleaseAgeExclude for trusted packages
Yarn 4.10.0 (September 2025) — npmMinimalAgeGate (in minutes) with npmPreapprovedPackages for exemptions
Bun 1.3 (October 2025) — minimumReleaseAge via bunfig.toml
Deno 2.6 (December 2025) — --minimum-dependency-age for deno update and deno outdated
uv 0.9.17 (December 2025) — added relative duration support to existing --exclude-newer, plus per-package overrides via exclude-newer-package
pip 26.0 (January 2026) — --uploaded-prior-to (absolute timestamps only; relative duration support requested)
npm 11.10.0 (February 2026) — min-release-age

pip currently only supports absolute rather than relative dates but Seth Larson has a workaround for that using a scheduled cron to update the absolute date in the pip.conf config file.

Tags: javascript, packaging, pip, pypi, python, security, npm, deno, supply-chain, uv

ChatGPT Containers can now run bash, pip/npm install packages, and download files

2026-01-26T19:19:31+00:00

One of my favourite features of ChatGPT is its ability to write and execute code in a container. This feature launched as ChatGPT Code Interpreter nearly three years ago, was half-heartedly rebranded to "Advanced Data Analysis" at some point and is generally really difficult to find detailed documentation about. Case in point: it appears to have had a massive upgrade at some point in the past few months, and I can't find documentation about the new capabilities anywhere!

Here are the most notable new features:

ChatGPT can directly run Bash commands now. Previously it was limited to Python code only, although it could run shell commands via the Python subprocess module.
It has Node.js and can run JavaScript directly in addition to Python. I also got it to run "hello world" in Ruby, Perl, PHP, Go, Java, Swift, Kotlin, C and C++. No Rust yet though!
While the container still can't make outbound network requests, pip install package and npm install package both work now via a custom proxy mechanism.
ChatGPT can locate the URL for a file on the web and use a container.download tool to download that file and save it to a path within the sandboxed container.

This is a substantial upgrade! ChatGPT can now write and then test code in 10 new languages (11 if you count Bash), can find files online and download them into the container, and can install additional packages via pip and npm to help it solve problems.

(OpenAI really need to develop better habits at keeping their release notes up-to-date!)

I was initially suspicious that maybe I'd stumbled into a new preview feature that wasn't available to everyone, but I tried some experiments in a free ChatGPT account and confirmed that the new features are available there as well.

container.download

My first clue to the new features came the other day when I got curious about Los Angeles air quality, in particular has the growing number of electric vehicles there had a measurable impact?

I prompted a fresh GPT-5.2 Thinking session with:

Show me Los Angeles air quality over time for last 20 years

Here's the shared transcript. Annoyingly that doesn't make the thinking trace available, so here's a screenshot:

This bit instantly caught my eye:

[...] The URL from the error message leads to an Excel file containing air quality trends. I can use the container.download tool to fetch it. [...]

What's container.download? I asked ChatGPT:

Without running any searches tell me about your container.download tool

And got back:

Sure. container.download is a built-in tool I can call that fetches a file from a URL and saves it into the sandboxed container filesystem for this chat session.

What it does

Takes a publicly reachable URL and a destination filepath in the container.

Downloads the bytes from that URL and writes them to the given path.

After that, I can read/process the file locally in the container (e.g., unzip it, parse it with Python, open it as an image, convert it, etc.).

Signature (inputs)

It’s called like this (conceptually):
{
  "url": "https://example.com/somefile.zip",
  "filepath": "/mnt/data/somefile.zip"
}

Further experimentation confirmed that container.download behaves as described.

I had it download one file from a server with logs enabled and captured the IP address and user-agent. Here are the most interesting headers:

User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Accept: text/html, application/xhtml+xml, application/xml;q=0.9, image/avif, image/webp, image/apng, */*;q=0.8, application/signed-exchange;v=b3;q=0.9
Cf-Connecting-Ip: 52.230.164.178

That 52.230.164.178 IP address resolves to Microsoft Azure Cloud (centralus) in Des Moines, Iowa.

Is container.download a data exfiltration vulnerability?

On the one hand, this is really useful! ChatGPT can navigate around websites looking for useful files, download those files to a container and then process them using Python or other languages.

Is this a data exfiltration vulnerability though? Could a prompt injection attack trick ChatGPT into leaking private data out to a container.download call to a URL with a query string that includes sensitive information?

I don't think it can. I tried getting it to assemble a URL with a query string and access it using container.download and it couldn't do it. It told me that it got back this error:

ERROR: download failed because url not viewed in conversation before. open the file or url using web.run first.

This looks to me like the same safety trick used by Claude's Web Fetch tool: only allow URL access if that URL was either directly entered by the user or if it came from search results that could not have been influenced by a prompt injection.

(I poked at this a bit more and managed to get a simple constructed query string to pass through web.run - a different tool entirely - but when I tried to compose a longer query string containing the previous prompt history a web.run filter blocked it.)

So I think this is all safe, though I'm curious if it could hold firm against a more aggressive round of attacks from a seasoned security researcher.

Bash and other languages

The key lesson from coding agents like Claude Code and Codex CLI is that Bash rules everything: if an agent can run Bash commands in an environment it can do almost anything that can be achieved by typing commands into a computer.

When Anthropic added their own code interpreter feature to Claude last September they built that around Bash rather than just Python. It looks to me like OpenAI have now done the same thing for ChatGPT.

Here's what ChatGPT looks like when it runs a Bash command - here my prompt was:

npm install a fun package and demonstrate using it

It's useful to click on the "Thinking" or "Thought for 32s" links as that opens the Activity sidebar with a detailed trace of what ChatGPT did to arrive at its answer. This helps guard against cheating - ChatGPT might claim to have run Bash in the main window but it can't fake those black and white logs in the Activity panel.

I had it run Hello World in various languages later in that same session.

Installing packages from pip and npm

In the previous example ChatGPT installed the cowsay package from npm and used it to draw an ASCII-art cow. But how could it do that if the container can't make outbound network requests?

In another session I challenged it to explore its environment. and figure out how that worked.

Here's the resulting Markdown report it created.

The key magic appears to be a applied-caas-gateway1.internal.api.openai.org proxy, available within the container and with various packaging tools configured to use it.

The following environment variables cause pip and uv to install packages from that proxy instead of directly from PyPI:

PIP_INDEX_URL=https://reader:****@packages.applied-caas-gateway1.internal.api.openai.org/.../pypi-public/simple
PIP_TRUSTED_HOST=packages.applied-caas-gateway1.internal.api.openai.org
UV_INDEX_URL=https://reader:****@packages.applied-caas-gateway1.internal.api.openai.org/.../pypi-public/simple
UV_INSECURE_HOST=https://packages.applied-caas-gateway1.internal.api.openai.org

This one appears to get npm to work:

NPM_CONFIG_REGISTRY=https://reader:****@packages.applied-caas-gateway1.internal.api.openai.org/.../npm-public

And it reported these suspicious looking variables as well:

CAAS_ARTIFACTORY_BASE_URL=packages.applied-caas-gateway1.internal.api.openai.org
CAAS_ARTIFACTORY_PYPI_REGISTRY=.../artifactory/api/pypi/pypi-public
CAAS_ARTIFACTORY_NPM_REGISTRY=.../artifactory/api/npm/npm-public
CAAS_ARTIFACTORY_GO_REGISTRY=.../artifactory/api/go/golang-main
CAAS_ARTIFACTORY_MAVEN_REGISTRY=.../artifactory/maven-public
CAAS_ARTIFACTORY_GRADLE_REGISTRY=.../artifactory/gradle-public
CAAS_ARTIFACTORY_CARGO_REGISTRY=.../artifactory/api/cargo/cargo-public/index
CAAS_ARTIFACTORY_DOCKER_REGISTRY=.../dockerhub-public
CAAS_ARTIFACTORY_READER_USERNAME=reader
CAAS_ARTIFACTORY_READER_PASSWORD=****
NETWORK=caas_packages_only

Neither Rust nor Docker are installed in the container environment, but maybe those registry references are a clue of features still to come.

Adding it all together

The result of all of this? You can tell ChatGPT to use Python or Node.js packages as part of a conversation and it will be able to install them and apply them against files you upload or that it downloads from the public web. That's really cool.

The big missing feature here should be the easiest to provide: we need official documentation! A release notes entry would be a good start, but there are a lot of subtle details to how this new stuff works, its limitations and what it can be used for.

As always, I'd also encourage OpenAI to come up with a name for this set of features that properly represents how it works and what it can do.

In the meantime, I'm going to call this ChatGPT Containers.

Update: a full list of tools

I decided to ask ChatGPT about other tools that were available to it in case there was anything interesting in there:

List all tools that are available to you, with their exact names and descriptions and signatures

Here's what I got back.

List of ChatGPT tools according to GPT-5.2 Thinking on 26th January 2026

Below are all tools available to me in this chat, with their exact names, descriptions, and signatures (as defined in my tool interface).

python

`python.exec` (FREEFORM) → any

Description: Execute Python code in a Jupyter environment for private/internal reasoning (not shown to you). Internet access is disabled.
Signature: python.exec(FREEFORM) -> any

web

`web.run` (ToolCallV5 JSON) → any

Description: Access the internet and specialized lookups (search, open/click/find pages, screenshot PDFs, image search, product search, finance, weather, sports, calculator, time). Supports citations via returned turn... reference IDs.
Signature:
web.run({ open?, click?, find?, screenshot?, image_query?, product_query?, sports?, finance?, weather?, calculator?, time?, search_query?, response_length? }) -> any

automations

`automations.create` → any

Description: Schedule a future/recurring task/reminder/search via iCal VEVENT.
Signature:
automations.create({ prompt: string, title: string, schedule?: string, dtstart_offset_json?: string }) -> any

`automations.update` → any

Description: Modify an existing scheduled task.
Signature:
automations.update({ jawbone_id: string, schedule?: string, dtstart_offset_json?: string, prompt?: string, title?: string, is_enabled?: boolean }) -> any

`automations.list` → any

Description: List existing automations.
Signature: automations.list() -> any

guardian_tool

`guardian_tool.get_policy` → str

Description: Retrieve policy guidance for specific categories (currently: election_voting).
Signature: guardian_tool.get_policy({ category: string }) -> str

file_search

`file_search.msearch` → any

Description: Search across available internal/connected sources (in this environment: primarily “recording_knowledge” transcripts when relevant).
Signature:
file_search.msearch({ queries?: string[], source_filter?: string[], file_type_filter?: string[], intent?: string, time_frame_filter?: { start_date?: string, end_date?: string } }) -> any

`file_search.mclick` → any

Description: Expand/open previously returned search results (or connector URLs) for more context.
Signature:
file_search.mclick({ pointers?: string[], start_date?: string, end_date?: string }) -> any

gmail (read-only)

`gmail.search_email_ids` → any

Description: Search Gmail message IDs by query/tags (read-only).
Signature:
gmail.search_email_ids({ query?: string, tags?: string[], max_results?: integer, next_page_token?: string }) -> any

`gmail.batch_read_email` → any

Description: Read a batch of Gmail messages by message IDs (read-only).
Signature:
gmail.batch_read_email({ message_ids: string[] }) -> any

gcal (read-only)

`gcal.search_events` → any

Description: Search Google Calendar events in a time range and/or by keyword (read-only).
Signature:
gcal.search_events({ time_min?: string, time_max?: string, timezone_str?: string, max_results?: integer, query?: string, calendar_id?: string, next_page_token?: string }) -> any

`gcal.read_event` → any

Description: Read a specific calendar event by ID (read-only).
Signature:
gcal.read_event({ event_id: string, calendar_id?: string }) -> any

gcontacts (read-only)

`gcontacts.search_contacts` → any

Description: Search Google Contacts by keyword (read-only).
Signature:
gcontacts.search_contacts({ query: string, max_results?: integer }) -> any

canmore (canvas)

`canmore.create_textdoc` → any

Description: Create a new canvas text document (markdown or code).
Signature:
canmore.create_textdoc({ name: string, type: "document" | "code/bash" | "code/zsh" | "code/javascript" | "code/typescript" | "code/html" | "code/css" | "code/python" | "code/json" | "code/sql" | "code/go" | "code/yaml" | "code/java" | "code/rust" | "code/cpp" | "code/swift" | "code/php" | "code/xml" | "code/ruby" | "code/haskell" | "code/kotlin" | "code/csharp" | "code/c" | "code/objectivec" | "code/r" | "code/lua" | "code/dart" | "code/scala" | "code/perl" | "code/commonlisp" | "code/clojure" | "code/ocaml" | "code/powershell" | "code/verilog" | "code/dockerfile" | "code/vue" | "code/react" | "code/other", content: string }) -> any

`canmore.update_textdoc` → any

Description: Apply regex-based edits to the current canvas textdoc.
Signature:
canmore.update_textdoc({ updates: { pattern: string, multiple?: boolean, replacement: string }[] }) -> any

`canmore.comment_textdoc` → any

Description: Add actionable comments on matched regions of the current canvas textdoc.
Signature:
canmore.comment_textdoc({ comments: { pattern: string, comment: string }[] }) -> any

python_user_visible

`python_user_visible.exec` (FREEFORM) → any

Description: Execute Python code that you will see (tables, plots, generated files). Internet disabled.
Signature: python_user_visible.exec(FREEFORM) -> any

user_info

`user_info.get_user_info` → any

Description: Get your current location and local time (or UTC if unknown).
Signature: user_info.get_user_info({}) -> any

summary_reader

`summary_reader.read` → any

Description: Read safe-to-share internal reasoning summaries from earlier turns (used when asked how I arrived at something).
Signature: summary_reader.read({ limit?: integer, offset?: integer }) -> any

container

`container.exec` → any

Description: Run a command inside the container and return stdout/stderr.
Signature:
container.exec({ cmd: string[], session_name?: string | null, workdir?: string | null, timeout?: integer | null, env?: object | null, user?: string | null }) -> any

`container.feed_chars` → any

Description: Send input to a running interactive exec session and flush output.
Signature:
container.feed_chars({ session_name: string, chars: string, yield_time_ms?: integer }) -> any

`container.open_image` → any

Description: Display an image from an absolute path in the container (jpg/jpeg/png/webp).
Signature:
container.open_image({ path: string, user?: string | null }) -> any

`container.download` → any

Description: Download a file from a URL into the container filesystem.
Signature:
container.download({ url: string, filepath: string }) -> any

bio (memory)

`bio.update` (FREEFORM) → any

Description: Persist (or forget) user-specific info across conversations (“memory”).
Signature: bio.update(FREEFORM) -> any

api_tool

`api_tool.list_resources` → any

Description: List available “resources” (including invokable tools) in an OP-style file tree.
Signature:
api_tool.list_resources({ path?: string, cursor?: string | null, only_tools?: boolean, refetch_tools?: boolean }) -> any

`api_tool.call_tool` → any

Description: Invoke a resource as a tool (must know its full path/URI from list_resources).
Signature:
api_tool.call_tool({ path: string, args: object }) -> any

personal_context

`personal_context.search` → any

Description: Retrieve user-specific context from prior interactions/linked sources to maintain continuity.
Signature:
personal_context.search({ query: string }) -> any

image_gen

`image_gen.text2im` → any

Description: Generate or edit images from text instructions (and optionally referenced image IDs).
Signature:
image_gen.text2im({ prompt?: string | null, size?: string | null, n?: integer | null, transparent_background?: boolean | null, is_style_transfer?: boolean | null, referenced_image_ids?: string[] | null }) -> any

user_settings

`user_settings.get_user_settings` → any

Description: Read current settings and allowed values (personality, accent color, appearance).
Signature: user_settings.get_user_settings() -> any

`user_settings.set_setting` → any

Description: Change one setting (accent color, appearance, or personality) to an allowed value.
Signature:
user_settings.set_setting({ setting_name: "accent_color" | "appearance" | "personality", setting_value: string }) -> any

artifact_handoff

`artifact_handoff.prepare_artifact_generation` → any

Description: Must be called immediately when the user asks for a spreadsheet or slide deck artifact.
Signature: artifact_handoff.prepare_artifact_generation() -> any

Tags: pypi, sandboxing, npm, ai, openai, generative-ai, chatgpt, llms, ai-assisted-programming, code-interpreter

Introducing OSS Rebuild: Open Source, Rebuilt to Last

2025-07-23T17:16:32+00:00

Introducing OSS Rebuild: Open Source, Rebuilt to Last

Major news on the Reproducible Builds front: the Google Security team have announced OSS Rebuild, their project to provide build attestations for open source packages released through the NPM, PyPI and Crates ecosystom (and more to come).

They currently run builds against the "most popular" packages from those ecosystems:

Through automation and heuristics, we determine a prospective build definition for a target package and rebuild it. We semantically compare the result with the existing upstream artifact, normalizing each one to remove instabilities that cause bit-for-bit comparisons to fail (e.g. archive compression). Once we reproduce the package, we publish the build definition and outcome via SLSA Provenance. This attestation allows consumers to reliably verify a package's origin within the source history, understand and repeat its build process, and customize the build from a known-functional baseline

The only way to interact with the Rebuild data right now is through their Go CLI tool. I reverse-engineered it using Gemini 2.5 Pro and derived this command to get a list of all of their built packages:

 gsutil ls -r 'gs://google-rebuild-attestations/**'

There are 9,513 total lines, here's a Gist. I used Claude Code to count them across the different ecosystems (discounting duplicates for different versions of the same package):

pypi: 5,028 packages
cratesio: 2,437 packages
npm: 2,048 packages

Then I got a bit ambitious... since the files themselves are hosted in a Google Cloud Bucket, could I run my own web app somewhere on storage.googleapis.com that could use fetch() to retrieve that data, working around the lack of open CORS headers?

I got Claude Code to try that for me (I didn't want to have to figure out how to create a bucket and configure it for web access just for this one experiment) and it built and then deployed https://storage.googleapis.com/rebuild-ui/index.html, which did indeed work!

It lets you search against that list of packages from the Gist and then select one to view the pretty-printed newline-delimited JSON that was stored for that package.

The output isn't as interesting as I was expecting, but it was fun demonstrating that it's possible to build and deploy web apps to Google Cloud that can then make fetch() requests to other public buckets.

Hopefully the OSS Rebuild team will add a web UI to their project at some point in the future.

Via Hacker News

Tags: google, packaging, pypi, security, npm, ai, generative-ai, llms, ai-assisted-programming, supply-chain, vibe-coding, claude-code

Prompts.js

2024-12-07T20:35:26+00:00

I've been putting the new o1 model from OpenAI through its paces, in particular for code. I'm very impressed - it feels like it's giving me a similar code quality to Claude 3.5 Sonnet, at least for Python and JavaScript and Bash... but it's returning output noticeably faster.

I decided to try building a library I've had in mind for a while - an await ... based alternative implementation of the browser's built-in alert(), confirm() and prompt() functions.

Short version: it lets you do this:

await Prompts.alert(
    "This is an alert message!"
);

const confirmedBoolean = await Prompts.confirm(
    "Are you sure you want to proceed?"
);

const nameString = await Prompts.prompt(
    "Please enter your name"
);

Here's the source code and a a live demo where you can try it out:

I think there's something really interesting about using await in this way.

In the past every time I've used it in Python or JavaScript I've had an expectation that the thing I'm awaiting is going to return as quickly as possible - that I'm really just using this as a performance hack to unblock the event loop and allow it to do something else while I'm waiting for an operation to complete.

That's not actually necessary at all! There's no reason not to use await for operations that could take a long time to complete, such as a user interacting with a modal dialog.

Having LLMs around to help prototype this kind of library idea is really fun. This is another example of something I probably wouldn't have bothered exploring without a model to do most of the code writing work for me.

I didn't quite get it with a single prompt, but after a little bit of back-and-forth with o1 I got what I wanted - the main thing missing at first was sensible keyboard support (in particular the Enter and Escape keys).

My opening prompt was the following:

Write me a JavaScript library - no extra dependencies - which gives me the following functions:

await Prompts.alert("hi there"); -> displays a modal with a message and waits for you to click OK on it
await Prompts.confirm("Are you sure") -> an OK and cancel option, returns true or false<br>
await Prompts.prompt("What is your name?") -> a form asking the user's name, an OK button and cancel - if cancel returns null otherwise returns a string

These are equivalent to the browser builtin alert() and confirm() and prompt() - but I want them to work as async functions and to implement their own thing where they dull out the screen and show as a nicely styled modal

All CSS should be set by the Javascript, trying to avoid risk of existing CSS interfering with it

Here's the full shared ChatGPT/o1 transcript.

I then got Google's new gemini-exp-1206 model to write the first draft of the README, this time via my LLM tool:

cat index.js | llm -m gemini-exp-1206 -s \
  'write a readme for this suitable for display on npm'

Here's the response. I ended up editing this quite a bit.

I published the result to npm as prompts-js, partly to exercise those muscles again - this is only the second package I've ever published there (the first was a Web Component).

This means it's available via CDNs such as jsDelivr - so you can load it into a page and start using it like this:

<script
  src="https://cdn.jsdelivr.net/npm/prompts-js"
></script>

I haven't yet figured out how to get it working as an ES module - there's an open issue for that here.

Update: 0.0.3 switches to dialog.showModal()

I got some excellent feedback on Mastodon and on Twitter suggesting that I improve its accessibility by switching to using the built-in browser dialog.showModal().

This was a great idea! I ran a couple of rounds more with o1 and then switched to Claude 3.5 Sonnet for one last bug fix. Here's a PR where I reviewed those changes.

I shipped that as release 0.0.3, which is now powering the demo.

I also hit this message, so I guess I won't be using o1 as often as I had hoped!

Upgrading to unlimited o1 currently costs $200/month with the new ChatGPT Pro.

Things I learned from this project

Outsourcing code like this to an LLM is a great way to get something done quickly, and for me often means the difference between doing a project versus not bothering at all.

Paying attention to what the model is writing - and then iterating on it, spotting bugs and generally trying to knock it into shape - is also a great way to learn new tricks.

Here are some of the things I've learned from working on Prompts.js so far:

The const name = await askUserSomething() pattern really does work, and it feels great. I love the idea of being able to await a potentially lengthy user interaction like this.
HTML <dialog> elements are usable across multiple browsers now.
Using a <dialog> means you can skip implementing an overlay that dims out the rest of the screen yourself - that will happen automatically.
A <dialog> also does the right thing with respect to accessibility and preventing keyboard access to other elements on the page while that dialog is open.
If you set <form method="dialog"> in a form inside a dialog, submitting that form will close the dialog automatically.
The dialog.returnValue will be set to the value of the button used to submit the form.
I also learned how to create a no-dependency, no build-step single file NPM package and how to ship that to NPM automatically using GitHub Actions and GitHub Releases. I wrote that up in this TIL: Publishing a simple client-side JavaScript package to npm with GitHub Actions.

Tags: javascript, projects, releases, npm, openai, llms, ai-assisted-programming, llm, gemini, claude-3-5-sonnet, o1

Importing a frontend Javascript library without a build system

2024-11-23T19:18:20+00:00

Importing a frontend Javascript library without a build system

I sometimes think the hardest problem in computer science right now is taking an NPM library and figuring out how to download it and use it from a <script> tag without needing to involve some sort of convoluted build system.

Julia Evans shares my preference for build-free JavaScript, and has shared notes about figuring out how to turn an arbitrary NPM package into something that can be loaded in a browser.

It's so complicated! This is the best exploration I've seen yet of the topic but wow, this really needs to be easier.

My download-esm tool gets a mention, but I have to admit I'm not 100% confident in that as a robust solution. I don't know nearly enough about the full scope of the problem here to confidently recommend my own tool!

Right now my ideal solution would turn almost anything from NPM into an ES module that I can self-host and then load using import ... from in a <script type="module"> block, maybe with an importmap as long as I don't have to think too hard about what to put in it.

I'm intrigued by esm.sh (mentioned by Julia as a new solution worth exploring). The length of the documentation on that page further reinforces quite how much there is that I need to understand here.

Tags: javascript, npm, julia-evans

Announcing Deno 2

2024-10-10T04:11:02+00:00

Announcing Deno 2

The big focus of Deno 2 is compatibility with the existing Node.js and npm ecosystem:

Deno 2 takes all of the features developers love about Deno 1.x — zero-config, all-in-one toolchain for JavaScript and TypeScript development, web standard API support, secure by default — and makes it fully backwards compatible with Node and npm (in ESM).

The npm support is documented here. You can write a script like this:

import * as emoji from "npm:node-emoji";
console.log(emoji.emojify(`:sauropod: :heart:  npm`));

And when you run it Deno will automatically fetch and cache the required dependencies:

deno run main.js

Another new feature that caught my eye was this:

deno jupyter now supports outputting images, graphs, and HTML

Deno has apparently shipped with a Jupyter notebook kernel for a while, and it's had a major upgrade in this release.

Here's Ryan Dahl's demo of the new notebook support in his Deno 2 release video.

I tried this out myself, and it's really neat. First you need to install the kernel:

deno juptyer --install

I was curious to find out what this actually did, so I dug around in the code and then further in the Rust runtimed dependency. It turns out installing Jupyter kernels, at least on macOS, involves creating a directory in ~/Library/Jupyter/kernels/deno and writing a kernel.json file containing the following:

{
  "argv": [
    "/opt/homebrew/bin/deno",
    "jupyter",
    "--kernel",
    "--conn",
    "{connection_file}"
  ],
  "display_name": "Deno",
  "language": "typescript"
}

That file is picked up by any Jupyter servers running on your machine, and tells them to run deno jupyter --kernel ... to start a kernel.

I started Jupyter like this:

jupyter-notebook /tmp

Then started a new notebook, selected the Deno kernel and it worked as advertised:

import * as Plot from "npm:@observablehq/plot";
import { document, penguins } from "jsr:@ry/jupyter-helper";
let p = await penguins();

Plot.plot({
  marks: [
    Plot.dot(p.toRecords(), {
      x: "culmen_depth_mm",
      y: "culmen_length_mm",
      fill: "species",
    }),
  ],
  document,
});

Tags: javascript, nodejs, npm, jupyter, typescript, deno, observable-plot

npm install everything, and the complete and utter chaos that follows

2024-03-16T05:18:51+00:00

npm install everything, and the complete and utter chaos that follows

Here’s an experiment which went really badly wrong: a team of mostly-students decided to see if it was possible to install every package from npm (all 2.5 million of them) on the same machine. As part of that experiment they created and published their own npm package that depended on every other package in the registry.

Unfortunately, in response to the leftpad incident a few years ago npm had introduced a policy that a package cannot be removed from the registry if there exists at least one other package that lists it as a dependency. The new “everything” package inadvertently prevented all 2.5m packages—including many that had no other dependencies—from ever being removed!

Via lobste.rs

Tags: packaging, security, npm

Observable Framework 1.1

2024-03-05T21:12:48+00:00

Observable Framework 1.1

Less than three weeks after 1.0, the 1.1 release adds a whole lot of interesting new stuff. The signature feature is self-hosted npm imports: Framework 1.0 linked out to CDN hosted copies of libraries, but 1.1 fetches copies locally and then bundles that code with the deployed static site.

This works by using the acorn JavaScript parsing library to statically analyze the code and find all of the relevant imports.

Via @mbostock

Tags: javascript, npm, observable, mike-bostock, observable-framework

NPM: modele-social

2024-01-02T17:55:59+00:00

NPM: modele-social

This is a fascinating open source package: it’s an NPM module containing an implementation of the rules for calculating social security contributions in France, maintained by a team at Urssaf, the not-quite-government organization in France that manages the collection of social security contributions there.

The rules themselves can be found in the associated GitHub repository, encoded in a YAML-like declarative language called Publicodes that was developed by the French government for this and similar purposes.

Via Hacker News comment

Tags: government, open-source, npm

Deno 1.34: deno compile supports npm packages

2023-05-25T17:01:08+00:00

Deno 1.34: deno compile supports npm packages

This feels like it could be extremely useful: Deno can load code from npm these days (import { say } from "npm:cowsay@1.5.0") and now the deno compile command can resolve those imports, fetch all of the dependencies and bundle them together with Deno itself into a single executable binary. This means pretty much anything that's been built as an npm package can now be easily converted into a standalone binary, including cross-compilation to Windows x64, macOS x64, macOS ARM and Linux x64.

Tags: javascript, npm, deno

download-esm: a tool for downloading ECMAScript modules

2023-05-02T04:47:29+00:00

I've built a new CLI tool, download-esm, which takes the name of an npm package and will attempt to download the ECMAScript module version of that package, plus all of its dependencies, directly from the jsDelivr CDN - and then rewrite all of the import statements to point to those local copies.

Why I built this

I have somewhat unconventional tastes when it comes to JavaScript.

I really, really dislike having to use a local build script when I'm working with JavaScript code. I've tried plenty, and inevitably I find that six months later I return to the project and stuff doesn't work any more - dependencies need updating, or my Node.js is out of date, or the build tool I'm using has gone out of fashion.

Julia Evans captured how I feel about this really clearly in Writing Javascript without a build system.

I just want to drop some .js files into a directory, load them into an HTML file and start writing code.

Working the way I want to work is becoming increasingly difficult over time. Many modern JavaScript packages assume you'll be using npm and a set of build tools, and their documentation gets as far as npm install package and then moves on to more exciting things.

Some tools do offer a second option: a CDN link. This is great, and almost what I want... but when I'm building software for other people (Datasette plugins for example) I like to include the JavaScript dependencies in my installable package, rather than depending on a CDN staying available at that URL forever more.

This is a key point: I don't want to depend on a fixed CDN. If you're happy using a CDN then download-esm is not a useful tool for you.

Usually, that CDN link is enough: I can download the .js file from the CDN, stash it in my own directory and get on with my project.

This is getting increasingly difficult now, thanks to the growing popularity of ECMAScript modules.

ECMAScript modules

I love the general idea of ECMAScript modules, which have been supported by all of the major browsers for a few years now.

If you're not familiar with them, they let you do things like this (example from the Observable Plot getting started guide):

<div id="myplot"></div>
<script type="module">
import * as Plot from "https://cdn.jsdelivr.net/npm/@observablehq/plot@0.6/+esm";

const plot = Plot.rectY(
    {length: 10000},
    Plot.binX(
        {y: "count"},
        {x: Math.random}
    )
).plot();
const div = document.querySelector("#myplot");
div.append(plot);
</script>

This is beautiful. You can import code on-demand, which makes lazy loading easier. Modules can themselves import other modules, and the browser will download them in parallel over HTTP/2 and cache them for future use.

There's one big catch here: downloading these files from the CDN and storing them locally is surprisingly fiddly.

Observable Plot for example has 40 nested dependency modules. And downloading all 40 isn't enough, because most of those modules include their own references that look like this:

export*from"/npm/d3-array@3.2.3/+esm";
export*from"/npm/d3-axis@3.0.0/+esm";

These references all need to be rewritten to point to the local copies of the modules.

Inspiration from Observable Plot

I opened an issue on the Observable Plot repository: Getting started documentation request: Vanilla JS with no CDN.

An hour later Mike Bostock committed a fix linking to UMB bundles for d3.js and plot3.js - which is a good solution, but doesn't let me import them as modules. But he also posted this intriguing comment:

I think maybe the answer here is that someone should write a “downloader” tool that downloads the compiled ES modules from jsDelivr (or other CDN) and rewrites the import statements to use relative paths. Then you could just download this URL

https://cdn.jsdelivr.net/npm/@observablehq/plot/+esm

and you’d get the direct dependencies

https://cdn.jsdelivr.net/npm/d3@7.8.4/+esm https://cdn.jsdelivr.net/npm/isoformat@0.2.1/+esm https://cdn.jsdelivr.net/npm/interval-tree-1d@1.0.4/+esm

and the transitive dependencies and so on as separate files.

So I built that!

download-esm

The new tool I've built is called download-esm. You can install it using pip install download-esm, or pipx install download-esm, or even rye install download-esm if that's your new installation tool of choice.

Once installed, you can attempt to download the ECMAScript module version of any npm package - plus its dependencies - like this:

download-esm @observablehq/plot plot/

This will download the module versions of every file, rewrite their imports and save them in the plot/ directory.

When I run the above I get the following from ls plot/:

binary-search-bounds-2-0-5.js
d3-7-8-4.js
d3-array-3-2-0.js
d3-array-3-2-1.js
d3-array-3-2-3.js
d3-axis-3-0-0.js
d3-brush-3-0-0.js
d3-chord-3-0-1.js
d3-color-3-1-0.js
d3-contour-4-0-2.js
d3-delaunay-6-0-4.js
d3-dispatch-3-0-1.js
d3-drag-3-0-0.js
d3-dsv-3-0-1.js
d3-ease-3-0-1.js
d3-fetch-3-0-1.js
d3-force-3-0-0.js
d3-format-3-1-0.js
d3-geo-3-1-0.js
d3-hierarchy-3-1-2.js
d3-interpolate-3-0-1.js
d3-path-3-1-0.js
d3-polygon-3-0-1.js
d3-quadtree-3-0-1.js
d3-random-3-0-1.js
d3-scale-4-0-2.js
d3-scale-chromatic-3-0-0.js
d3-selection-3-0-0.js
d3-shape-3-2-0.js
d3-time-3-1-0.js
d3-time-format-4-1-0.js
d3-timer-3-0-1.js
d3-transition-3-0-1.js
d3-zoom-3-0-0.js
delaunator-5-0-0.js
internmap-2-0-3.js
interval-tree-1d-1-0-4.js
isoformat-0-2-1.js
observablehq-plot-0-6-6.js
robust-predicates-3-0-1.js

Then to use Observable Plot you can put this in an index.html file in the same directory:

<div id="myplot"></div>
<script type="module">
import * as Plot from "./observablehq-plot-0-6-6.js";
const plot = Plot.rectY(
    {length: 10000}, Plot.binX({y: "count"}, {x: Math.random})
).plot();
const div = document.querySelector("#myplot");
div.append(plot);
</script>

Then run python3 -m http.server to start a server on port 8000 (ECMAScript modules don't work directly from opening files), and open http://localhost:8000/ in your browser.

How it works

There's honestly not a lot to this. It's 100 lines of Python in this file - most of the work is done by some regular expressions, which were themselves mostly written by ChatGPT.

I shipped the first alpha release as soon as it could get Observable Plot working, because that was my initial reason for creating the project.

I have an open issue inviting people to help test it with other packages. That issue includes my own comments of stuff I've tried with it so far.

So far I've successfully used it for preact and htm, for codemirror and partially for monaco-editor - though Monaco breaks when you attempt to enable syntax highlighting, as it attempts to dynamically load additional modules from the wrong place.

Your help needed

It seems very unlikely to me that no-one has solved this problem - I would be delighted if I could retire download-esm in favour of some other solution.

If this tool does turn out to fill a new niche, I'd love to make it more robust. I'm not a frequent JavaScript developer so I'm certain there are all sorts of edge-cases and capabilities I haven't thought of.

Contributions welcome!

Tags: cli, ecmascript, javascript, projects, npm, ai-assisted-programming

Making SQLite extensions npm install'able for Node.js, and on deno.land/x for Deno

2023-03-29T22:13:21+00:00

Making SQLite extensions npm install'able for Node.js, and on deno.land/x for Deno

Alex Garcia figured out how to get his “pip install X” trick for distributing compiled SQLite extensions to work for Node too! Now you can “npm install” 10 of his extensions, including sqlite-regex and sqlite-xsv and sqlite-http and sqlite-html and more, and attach them to a node-sqlite3 or better-sqlite3 connection. He’s bundled them for Deno too!

Via Datasette Discord

Tags: nodejs, sqlite, npm, deno, alex-garcia

Bun

2022-07-06T17:24:13+00:00

Bun

“Bun is a fast all-in-one JavaScript runtime”—this is very interesting. It’s the first project I’ve seen written using the Zig language, which I see as somewhat equivalent to Rust. Bun provides a full Node.js-style JavaScript environment plus a host of packaged tools—an npm install client, a TypeScript transpiler, bundling tools—all wrapped up in a single binary. The JavaScript engine itself extends JavaScriptCore. Bun also ships with its own wrapper for SQLite.

Tags: javascript, sqlite, npm, typescript, zig, bun

Weeknotes: Shaving some beautiful yaks

2021-12-01T03:43:18+00:00

I've been mostly shaving yaks this week - two in particular: the Datasette table refactor and the next release of git-history. I also built and released my first Web Component!

A Web Component for embedding Datasette tables

A longer term goal that I have for Datasette is to figure out a good way of using it to build dashboards, tying together summaries and visualizations of the latest data from a bunch of different sources.

I'm excited about the potential of Web Components to help solve this problem.

My datasette-notebook project is a very early experiment in this direction: it's a Datasette notebook that provides a Markdown wiki (persisted to SQLite) to which I plan to add the ability to embed tables and visualizations in wiki pages - forming a hybrid of a wiki, dashboarding system and Notion/Airtable-style database.

It does almost none of those things right now, which is why I've not really talked about it here.

Web Components offer a standards-based mechanism for creating custom HTML tags. Imagine being able to embed a Datasette table on a page by adding the following to your HTML:

<datasette-table
    url="https://global-power-plants.datasettes.com/global-power-plants/global-power-plants.json"
></datasette-table>

That's exactly what datasette-table lets you do! Here's a demo of it in action.

This is version 0.1.0 - it works, but I've not even started to flesh it out.

I did learn a bunch of things building it though: it's my first Web Component, my first time using Lit, my first time using Vite and the first JavaScript library I've ever packaged and published to npm.

Here's a detailed TIL on Publishing a Web Component to npm encapsulating everything I've learned from this project so far.

This is also my first piece of yak shaving this week: I built this partly to make progress on datasette-notebook, but also because my big Datasette refactor involves finalizing the design of the JSON API for version 1.0. I realized that I don't actually have a project that makes full use of that API, which has been hindering my attempts to redesign it. Having one or more Web Components that consume the API will be a fantastic way for me to eat my own dog food.

Link: rel="alternate" for Datasette tables

Here's an interesting problem that came up while I was working on the datasette-table component.

As designed right now, you need to figure out the JSON URL for a table and pass that to the component.

This is usually a case of adding .json to the path, while preserving any query string parameters - but there's a nasty edge-case: if your SQLite table itself ends with the string .json (which could happen! Especially since Datasette promises to work with any existing SQLite database) the URL becomes this instead:

/mydb/table.json?_format=json

Telling users of my component that they need to first construct the JSON URL for their page isn't the best experience: I'd much rather let people paste in the URL to the HTML version and derive the JSON from that.

This is made more complex by the fact that, thanks to --cors, the Web Component can be embedded on any page. And for datasette-notebook I'd like to provide a feature where any URLs to Datasette instances - no matter where they are hosted - are turned into embedded tables automatically.

To do this, I need an efficient way to tell that an arbitrary URL corresponds to a Datasette table.

My latest idea here is to use a combination of HTTP HEAD requests and a Link: rel="alternate" header - something like this:

~ % curl -I 'https://latest.datasette.io/fixtures/compound_three_primary_keys'
HTTP/1.1 200 OK
date: Sat, 27 Nov 2021 20:09:36 GMT
server: uvicorn
Link: https://latest.datasette.io/fixtures/compound_three_primary_keys.json; rel="alternate"; type="application/datasette+json"

This would allow a (hopefully fast) fetch() call from JavaScript to confirm that a URL is a Datasette table, and get back the JSON that should be fetched by the component in order to render it on the page.

I have a prototype of this in Datasette issue #1533. I think it's a promising approach!

It's also now part of the ever-growing table refactor. Adding custom headers to page responses is currently far harder than it should be.

sqlite-utils STRICT tables

SQLite 3.37.0 came out at the weekend with a long-awaited feature: STRICT tables, which enforce column types such that you get an error if you try to insert a string into an integer column.

(This has been a long-standing complaint about SQLite by people who love strong typing, and D. Richard Hipp finally shipped the change for them with some salty release notes saying it's "for developers who prefer that kind of thing.")

I started researching how to add support for this to my sqlite-utils Python library. You can follow my thinking in sqlite-utils issue #344 - I'm planning to add a strict=True option to methods that create tables, but for the moment I've shipped new introspection properties for seeing if a table uses strict mode or not.

git-history update

My other big yak this week has been work on git-history. I'm determined to get it into a stable state such that I can write it up, produce a tutorial and maybe produce a video demonstration as well - but I keep on finding things I want to change about how it works.

The big challenge is how to most effectively represent the history of a bunch of different items over time in a relational database schema.

I started with a item table that presents just the most recent version of each item, and an item_version table with a row for every subsequent version.

That table got pretty big, with vast amounts of duplicated data in it.

So I've been working on an optimization where columns are only included in an item_version row if they have changed since the previous version.

The problem there is what to do about null - does null mean "this column didn't change" or does it mean "this column was set from some other value back to null"?

After a few different attempts I've decided to solve this with a many-to-many table, so for any row in the item_version table you can see which columns were explicitly changed by that version.

This is all working pretty nicely now, but still needs documentation, and tests, and then a solid write-up and tutorial and demos and a video... hopefully tomorrow!

One of my design decisions for this tool has been to use an underscore prefix for "reserved columns", such that non-reserved columns can be safely used by the arbitrary data that is being tracked by the tool.

Having columns with names like _id and _item has highlighted several bugs with Datasette's handling of these column names, since Datasette itself tries to use things like ?_search= for special query string parameters. I released Datasette 0.59.4 with some relevant fixes.

A beautiful yak

As a consumate yak shaver this beautiful yak that showed up on Reddit a few weeks ago has me absolutely delighted. I've not been able to determine the photography credit.

Releases this week

s3-credentials: 0.7 - (7 releases total) - 2021-11-30
A tool for creating credentials for accessing S3 buckets
datasette: 0.59.4 - (102 releases total) - 2021-11-30
An open source multi-tool for exploring and publishing data
datasette-table: 0.1.0 - 2021-11-28
A Web Component for embedding a Datasette table on a page

TIL this week

Tags: projects, npm, datasette, web-components, weeknotes, git-scraping

Dependency Confusion: How I Hacked Into Apple, Microsoft and Dozens of Other Companies

2021-02-10T20:42:06+00:00

Dependency Confusion: How I Hacked Into Apple, Microsoft and Dozens of Other Companies

Alex Birsan describes a new category of security vulnerability he discovered in the npm, pip and gem packaging ecosystems: if a company uses a private repository with internal package names, uploading a package with the same name to the public repository can often result in an attacker being able to execute their own code inside the networks of their target. Alex scored over $130,000 in bug bounties from this one, from a number of name-brand companies. Of particular note for Python developers: the --extra-index-url argument to pip will consult both public and private registries and install the package with the highest version number!

Via @jacobian

Tags: pip, python, security, npm

Two malicious Python libraries caught stealing SSH and GPG keys

2019-12-05T06:07:54+00:00

Two malicious Python libraries caught stealing SSH and GPG keys

Nasty. Two typosquatting libraries were spotted on PyPI—targetting dateutil and jellyfish but with tricky variants of their names. They attempted to exfiltrate SSH and GPG keys and send them to an IP address defined server. npm has seen this kind of activity too—it’s important to consider this when installing packages.

Tags: pypi, security, npm

Quoting Laurie Voss

2018-12-04T02:02:31+00:00

npm users have downloaded more than 489 billion packages in the 9 year life of the project, with the strange effect of exponential growth being that 286 billion, or 58% of those, were just in the last year.

— Laurie Voss

Tags: npm, laurie-voss

Quoting Sébastien Cevey

2018-12-02T21:32:34+00:00

The nature of NPM is such that I'd expect most large corporate Node software to depend on at least a couple of single individuals' hobby projects. The problem is that those projects don't tend to fulfill the same expectations of security, quality and maintenance.

— Sébastien Cevey

Tags: javascript, packaging, npm

source-map-explorer

2018-06-24T21:37:38+00:00

source-map-explorer

Very neat tool for creating a tree map visualization of the size of the components of a bundled JavaScript file created by webpack (or if you’re using create-react-app by “npm run build”). I ran this using “npx source-map-explorer build/static/js/main.d63f3f34.js” (since I don’t like using “npm install -g”).

Tags: javascript, npm

Incident report: npm

2018-01-11T17:27:25+00:00

Incident report: npm

Fascinating insight into the challenges involved in managing a massive scale community code repository. An algorithm incorrectly labeled a legit user as spam, an NPM staff member acted on the report, dependent package installations started failing and because the package had been removed as spam other users were able to try and fix the bug by publishing fresh copies of the missing package to the same namespace.

Tags: security, spammers, npm

Quoting Laurie Voss

2017-10-20T14:54:22+00:00

Serverless is a somewhat unhelpfully misleading term for "highly scalable stateless code". All the times I've seen serverless stuff work really well it was workloads that were usually zero but occasionally 30k/sec without warning. I've run a company with that kind of workload and serverless stuff would have saved us a ton of money. Publishing to the [npm] registry could be done as a serverless app but there's little benefit because we do not get huge spikes in publishing. We get huge spikes in downloads but serverless isn't useful there because it's a read-only case and very little processing is done. Serverless is a great solution to one type of problem. It's very seldom the case that you can convert all your problems into that shape.

— Laurie Voss

Tags: npm, serverless, laurie-voss

Does Go have an equivalent node.js's NPM?

2014-01-01T13:22:00+00:00

My answer to Does Go have an equivalent node.js's NPM? on Quora

Sort of. Go ships with a command that can download and compile a dependency for your project ("go get github.com/russross/blackfriday") but it doesn't have a solution for library versioning yet (as far as I can tell).

http://golang.org/doc/code.html is a useful guide to how Go's development workflow works.

Tags: go, nodejs, quora, npm