Simon Willison's Weblog: vicuna

A practical guide to deploying Large Language Models Cheap, Good and Fast

2023-09-04T13:43:03+00:00

A practical guide to deploying Large Language Models Cheap, Good *and* Fast

Joel Kang’s extremely comprehensive notes on what he learned trying to run Vicuna-13B-v1.5 on an affordable cloud GPU server (a T4 at $0.615/hour). The space is in so much flux right now—Joel ended up using MLC but the best option could change any minute.

Vicuna 13B quantized to 4-bit integers needed 7.5GB of the T4’s 16GB of VRAM, and returned tokens at 20/second.

An open challenge running MLC right now is around batching and concurrency: “I did try making 3 concurrent requests to the endpoint, and while they all stream tokens back and the server doesn’t OOM, the output of all 3 streams seem to actually belong to a single prompt.”

Tags: ai, generative-ai, llama, llms, mlc, vicuna

Let's be bear or bunny

2023-05-01T18:37:20+00:00

The Machine Learning Compilation group (MLC) are my favourite team of AI researchers at the moment.

In less than two months they've released:

Web Stable Diffusion - the full Stable Diffusion image generation model running entirely in a browser, after a 2GB download, powered by WebGPU. Try it here using Chrome Canary.
Web LLM which pulls off the same trick but for vicuna-7b-v1, a genuinely useful LLaMA-derived ChatGPT-style Large Language Model! I wrote more about that in Web LLM runs the vicuna-7b Large Language Model entirely in your browser, and it’s very impressive.

And then a few days ago they released MLC LLM - "a universal solution that allows any language model to be deployed natively on a diverse set of hardware backends and native applications".

As part of that release, they put up a TestFlight preview of an iPhone app that runs the LLM model locally on your phone. It needs 4GB of RAM just for the application, so a iPhone 12 Pro or later should work.

I've been running it on my phone. It totally works!

(This is actually the second LLM I've been able to run on my phone - I previously also tried Sheepy-T which runs the apparently less capable GPT-J.)

Since I've been writing online for more than twenty years I often try an ego search first, to see if the weird bundle of binary floating point numbers has picked up anything about me. Here's what I got from the MLC LLM on my phone.

I could feel the back of my phone heating up as it slowly calculated all of those tokens!

Honestly, this is wildly better than I expected. I'm from the UK (but based in California since 2014). I wouldn't call myself an "organizer" and while I care deeply about accessibility I'm not sure I'd credit myself as an outspoken advocate, but I've definitely advocated for web standards in the past.

But what on earth is that "Let's be bear or bunny" pattern?

So I asked:

I think this is my favourite AI hallucination yet!

Here's that definition, which I'm going to claim is co-written by myself and my hard working iPhone:

The "Let's be bear or bunny" pattern is a way of looking at situations and problems in a certain way. It is a way of approaching a problem or situation with a sense of playfulness and lightheartedness, while still being able to think critically and analytically.

It is also a way of embracing the unexpected and being open to new possibilities. This pattern is often associated with the concept of "go with the flow" or "roll with the punches" in the face of challenges and difficulties. It encourages an attitude of relaxation and ease in the face of obstacles and difficulties

Honestly, I kind of love this. I've been trying out saying "Let's be bear or bunny that!" at home. Natalie has hinted that this may have already played itself out, but I disagree.

Tags: ai, generative-ai, llama, local-llms, llms, mlc, vicuna, hallucinations

LLaVA: Large Language and Vision Assistant

2023-04-19T01:14:37+00:00

LLaVA: Large Language and Vision Assistant

Yet another multi-modal model combining a vision model (pre-trained CLIP ViT-L/14) and a LLaMA derivative model (Vicuna). The results I get from their demo are even more impressive than MiniGPT-4. Also includes a new training dataset, LLaVA-Instruct-150K, derived from GPT-4 and subject to the same warnings about the OpenAI terms of service.

Via Hacker News

Tags: computer-vision, ai, generative-ai, llama, llms, vicuna

MiniGPT-4

2023-04-17T14:21:40+00:00

MiniGPT-4

An incredible project with a poorly chosen name. A team from King Abdullah University of Science and Technology in Saudi Arabia combined Vicuna-13B (a model fine-tuned on top of Facebook’s LLaMA) with the BLIP-2 vision-language model to create a model that can conduct ChatGPT-style conversations around an uploaded image. The demo is very impressive, and the weights are available to download—45MB for MiniGPT-4, but you’ll need the much larger Vicuna and LLaMA weights as well.

Via Hacker News

Tags: computer-vision, ai, generative-ai, local-llms, llms, vicuna

Web LLM runs the vicuna-7b Large Language Model entirely in your browser, and it's very impressive

2023-04-16T15:10:45+00:00

A month ago I asked Could you train a ChatGPT-beating model for $85,000 and run it in a browser?. $85,000 was a hypothetical training cost for LLaMA 7B plus Stanford Alpaca. "Run it in a browser" was based on the fact that Web Stable Diffusion runs a 1.9GB Stable Diffusion model in a browser, so maybe it's not such a big leap to run a small Large Language Model there as well.

That second part has now happened.

Web LLM is a project from the same team as Web Stable Diffusion which runs the vicuna-7b-delta-v0 model in a browser, taking advantage of the brand new WebGPU API that just arrived in Chrome in beta.

I got their browser demo running on my M2 MacBook Pro using Chrome Canary.

Update: As of Chrome 113 released in May 2023 you no longer need to use Chrome Canary for this - regular Chrome works fine too.

It's really, really good. It's actually the most impressive Large Language Model I've run on my own hardware to date - and the fact that it's running entirely in the browser makes that even more impressive.

It's really fast too: I'm seeing around 15 tokens a second, which is better performance than almost all of the other models I've tried running on my own machine.

I started it out with something easy - a straight factual lookup. "Who landed on the moon?"

That's a good answer, and it passes a quick fact check.

Next, I tried something a lot harder: "five albums by Cher as a markdown list"

It managed to count to five, which is no easy thing for an LLM. It also appears to know what a Markdown list looks like.

But... www.cherproject.com is a hallucinated domain name, and two of those albums appear to be wrong to me - "Cher's Gold" should be "Cher's Golden Greats", and I while Cher did sign with Geffen Records I couldn't find any mention anywhere of an album called "Greatest Hits: Geffen Years".

I did not expect it to be able to handle this prompt at all though, so I'm still very impressed to see even a partially correct answer here.

I decided to see if it knew who I am. "Who is Simon Willison?"

It answered "Human: Who is peanut?". Zero marks for that one.

I decided to try it on a summary. I copied some random paragraphs of text from a recent blog entry and asked it to "Summarize this: PASTE".

It did a very, very good job!

At this point I started to get excited.

As I've noted before, I don't particularly care about having a locally executing LLM that can answer questions about every factual topic under the sun.

What I want instead is a calculator for words. I want a model that I can feed content into and have it manipulate the language in that input - summarization, fact extraction, question answering based on a carefully crafted prompt - that kind of thing.

If Web LLM + vicuna-7b-delta-v0 can summarize text like this, it's looking like it might be the level of capability I've been hoping for.

Time to try one of my favourite tests for an LLM: can it generate pun names for a coffee shop run by otters?

(It actually returned 54, I'm listing just the first 20 here.)

Are these brilliant puns? No. But they're recognizable as puns! This was honestly far beyond my wildest dreams for what I might get out of an LLM that can run in a browser.

Just to see what happened, I threw what I thought would be an impossible prompt at it: "A rap battle between a pelican and a sea otter".

Wow. I mean it's bad, but it's also amazing.

How about writing code? I tried "Write a JavaScript function to extract data from a table and log it to the console as CSV"

${header.toUpperCase()}).join(','); const parsedData = rowString.split(','); parsedData.forEach(dataElement => { const { tag, value } = dataElement.split(':'); data.push(value); }); } // Log data as CSV data.forEach(dataElement => console.log(`${dataElement.tag},${dataElement.value}`)); }" style="max-width: 100%;" />

This looks convincing at first glance, but it's useless: table.headers.split(",") is not how an HTML table works in the JavaScript DOM.

Again though, this result hints in a very useful direction - particularly for something that's small enough to run in my browser.

Is this enough to be useful?

Despite the flaws demonstrated above, I think this has passed my threshold for being something I could use as a building block for all sorts of genuinely useful things.

I don't need a language model that can answer any question I have about the world from its baked in training data.

I need something that can manipulate language in useful ways. I care about summarization, and fact extraction, and answering questions about larger text.

(And maybe inventing pun names for coffee shops.)

The most useful innovation happening around language models right now involves giving them access to tools.

It turns out it's really easy to teach a language model how to turn "Summarize my latest email" into a command, 'action: fetch_latest_email' which can then be carried out by an outer layer of code, with the results being fed back into the model for further processing.

One popular version of this is the ReAct model, which I implemented in a few dozen lines of Python here. ChatGPT Plugins and Auto-GPT are more examples of this pattern in action.

You don't need a model with the power of GPT-4 to implement this pattern. I fully expect that vicuna-7b is capable enough to get this kind of thing to work.

An LLM that runs on my own hardware - that runs in my browser! - and can make use of additional tools that I grant to it is a very exciting thing.

Here's another thing everyone wants: a LLM-powered chatbot that can answer questions against their own documentation.

I wrote about a way of doing that in How to implement Q&A against your documentation with GPT3, embeddings and Datasette. I think vicuna-7b is powerful enough to implement that pattern, too.

Why the browser matters

Running in the browser feels like a little bit of a gimmick - especially since it has to pull down GBs of model data in order to start running.

I think the browser is actually a really great place to run an LLM, because it provides a secure sandbox.

LLMs are inherently risky technology. Not because they might break out and try to kill all humans - that remains pure science fiction. They're dangerous because they will follow instructions no matter where those instructions came from. Ask your LLM assistant to summarize the wrong web page and an attacker could trick it into leaking all your private data, or deleting all of your emails, or worse.

I wrote about this at length in Prompt injection: what’s the worst that can happen? - using personal AI assistants as an explicit example of why this is so dangerous.

To run personal AI assistants safely, we need to use a sandbox where we can carefully control what information and tools they have available to then.

Web browsers are the most robustly tested sandboxes we have ever built.

Some of the challenges the browser sandbox can help with include:

Using CORS and Content-Security-Policy as an additional layer of security controlling which HTTP APIs an assistant is allowed to access
Want your assistant to generate and then execute code? WebAssembly sandboxes - supported in all mainstream browsers for several years at this point - are a robust way to do that.

It's possible to solve these problems outside of the browser too, but the browser provides us with some very robust primitives to help along the way.

Vicuna isn't openly licensed

The Vicuna model card explains how the underlying model works:

Vicuna is an open-source chatbot trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.

This isn't ideal. Facebook LLaMA is licensed for non-commercial and research purposes only. ShareGPT is a site where people share their ChatGPT transcripts, which means the fine-tuning was conducted using data that isn't licensed for such purposes (the OpenAI terms and condition disallow using the data to train rival language models.)

So there are severe limits on what you could build on top of this project.

But, as with LLaMA and Alpaca before it, the exciting thing about this project is what it demonstrates: we can now run an extremely capable LLM entirely in a browser - albeit with a beta browser release, and on a very powerful laptop.

The next milestone to look forward to is going to be a fully openly licensed LLM - something along the lines of Dolly 2 - running entirely in the browser using a similar stack to this Web LLM demo.

The OpenAssistant project is worth watching here too: they've been crowdsourcing large amounts of openly licensed fine-tuning data, and are beginning to publish their own models - mostly derived from LLaMA, but that training data will unlock a lot more possibilities.

Tags: chrome, ai, webassembly, generative-ai, llama, local-llms, llms, mlc, vicuna