Chatbots like ChatGPT, Claude.ai, and phind can be quite helpful, but you might not always want your questions or sensitive data handled by an external application. That's especially true on platforms where your interactions may be reviewed by humans and otherwise used to help train future models.
One solution is to download a large language model (LLM) and run it on your own machine. That way, an outside company never has access to your data. This is also a quick option to try some new specialty models such as Meta's recently announced Code Llama family of models, which are tuned for coding, and SeamlessM4T, aimed at text-to-speech and language translations.
Running your own LLM might sound complicated, but with the right tools, it’s surprisingly easy. And the hardware requirements for many models aren’t crazy. I've tested the options presented in this article on two systems: a Dell PC with an Intel i9 processor, 64GB of RAM, and a Nvidia GeForce 12GB GPU (which likely wasn't engaged running much of this software), and on a Mac with an M1 chip but just 16GB of RAM.
Be advised that it may take a little research to find a model that performs reasonably well for your task and runs on your desktop hardware. And, few may be as good as what you're used to with a tool like ChatGPT (especially with GPT-4) or Claude.ai. Simon Willison, creator of the command-line tool LLM, argued in a presentation last week that running a local model could be worthwhile even if its responses are wrong:
[Some of] the ones that run on your laptop will hallucinate like wild— which I think is actually a great reason to run them, because running the weak models on your laptop is a much faster way of understanding how these things work and what their limitations are.
It's also worth noting that open source models are likely to keep improving, and some industry watchers expect the gap between them and commercial leaders to narrow.
Run a local chatbot with GPT4All
If you want a chatbot that runs locally and won't send data elsewhere, GPT4All offers a desktop client for download that's quite easy to set up. It includes options for models that run on your own system, and there are versions for Windows, macOS, and Ubuntu.
When you open the GPT4All desktop application for the first time, you'll see options to download around 10 (as of this writing) models that can run locally. Among them is Llama-2-7B chat, a model from Meta AI. You can also set up OpenAI's GPT-3.5 and GPT-4 (if you have access) for non-local use if you have an API key.
The model-download portion of the GPT4All interface was a bit confusing at first. After I downloaded several models, I still saw the option to download them all. That suggested the downloads didn't work. However, when I checked the download path, the models were there.
Once the models are set up, the chatbot interface itself is clean and easy to use. Handy options include copying a chat to a clipboard and generating a response.
There's also a new beta LocalDocs plugin that lets you "chat" with your own documents locally. You can enable it in the Settings Plugins tab, where you'll see a "LocalDocs Plugin (BETA) Settings" header and an option to create a collection at a specific folder path.
The plugin is a work in progress, and documentation warns that the LLM may still "hallucinate" (make things up) even when it has access to your added expert information. Nevertheless, it's an interesting feature that's likely to improve as open source models become more capable.
In addition to the chatbot application, GPT4All also has bindings for Python, Node, and a command-line interface (CLI). There's also a server mode that lets you interact with the local LLM through an HTTP API structured very much like OpenAI's. The goal is to let you swap in a local LLM for OpenAI's by changing a couple of lines of code.
LLMs on the command line
LLM by Simon Willison is one of the easier ways I've seen to download and use open source LLMs locally on your own machine. While you do need Python installed to run it, you shouldn't need to touch any Python code. If you're on a Mac and use Homebrew, just install with
brew install llm
If you're on a Windows machine, use your favorite way of installing Python libraries, such as
pip install llm
LLM defaults to using OpenAI models, but you can use plugins to run other models locally. For example, if you install the
gpt4all plugin, you'll have access to additional local models from GPT4All. There are also plugins for llama, the MLC project, and MPT-30B, as well as additional remote models.
Install a plugin on the command line with
llm install model-name:
llm install llm-gpt4all
You can see all available models—remote and the ones you've installed, including brief info about each one, with the command:
llm models list.
To send a query to a local LLM, use the syntax:
llm -m the-model-name "Your query"
I then asked it a ChatGPT-like question without issuing a separate command to download the model:
llm -m ggml-model-gpt4all-falcon-q4_0 "Tell me a joke about computer programming"
This is one thing that makes the LLM user experience so elegant: If the GPT4All model doesn't exist on your local system, the LLM tool automatically downloads it for you before running your query. You'll see a progress bar in the terminal as the model is downloading.
The joke itself wasn't outstanding—"Why did the programmer turn off his computer? Because he wanted to see if it was still working!"—but the query did, in fact, work. And if results are disappointing, that's because of model performance or inadequate user prompting, not the LLM tool.
You can also set aliases for models within LLM, so that you can refer to them by shorter names:
llm aliases set falcon ggml-model-gpt4all-falcon-q4_0
To see all your available aliases, enter:
The LLM plugin for Meta's Llama models requires a bit more setup than GPT4All does. Read the details on the LLM plugin's GitHub repo. Note that the general-purpose
llama-2-7b-chat did manage to run on my work Mac with the M1 Pro chip and just 16GB of RAM. It ran rather slowly compared with the GPT4All models optimized for smaller machines without GPUs, and performed better on my more robust home PC.
LLM has other features, such as an
argument flag that lets you continue from a prior chat and the ability to use it within a Python script. You can see more on the LLM website. Willison, co-creator of the popular Python Django framework, hopes that others in the community will contribute more plugins to the LLM ecosystem.
Llama models on a Mac: Ollama
Ollama is an even easier way to download and run models than LLM, although it is also more limited. It is macOS-only for now, although they say support for Windows and Linux is "coming soon."
Installation is an elegant experience via point-and-click. And although Ollama is a command-line tool, there's just one command with the syntax
ollama run model-name. As with LLM, if the model isn't on your system already, it will automatically download.
You can see the list of available models at https://ollama.ai/library, which as of this writing included one version of a Code Llama model but not any of the tuned-for-Python options. That was on the day after Code Llama was announced, though, and more should be available in the future.
The Ollama GitHub repo's README includes helpful list of some model specs and advice that "You should have at least 8GB of RAM to run the 3B models, 16GB to run the 7B models, and 32GB to run the 13B models." On my 16GB RAM Mac, the 7B Code Llama performance was surprisingly snappy. It will answer questions about
Despite being the smallest model in the family, it was pretty good if imperfect at answering an R coding question that tripped up some larger models: "Write R code for a ggplot2 graph where the bars are steel blue color." The code was correct except for two extra closing parentheses in two of the lines of code, which were easy enough to spot in my IDE. I suspect the larger Code Llama could have done better.
If you're on a Mac and want to use Code Llama, you could have this running in a terminal window and pull it up every time you have a question. I'm looking forward to an Ollama Windows version to use on my home PC.
Chat with your own documents: h2oGPT
H2O.ai has been working on automated machine learning for some time, so it's natural that the company has moved into the chat LLM space. Some of its tools are best used by people with knowledge of the field, but instructions to install a test version of its h2oGPT chat desktop application were quick and straightforward, even for machine learning novices.
You can access a demo version on the web (obviously not using an LLM local to your system) at gpt.h2o.ai, which is a useful way to find out if you like the interface before downloading it onto your own system.
For a local version: Clone the GitHub repository, create and activate a Python virtual environment, and run the five lines of code found in the README file. Results give you "limited document Q/A capability" and one of Meta's Llama models, according to the documentation, but they work. You'll have a Llama model version downloaded locally and an application available at http://localhost:7860 when running a single line of code:
python generate.py --base_model='llama' --prompt_type=llama2
Without adding your own files, you can use the application as a general chatbot. Or, you can upload some documents and ask questions about those files. Compatible file formats include PDF, Excel, CSV, Word, text, markdown, and more.
The test application worked fine on my 16GB Mac, although the smaller model's results didn't compare to paid ChatGPT with GPT-4 (as always, that's a function of the model and not the application). The h2oGPT UI offers an Expert tab with a number of configuration options for users who know what they're doing. This gives more experienced users the option to try to improve their results.
If you want more control over the process and options for more models, download the complete application, although it may take more work to get a model running on limited hardware. There are installation instructions in the README for Windows, macOS, and Linux. Note that my Windows antivirus software was unhappy with the Windows version because it was both new and unsigned.
However, since I'm familiar with H2O.ai's other software and the code is available on GitHub, I was willing to download and install. As it turns out, though, since I don't have enough GPU power to run a decent-sized model, I would have been best off installing from source—at which point I'm not sure running this model is quite as easy as most of the other methods.
Rob Mulla, now at at H2O.ai, posted a YouTube video on his channel about installing the app on Linux. Although the video is a couple of months old now, and the application user interface appears to have changed, the video still has useful info, including helpful explanations about H2O.ai LLMs.
Easy but slow chat with your data: PrivateGPT
PrivateGPT is also designed to let you query your own documents using natural language and get a generative AI response. The documents in this application can include several dozen different formats. And the README assures you that the data is "100% private, no data leaves your execution environment at any point. You can ingest documents and ask questions without an internet connection!"
PrivateGPT features scripts to ingest data files, split them into chunks, create "embeddings" (numerical representations of the meaning of the text), and store those embeddings in a local Chroma vector store. When you ask a question, the app searches for relevant documents and sends just those to the LLM to generate an answer.
If you're familiar with Python and how to set up Python projects, you can clone the full PrivateGPT repository and run it locally. If you're less knowledgeable about Python, you may want to check out a simplified version of the project that author Iván Martínez set up for a conference workshop, which is considerably easier to set up.
That version's README file includes detailed instructions that don't assume Python sysadmin expertise. The repo comes with a
source_documents folder full of Penpot documentation, but you can delete those and add your own.
PrivateGPT includes the features you'd likely most want in a "chat with your own documents" app in the terminal, but the documentation warns it's not meant for production. And once you run it, you may see why: Even the small model option ran very slowly on my home PC. But just remember, the early days of home Internet were painfully slow, too. I expect these types of individual projects will speed up.
More ways to run a local LLM
There are more ways to run LLMs locally than just these five, ranging from other desktop applications to writing scripts from scratch, all with varying degrees of setup complexity.
A PrivateGPT spinoff, LocalGPT, includes more options for models and has detailed instructions as well as three how-to videos, including a 17-minute detailed code walk-through. Opinions may differ on whether this installation and setup is "easy," but it does look promising. As with PrivateGPT, though, documentation warns that running it on a CPU alone will be slow.
Another desktop app I tried, LM Studio, has an easy-to-use interface for running chats, but you're more on your own with picking models. If you know what model you want to download and run, this could be a good choice. If you're just coming from the world of using ChatGPT and have limited knowledge of how best to balance precision with size, all the choices may be a bit overwhelming at first. Hugging Face Hub is the main source of model downloads inside LM Studio, and it has a lot of models.
Unlike the other LLM options, which all downloaded the models I chose on the first try, I had problems downloading one of the models within LM Studio. Another didn't run well, which was my fault for maxing out my Mac's hardware, but I didn't immediately see a suggested minimum non-GPU RAM for model choices.
If you don't mind being patient about selecting and downloading models, though, LM Studio has a nice, clean interface once you're running the chat. As of this writing, the UI didn't have a built-in option for running the LLM over your own data.
It does have a built-in server that can be used "as a drop-in replacement for the OpenAI API," as the documentation notes, so code that was written to use an OpenAI model via the API will run instead on the local model you've selected.
Like h2oGPT, LM Studio also throws a warning on Windows that it's an unverified app. LM Studio code is not available on GitHub and isn't from a long-established organisation, though, so not everyone will be comfortable installing it.
In addition to using a pre-built model download interface through apps like h2oGPT, you can also download and run some models directly from Hugging Face, a platform and community for artificial intelligence that includes many LLMs. (Not all models there include download options.) Mark Needham, developer advocate at StarTree, has a nice explainer on how to do this, including a YouTube video. He's also got some related code in a GitHub repo, including sentiment analysis with a local LLM.
Hugging Face also has some documentation of its own about how to install and run available models locally.
Another popular option is to download and use LLMs locally in LangChain, a framework for creating end-to-end generative AI applications. That does require getting up to speed with writing code using the LangChain ecosystem. If you know LangChain basics, you may want to check out the documentation on Hugging Face Local Pipelines, Titan Takeoff (requires Docker as well as Python), and OpenLLM for running LangChain with local models. OpenLLM is another robust, standalone platform, designed for deploying LLM-based applications into production.