How to run large language models locally on Windows (text-generation-webui)

It’s never been easier to run large language models locally on your PC. Oobabooga text-generation-webui is a free GUI for running language models on Windows, Mac, and Linux. It offers many convenient features, such as managing multiple models and a variety of interaction modes.

In this article, you will learn what text-generation-webui is and how to install it on Windows.

(See this guide for installing on Mac.)

What is text-generation-webui?

OObabooga Text-generation-webui is a GUI (graphical user interface) for running Large Language Models (LLMs) like LLaMA, GPT-J, Pythia, OPT, and GALACTICA.

Why do you need a GUI for LLMs? The GUI is like a middleman, in a good sense, who makes using the models a more pleasant experience. Instead of interacting with the language models in a terminal, you can switch models, save/load prompts with mouse clicks, and write prompts in a text box.

To sum up, it offers a better user experience for using local LLMs.

The idea is similar to AUTOMATIC111 web UI for the Stable Diffusion AI image generator.

Systems requirement

Your Windows PC should have an Nvidia GPU video card. You can still run the model without a GPU card.

Install text-generation-webui on Windows

We will go through two options:

1. One-click installer
2. Installing with command lines.

The one-click installer is recommended for regular users. Installing with command lines is for those who want to modify and contribute to the code base.

Installing text-generation-webui with One-click installer

The easiest way is to use the 1-click installer provided by the repository to install the webui.

Step 1: Install Visual Studio 2019 build tool

Download and install Visual Studio 2019 Build Tools. (You can use the first one in the table).

Step 2: Download the installer

Download the Windows installer. It’s a zip file.

Step 3: Unzip the Installer

In File Explorer, extract the content by right-clicking the zip file and then “Extract All…”.

Click Extract. You should get a new folder called oobabooga_windows.

This is your webui’s folder. You can move this folder to a location you like.

Step 4: Run the installer

Inside the oobabooga_windows folder, you will find several .bat files.

Double-click the file start_windows.bat to start the installation. A warning may show up.

Click More info and then Run anyway.

Step 5: Answer some questions

Next, it will ask you what GPU do you have.

For Windows users, it will either be A) NVIDIA or D) None. Pick NVIDIA if you have a discrete Nvidia GPU like the RT or RTX series. (Integrated GPU doesn’t count). Select None if you don’t have a discrete GPU.

The installation will take a while. When it is done, you will see a local URL.

Step 6: Access the web-UI

Going to the local URL in a browser to access the web-UI.

http://127.0.0.1:7860

Ctrl + left click on the URL in the terminal should bring you there.

Step 7: Download a model

NVIDIA GPU users

If you have an NVIDIA GPU, download a model in GPTQ format.

In text-generation-webui, navigate to the Model page. In the Download custom model or LoRA section, put in one of the following.

localmodels/Llama-2-7B-Chat-GPTQ

CPU users

If you don’t have an NVIDIA GPU, you will need to download a model in GGML format. You can download the LLAMA 2 7B Chat model to get started. Put it in text-generation-webui > models folder.

Step 8: Load the model

Click the refresh button next to the Model dropdown menu. Select the model you just downloaded. Click the Load button to load.

Step 9: Test the model

Go to the Text generation page, and type a question in the input text box to start chatting.

Starting the web-ui again

To start the webui again next time, double-click the file start_windows.bat.

Installation using command lines

If the one-click installer doesn’t work for you or you are not comfortable running the script, follow these instructions to install text-generation-webui.

Make sure you are comfortable with using command lines before proceeding.

This option requires Python 3.10 and git. Install them if you don’t have them already.

The following instructions are for Nvidia GPU cards.

Step 1. Open the PowerShell app and clone the text-generation-webui repository.

git clone https://github.com/oobabooga/text-generation-webui.git

Step 2. Create a virtual environment.

cd text-generation-webui

python -m venv venv

Step 2. Enable running scripts in PowerShell.

set-executionpolicy RemoteSigned -Scope CurrentUser

Step 3. Activate the virtual environment.

venv\Scripts\Activate.ps1

You should see the label venv in front of the command prompt.

Step 4. Install pytorch.

pip install torchaudio torch==2.0.0+cu118 torchvision ninja --force-reinstall --extra-index-url https://download.pytorch.org/whl/cu118

Step 5. Install the Windows version of bitsandbytes.

pip install https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.38.1-py3-none-any.whl

Step 5. Install the required packages.

pip install -r requirements.txt

Step 5. Install GPTQ for LLaMA.

Make the repositories directory.

mkdir repositories; cd repositories

Clone and install GPT-for-LLaMa.

git clone https://github.com/oobabooga/GPTQ-for-LLaMa.git -b cuda

cd  GPTQ-for-LLaMa

python -m pip install -r requirements.txt

python setup_cuda.py install

Step 6. Download a model.

First, go back to text-generation-webui directory.

cd ~/text-generation-webui

Download a model. The following command downloads the Vicuna 7B model from this repository.

 python download-model.py eachadea/vicuna-7b-1.1

Step 7. Start text-generation-webui

python server.py --chat

Follow the local URL to start using text-generation-webui.

To make the startup easier next time, use a text editor to create a new text file start.bat with the following content.

call .\venv\Scripts\activate.bat
call python server.py --chat

Save it to text-generation-webui’s folder.

Next time, double-click start.bat to start the webui.