It’s never been easier to run large language models locally on your PC. Oobabooga text-generation-webui is a free GUI for running language models on Windows, Mac, and Linux. It offers many convenient features, such as managing multiple models and a variety of interaction modes.
In this article, you will learn what text-generation-webui is and how to install it on Windows.
(See this guide for installing on Mac.)
- What is text-generation-webui?
- Systems requirement
- Install text-generation-webui on Windows
- Installing text-generation-webui with One-click installer
- Installation using command lines
What is text-generation-webui?
OObabooga Text-generation-webui is a GUI (graphical user interface) for running Large Language Models (LLMs) like LLaMA, GPT-J, Pythia, OPT, and GALACTICA.
Why do you need a GUI for LLMs? The GUI is like a middleman, in a good sense, who makes using the models a more pleasant experience. Instead of interacting with the language models in a terminal, you can switch models, save/load prompts with mouse clicks, and write prompts in a text box.
To sum up, it offers a better user experience for using local LLMs.
The idea is similar to AUTOMATIC111 web UI for the Stable Diffusion AI image generator.
Your Windows PC should have an Nvidia GPU video card. You can still run the model without a GPU card.
Install text-generation-webui on Windows
We will go through two options:
- One-click installer
- Installing with command lines.
The one-click installer is recommended for regular users. Installing with command lines is for those who want to modify and contribute to the code base.
Installing text-generation-webui with One-click installer
The easiest way is to use the 1-click installer provided by the repository to install the webui.
Step 1: Install Visual Studio 2019 build tool
Download and install Visual Studio 2019 Build Tools. (You can use the first one in the table).
Step 2: Download the installer
Download the Windows installer. It’s a zip file.
Step 3: Unzip the Installer
In File Explorer, extract the content by right-clicking the zip file and then “Extract All…”.
Click Extract. You should get a new folder called oobabooga_windows.
This is your webui’s folder. You can move this folder to a location you like.
Step 4: Run the installer
Inside the oobabooga_windows folder, you will find several .bat files.
Double-click the file
start_windows.bat to start the installation. A warning may show up.
Click More info and then Run anyway.
Step 5: Answer some questions
Next, it will ask you what GPU do you have.
For Windows users, it will either be A) NVIDIA or D) None. Pick NVIDIA if you have a discrete Nvidia GPU like the RT or RTX series. (Integrated GPU doesn’t count). Select None if you don’t have a discrete GPU.
The installation will take a while. When it is done, you will see a local URL.
Step 6: Access the web-UI
Going to the local URL in a browser to access the web-UI.
Ctrl + left click on the URL in the terminal should bring you there.
Step 7: Download a model
NVIDIA GPU users
If you have an NVIDIA GPU, download a model in GPTQ format.
In text-generation-webui, navigate to the Model page. In the Download custom model or LoRA section, put in one of the following.
If you don’t have an NVIDIA GPU, you will need to download a model in GGML format. You can download the LLAMA 2 7B Chat model to get started. Put it in text-generation-webui > models folder.
Step 8: Load the model
Click the refresh button next to the Model dropdown menu. Select the model you just downloaded. Click the Load button to load.
Step 9: Test the model
Go to the Text generation page, and type a question in the input text box to start chatting.
Starting the web-ui again
To start the webui again next time, double-click the file
Installation using command lines
If the one-click installer doesn’t work for you or you are not comfortable running the script, follow these instructions to install text-generation-webui.
Make sure you are comfortable with using command lines before proceeding.
The following instructions are for Nvidia GPU cards.
Step 1. Open the PowerShell app and clone the text-generation-webui repository.
git clone https://github.com/oobabooga/text-generation-webui.git
Step 2. Create a virtual environment.
python -m venv venv
Step 2. Enable running scripts in PowerShell.
set-executionpolicy RemoteSigned -Scope CurrentUser
Step 3. Activate the virtual environment.
You should see the label
venv in front of the command prompt.
Step 4. Install pytorch.
pip install torchaudio torch==2.0.0+cu118 torchvision ninja --force-reinstall --extra-index-url https://download.pytorch.org/whl/cu118
Step 5. Install the Windows version of bitsandbytes.
pip install https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.38.1-py3-none-any.whl
Step 5. Install the required packages.
pip install -r requirements.txt
Step 5. Install GPTQ for LLaMA.
Make the repositories directory.
mkdir repositories; cd repositories
Clone and install GPT-for-LLaMa.
git clone https://github.com/oobabooga/GPTQ-for-LLaMa.git -b cuda
python -m pip install -r requirements.txt
python setup_cuda.py install
Step 6. Download a model.
First, go back to text-generation-webui directory.
Download a model. The following command downloads the Vicuna 7B model from this repository.
python download-model.py eachadea/vicuna-7b-1.1
Step 7. Start text-generation-webui
python server.py --chat
Follow the local URL to start using text-generation-webui.
To make the startup easier next time, use a text editor to create a new text file
start.bat with the following content.
call .\venv\Scripts\activate.bat call python server.py --chat
Save it to text-generation-webui’s folder.
Next time, double-click start.bat to start the webui.