Skip to content

How to install text-generation-webui (a GUI for local LLMs) on Mac

Oobabooga text-generation-webui is a GUI for running large language models. It offers many convenient features, such as managing multiple models and a variety of interaction modes.

In this article, you will learn what text-generation-webui is and how to install it on Apple Silicon M1/M2.

What is text-generation-webui?

OObabooga Text-generation-webui is a GUI (graphical user interface) for running Large Language Models (LLMs) like LLaMA, GPT-J, Pythia, OPT, and GALACTICA.

Why do you need a GUI for LLMs? The GUI is like a middleman, in a good sense, who makes the language model easier to use.

Instead of interacting with the language models in a terminal, you can switch models, save/load prompts with mouse clicks, and write prompts in a text box.

To sum up, it offers a better user experience for using local LLMs.

The idea is similar to AUTOMATIC111 web UI for the Stable Diffusion AI image generator.

Install text-generation-webui on Mac

Step 1: Upgrade MacOS if needed

You need to have macOS Ventura 13.3 or newer.

You can check by

  1. Click on the Apple Icon on the top left.
  2. Click on About this Mac

You are good to proceed to the next step if you see MacOS 13.3 or newer.

If not, follow the following steps to upgrade your Mac

  1. Click on the Apple Icon on the top left.
  2. Click on Systems Settings…
  3. Click on Update Apple ID settings
  4. Click on Software Upgrade Available

Upgrade to MacOS 13.3 or newer.

After the upgrade, confirm you have MacOS 13.3 or newer by clicking on the Apple Icon and then About this Mac.

Step 2: Install Homebrew

Install Homebrew, a package manager for Mac, if you haven’t already.

Open the Terminal app, type the following command, and press return.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Step 3: Install Git and Python

Run the following command in the Terminal app to install Git and Python 3.10 if you haven’t already.

brew install git [email protected]

Step 4: Clone the repository

In the terminal App, go to the home folder (Or any folder you wish to install the webui). Run

cd ~

Run the following command to copy the code of text-generation-webui to your local computer.

git clone https://github.com/oobabooga/text-generation-webui.git

Enter the webui’s folder by running

cd text-generation-webui

Step 5: Create and activate a virtual environment

Make sure you are using Python 3.10.

python3 --version

You should see you are running Python 3.10.x

Now creates a virtual environment to get a clean slate of Python environment.

python3 -m venv venv

Activate the virtual environment.

source venv/bin/activate

You should see the label (venv) in front of your prompt. like this:

(venv) XXXX@MacBook text-generation-webui %

Step 6: Install the required packages

Run the following command to install the required packages.

pip install -r requirements.txt

The released Pytorch package won’t work for text-generation-webui on Mac. You need to install the nightly development build.

pip install -U --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html

Step 7: Download a language model

Finally, you need a language model to say things to. Let’s download a small model for now. We can install others later. Run the command below to download the Facebook 1.3B OPT (Open Pretrained Transformer) Language model.

python download-model.py facebook/opt-1.3b

That’s it!

Using the webui

Step 1: Starting webui

Open the Terminal App. Run the following command to change to

cd ~/text-generation-webui

Activate the virtual environment.

source venv/bin/activate

You should see the label (venv) in front of your prompt.

Run the following to start text-generation-webui.

python server.py

You should see something like

Running on local URL: http://127.0.0.1:7860

Go to the URL in your browser to start the GUI. Although you are using a browser, this URL is served locally from your computer.

You should see the text-generation-webui like the screenshot below.

text-generation-webui install on Mac

Step 2: Select a model

You will need to select a model before using the webui.

Go to the Model page.

Under the Model section, select facebook_opt-1.3b.

Step 3: Write a prompt

Test to make sure the text generation is working.

For example, put in “Who is JFK?” after “Question:”.

Press Generate.

You should see the language model’s response in the Output console.

Shutting down text-generation-webui

When you are done, shut down text-generation-webui by pressing Control-C in the terminal.

Alternatively, closing the Terminal App will also shut down the GUI.

Download others models

You can easily download any model from HuggingFace. Below are some models available to use in the GUI.

The command for downloading a model is

python download-model.py organization/model

For example, to download a GALACTICA model, follow the GALACTICA link above.

Pick a model size. E.g. The 1.3B model can be downloaded using the following command.

python download-model.py facebook/galactica-1.3b
    Tags:

    21 thoughts on “How to install text-generation-webui (a GUI for local LLMs) on Mac”

    1. Eddie Huang
      Reply

      Hey! great write up on this. I’ve run this on a windows based WSL deployment, but i was wondering whats the lag time you’ve gotten regarding the time to first token generation and also whats the token generation speed per second? I’m curious if the M1 is any more optimized than say a regular AMD CPU or a dedicated GPU.

      Thanks!

      -Eddie

    4. IOZNY2
      Reply

      I can’t make it to work. I followed all the instructions but I get the following erro:
      ModuleNotFoundError: No module named ‘torch’
      I installed the latest compatible Pytorch for mac, and also installed the nightly development build.

      On Automatic 111 I remeber I had to add the following commandARg to make it work :–skip-torch-cuda-test”

      What am I missing? Thanks,

      1. Andrew
        Reply

        It couldn’t find any pytorch installed.

        Have you activated virtual environment before installing the required package and running the server? Make sure you see (venv) before your prompt in Terminal.

    6. [email protected]
      Reply

      Hi andrew – I’m getting a strange error when I try to run the model (python server.py)
      ImportError: dlopen(/Users/marcstieglitz/text-generation-webui/venv/lib/python3.10/site-packages/orjson/orjson.cpython-310-darwin.so, 0x0002): tried: ‘/Users/marcstieglitz/text-generation-webui/venv/lib/python3.10/site-packages/orjson/orjson.cpython-310-darwin.so’ (mach-o file, but is an incompatible architecture (have ‘arm64’, need ‘x86_64’)), ‘/System/Volumes/Preboot/Cryptexes/OS/Users/marcstieglitz/text-generation-webui/venv/lib/python3.10/site-packages/orjson/orjson.cpython-310-darwin.so’ (no such file), ‘/Users/marcstieglitz/text-generation-webui/venv/lib/python3.10/site-packages/orjson/orjson.cpython-310-darwin.so’ (mach-o file, but is an incompatible architecture (have ‘arm64’, need ‘x86_64’))
      (venv) (base) marcstieglitz@marcs-air text-generation-webui %

      I’m using the right python version

    10. S R
      Reply

      Hey, whenever I try to run the “server.py” script, I keep on getting “RuntimeError: MPS backend out of memory (MPS allocated: 17.79 GB, other allocations: 303.87 MB, max allowed: 18.13 GB). Tried to allocate 200.00 MB on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).” I feel like disabling the upper limit sounds way too risky, do you know how to fix this by any chance?

      1. Andrew
        Reply

        What model are you running? You shouldn’t need to allocate that much for 7B or 13B models.

        You can try restarting your PC to see if that helps. A lot of programs have memory leak on Mac.

    13. Ryan
      Reply

      Thanks Andrew! Upon reinstalling the requirements I was able to figure out that this had actually errored during install, which is likely why the gradio wasn’t found. It’s saying this:

      Preparing metadata (pyproject.toml) … error
      error: subprocess-exited-with-error

      × Preparing metadata (pyproject.toml) did not run successfully.
      │ exit code: 1
      ╰─> [6 lines of output]

      Cargo, the Rust package manager, is not installed or is not on PATH.
      This package requires Rust and Cargo to compile extensions. Install it through
      the system's package manager or via https://rustup.rs/

      Checking for Rust toolchain....
      [end of output]

      note: This error originates from a subprocess, and is likely not a problem with pip.
      error: metadata-generation-failed

      I went to the link and installed rust, but I’m still getting the same error… Does it need to be in a certain directory?

      Thanks again

      1. Andrew
        Reply

        Hi, please check if you are using the right python version. After activating venv, with (venv) in front of your prompt, type
        python --version
        You should be using 3.10.x

        Try upgrading pip
        pip install --upgrade pip

        If it still doesn’t work, try installing rust
        brew install rust

    15. Ryan
      Reply

      This is a super easy tutorial to follow. I just hit one snag at the end.
      When I try type the command ‘Python server.py” it gives me an error:

      /text-generation-webui/server.py”, line 21, in
      import gradio as gr
      ModuleNotFoundError: No module named ‘gradio’

      Any idea what that might be from?

      thanks!

      1. Andrew
        Reply

        Hi Ryan, gradio should be installed with the step “pip install -r requirements.txt”

        Make sure you were see “(venv)” before your command prompt when you do this step and also when you start “python server.py”

        You can check if gradio has been installed by:
        1. Enter the virtual environment
        2. type “python” to enter the python
        3. type “import gradio”

        You are good if you don’t see any errors.

    20. Me
      Reply

      Hi, thanks so much for putting in the time to create this excellent resource for installing AI tools. I was wondering, could you provide a step by step guide for using the API with text-generation-webui? I tried to run the API a few days ago using the api-example.py and just ended up getting the response NONE. A helpful guide for this would fanstastic!

    Leave a Reply

    Your email address will not be published. Required fields are marked *