How to Run Large Language Models Locally on a Windows Machine Using WSL and Ollama

How to Run Large Language Models Locally on a Windows Machine Using WSL and Ollama

prerequisite

Install WSL

sudo apt install curl

What is Ollama?

Get up and running with large language models, locally.

Ollama

Install Ollama via curl

curl https://ollama.ai/install.sh | sh

Run Ollama

In this case, we will try to run Mistral-7B.
If you want to try another model, you can pick from the following site.
https://ollama.ai/library

ollama serve

And open another Terminal tab and run the following command. The following command will pull a model.

ollama run mistral

If everything works properly, you will see something like below.

My machine has a GPU, RTX3070. So Ollama is using the GPU.

Terminate Ollama

If you want to exit Ollama, you need to type the following.

/exit

Then ctrl + c in a terminal what you ran ollama serve.