> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cudocompute.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploying LLMs like Google Gemma

> In this tutorial we will run Google Gemma with Ollama so that you can send queries via a REST API.

Ollama empowers users to work with large language models (LLMs) through its library of open-source models and its user-friendly API. This allows users to choose the best LLM for their specific task, whether it's text generation, translation, or code analysis. Ollama also simplifies interaction with different LLMs, making them accessible to a wider audience and fostering a more flexible and efficient LLM experience.

In this tutorial we will run Google Gemma with Ollama so that you can send queries via a REST API.

## Quick start guide

1. Prerequisites
2. Starting a virtual machine with cudoctl
3. Installing Ollama via SSH
4. Using Docker to start a LLM API

## Prerequisites

* Create a project and [add an SSH key](/docs/tutorials/how-to-generate-ssh-keys)
* Download the [CLI tool](/cli-tool)

## Starting a virtual machine with cudoctl

Start a virtual machine with the base image you require, here we will start with an image that already has NVIDIA drivers.

You can use the CUDO Compute console to start a virtual machine using the **Ubuntu 22.04 + NVIDIA drivers + Docker** image or alternatively use the command line tool `cudoctl`

To use the command line tool you will need to get an API key from the CUDO Compute console, see here: [API key](/docs/quickstart)
Then run `cudoctl init` and enter your API key.

First we search to find a virtual machine type to start

```shell theme={null}
cudoctl search --vcpus 4 --mem 8 --gpus 1
```

Find an image:

```shell theme={null}
cudoctl search images
```

After deciding on a machine type of `epyc-milan-rtx-a4000` (16GB GPU) in the `se-smedjebacken-1` data center and image `ubuntu-2204-nvidia-535-docker-v20240214` we can start a virtual machine:

```shell theme={null}
cudoctl vm create --id my-ollama --image ubuntu-2204-nvidia-535-docker-v20240214 --machine-type epyc-milan-rtx-a4000 --memory 8 --vcpus 4  --gpus 1 --boot-disk-size 80 -boot-disk-class network --data-center se-smedjebacken-1
```

## Installing Ollama via SSH

Get the IP address of the virtual machine

```shell theme={null}
cudoctl -json vm get my-ollama | jq '.externalIP'
```

SSH into the virtual machine

```shell theme={null}
ssh root@<IP_ADDRESS>
```

Install ollama

```shell theme={null}
curl -fsSL https://ollama.com/install.sh | sh
```

Download and run Google Gemma LLM, then you can enter your prompt.

```shell theme={null}
ollama run gemma:7b
```

From the Ollama [docs](https://github.com/ollama/ollama):

| Model              | Parameters | Size  | Download                       |
| ------------------ | ---------- | ----- | ------------------------------ |
| Llama 2            | 7B         | 3.8GB | `ollama run llama2`            |
| Mistral            | 7B         | 4.1GB | `ollama run mistral`           |
| Dolphin Phi        | 2.7B       | 1.6GB | `ollama run dolphin-phi`       |
| Phi-2              | 2.7B       | 1.7GB | `ollama run phi`               |
| Neural Chat        | 7B         | 4.1GB | `ollama run neural-chat`       |
| Starling           | 7B         | 4.1GB | `ollama run starling-lm`       |
| Code Llama         | 7B         | 3.8GB | `ollama run codellama`         |
| Llama 2 Uncensored | 7B         | 3.8GB | `ollama run llama2-uncensored` |
| Llama 2 13B        | 13B        | 7.3GB | `ollama run llama2:13b`        |
| Llama 2 70B        | 70B        | 39GB  | `ollama run llama2:70b`        |
| Orca Mini          | 3B         | 1.9GB | `ollama run orca-mini`         |
| Vicuna             | 7B         | 3.8GB | `ollama run vicuna`            |
| LLaVA              | 7B         | 4.5GB | `ollama run llava`             |
| Gemma              | 2B         | 1.4GB | `ollama run gemma:2b`          |
| Gemma              | 7B         | 4.8GB | `ollama run gemma:7b`          |

## Using Docker to start a LLM API

If you had created a vm in the previous step delete it by running:

```shell theme={null}
cudoctl vm delete my-ollama
```

Create a text file with a command to start the Ollama docker container:
`start-ollama.txt`

```shell theme={null}
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
```

Create a virtual machine and include the command to add a start script `-start-script-file start-ollama.txt`:

```shell theme={null}
cudoctl vm create --id my-ollama --image ubuntu-2204-nvidia-535-docker-v20240214 \
--machine-type epyc-milan-rtx-a4000 --memory 8 --vcpus 4  --gpus 1 --boot-disk-size 80 \
-boot-disk-class network --data-center se-smedjebacken-1 -start-script-file start-ollama.txt
```

Once the virtual machine is running you can curl the API to pull the model you require, here we use `gemma:7b`

```shell theme={null}
curl http://<IP_ADDRESS>:11434/api/pull -d '{"name": "gemma:7b"}'
```

Now it is ready to respond to a prompt:

```shell theme={null}
curl http://<IP_ADDRESS>:11434/api/generate -d '{
"model": "gemma:7b",
"prompt":"Why when you leave water overnight in a glass does it create bubbles in the water ?",
"stream":false
}' | jq '.response'
```