Ollama Hosting integration reference

此内容尚不支持你的语言。

⭐ Community Toolkit

To get started with the Aspire Ollama integrations, follow the Get started with Ollama integrations guide.

This article includes full details about the Aspire Ollama Hosting integration.

Installation

The Aspire Ollama hosting integration models an Ollama server as the OllamaResource type, and provides the ability to add models to the server using the AddModel extension method, which represents the model as an OllamaModelResource type. To access these types and APIs for expressing them as resources in your AppHost project, install the 📦 CommunityToolkit.Aspire.Hosting.Ollama NuGet package:

aspire add communitytoolkit-ollama

Aspire CLI 是交互式的；按提示选择合适的搜索结果：

Select an integration to add:

> communitytoolkit-ollama (CommunityToolkit.Aspire.Hosting.Ollama)
> Other results listed as selectable options...

#:package CommunityToolkit.Aspire.Hosting.Ollama@*

<PackageReference Include="CommunityToolkit.Aspire.Hosting.Ollama" Version="*" />

Add Ollama resource

In the AppHost project, register and consume the Ollama integration using the AddOllama extension method to add the Ollama container to the application builder. You can then add models to the container, which download and run when the container starts, using the AddModel extension method:

var builder = DistributedApplication.CreateBuilder(args);

var ollama = builder.AddOllama("ollama");

var phi35 = ollama.AddModel("phi3.5");

var exampleProject = builder.AddProject<Projects.ExampleProject>()
                            .WithReference(phi35);

builder.Build().Run();

Alternatively, if you want to use a model from the Hugging Face model hub, you can use the AddHuggingFaceModel extension method:

var llama = ollama.AddHuggingFaceModel("llama", "bartowski/Llama-3.2-1B-Instruct-GGUF:IQ4_XS");

When Aspire adds a container image to the AppHost, as shown in the preceding example with the docker.io/ollama/ollama image, it creates a new Ollama instance on your local machine.

Download the LLM

When the Ollama container for this integration first spins up, it downloads the configured LLMs. The progress of this download displays in the State column for this integration on the Aspire dashboard.

Cache the LLM

One or more LLMs are downloaded into the container which Ollama is running from, and by default this container is ephemeral. If you need to persist one or more LLMs across container restarts, you need to mount a volume into the container using the WithDataVolume method:

var ollama = builder.AddOllama("ollama")
                    .WithDataVolume();

var llama = ollama.AddModel("llama3");

Use GPUs when available

One or more LLMs are downloaded into the container which Ollama is running from, and by default this container runs on CPU. If you need to run the container with GPU support, you can enable it using the WithGPUSupport() extension method.

Nvidia:

var ollama = builder.AddOllama("ollama")
                    .WithGPUSupport();
ollama.AddModel("llama3");

AMD:

var ollama = builder.AddOllama("ollama")
                    .WithGPUSupport(OllamaGpuVendor.AMD);
ollama.AddModel("llama3");

For more information, see GPU support in Docker Desktop and GPU support in Podman.

Hosting integration health checks

The Ollama hosting integration automatically adds a health check for the Ollama server and model resources. For the Ollama server, a health check is added to verify that the Ollama server is running and that a connection can be established to it. For the Ollama model resources, a health check is added to verify that the model is running and that the model is available, meaning the resource will be marked as unhealthy until the model has been downloaded.

Open WebUI support

The Ollama integration also provides support for running Open WebUI and having it communicate with the Ollama container:

var ollama = builder.AddOllama("ollama")
                    .WithOpenWebUI();
ollama.AddModel("llama3");