Skip to main content
Harbor is a framework for evaluating and training agents on sandboxed tasks. Each trial runs an agent against one task inside an isolated environment, then a verifier scores the result. Harbor ships environments for Docker on your own machine and for several cloud sandboxes, and loads any other environment by import path. Sail provides one:
Pass that import path to Harbor and every trial’s environment becomes a Sailbox. Tasks, agents, verifiers, and datasets are unchanged.

​
Quickstart

Install sail and harbor in the same Python environment (Harbor needs Python 3.12 or newer) and set your Sail API key. Then run Harbor’s hello-world dataset with the oracle agent, which runs each task’s reference solution and needs no model key:
Harbor creates a Sailbox, runs the task’s solution in it, runs the verifier, writes the trial’s results under ./jobs, and terminates the Sailbox when the trial ends. A real evaluation is the same command with a real agent, a model, and more trials in flight. Each trial gets its own Sailbox, so the machine running Harbor is no longer the limit on concurrency:
A job config file (harbor run -c job.yaml) selects the environment the same way:

​
Credentials and app

The process that runs Harbor needs a Sail credential: set SAIL_API_KEY, or run sail auth login once on that machine. Sailboxes are created in the harbor app in your organization, which is created on first use; set SAIL_APP to use a different app. Each Sailbox is named harbor- followed by the trial’s session id, and sail box list --app harbor (or the app you chose) lists the ones a run created.

​
Task images

Harbor decides where an environment’s image comes from, and Sail follows the same rules as Harbor’s other cloud providers.
  • A task that declares a prebuilt docker_image in its task.toml runs on that image, pulled from its registry. Docker-style short references work: python:3.11 means docker.io/library/python:3.11. The registry must be one Sail supports (docker.io, ghcr.io, public.ecr.aws, or quay.io), and the image must be Debian or Ubuntu based; see Bring your own base image for the full requirements.
  • A task that ships an environment/Dockerfile instead has it built into a Sailbox image. The build happens once per organization: the first trial of that task waits for it, and later trials start from the cached image. Harbor’s --force-build rebuilds it. See Build from a Dockerfile for what a Dockerfile can contain.
  • A task that ships an environment/docker-compose.yaml runs as a Docker Compose project, with the Sailbox as the Docker host: the services’ images are pulled or built inside it, commands run in the task’s main service, and Harbor’s per-service operations (exec, download, stop) reach the other services.

​
Sizing

A task’s CPU, memory, and storage requests pick the smallest Sailbox size that covers them: s (1 vCPU), m (4 vCPU), or l (8 vCPU). A request above the size’s default memory or disk ceiling raises that ceiling, up to the size’s maximum. A task that declares no resources gets an s Sailbox. A task that requests more than 8 CPUs fails up front, since no Sailbox provides more. The size is a ceiling, not a reservation: a Sailbox is billed for the CPU, memory, and disk it actually uses, so a task that requests 8 CPUs and uses one pays for one.

​
Cleanup

By default, harbor run deletes each environment when its trial ends, which terminates the Sailbox. A termination that fails is reported in that trial’s result, like any other trial error. With --no-delete, Harbor stops each environment without deleting it. The Sailbox goes to sleep with its filesystem and processes intact, and billing stops until it is resumed. Terminate those Sailboxes yourself when you are done with them, with sail box terminate <id> or from the dashboard. A trial whose environment fails to start terminates its Sailbox before the error is reported, even with --no-delete. If that termination fails, the trial’s normal cleanup stops the Sailbox the same way it would after a completed trial.

​
Unsupported task features

Harbor checks a task’s needs against the environment before starting it, so a task that needs something unsupported fails up front with a clear error instead of running without the feature. Sailboxes do not offer:
  • GPUs, TPUs, or Windows environments.
  • IPv6 allowlist entries. A Sailbox does not reach the internet over IPv6.
  • A no-network or allowlist policy for a Compose task, because bringing the project up pulls images over the network. Both are enforced for a task that does not use Compose.
A task with the default public networking is unaffected. The network policy guide has the allowlist entry rules. Outside Compose mode, host mount specs are accepted and unused, the same as Harbor’s other cloud providers. A Compose task’s mounts are bound into its main service.