Multi-node deployment

ART’s distributed runtime consumes a Monarch host mesh. SkyPilot can provision that mesh, but it is a deployment tool rather than an ART dependency. The ART process does not launch another SkyPilot cluster from inside its allocation.

Controller program

The bootstrap accepts a source script or import path to a top-level async function. A script reference such as train.py:main adds the script’s directory to every ART-owned worker’s Python path, so source synchronized by SkyPilot does not need to be packaged first. SkyPilot runs the bootstrap on every node, but only rank 0 imports the module and invokes the controller. examples/multinode/program.py is a complete CPU-runnable Yes/No/Maybe smoke:

import asyncio
import os
import socket

import art
from art.distributed import (
    ArtLaunchContext,
    ArtRuntime,
    InstalledAsyncCallable,
    compile_topology,
)

REWARDS = {"yes": 0.5, "no": 0.75, "maybe": 1.0}


async def rollout(
    _model: art.TrainableModel, answer: str, _config: None
) -> art.Trajectory:
    messages: art.MessagesAndChoices = [
        {"role": "user", "content": f"Respond with {answer}."},
        {"role": "assistant", "content": answer},
    ]
    return art.Trajectory(
        messages_and_choices=messages,
        reward=REWARDS[answer],
        metadata={
            "answer": answer,
            "hostname": socket.gethostname(),
            "process_id": os.getpid(),
        },
    )


async def main(launch: ArtLaunchContext) -> None:
    host_count = launch.host_count
    runtime = await ArtRuntime.start(
        launch.host_mesh,
        compile_topology(
            cluster=launch.homogeneous_cluster(
                cpu_slots=1,
                startup_timeout_s=90,
                rpc_timeout_s=30,
            )
        ),
    )
    try:
        workers = tuple(range(host_count))
        executor = runtime.rollout_executor(
            InstalledAsyncCallable.from_callable(rollout),
            target_workers=host_count,
        )
        executor.set_workers(workers)
        model = art.TrainableModel(
            name="multinode-smoke",
            project="art",
            base_model="not-loaded",
            run_name="multinode-smoke",
        )
        trajectories = []
        for answer in REWARDS:
            trajectories.extend(
                await asyncio.gather(
                    *(
                        executor.run(worker, rollout, model, answer, None)
                        for worker in workers
                    )
                )
            )
        answers = [
            str(trajectory.metadata["answer"]) for trajectory in trajectories
        ]
        expected = [answer for answer in REWARDS for _ in workers]
        placements = {
            (trajectory.metadata["hostname"], trajectory.metadata["process_id"])
            for trajectory in trajectories
        }
        if answers != expected or len(placements) != host_count:
            raise RuntimeError(
                f"distributed rollout mismatch: {answers=}, {placements=}"
            )
        print(f"ART_MULTINODE_SMOKE_PASS hosts={host_count} answers={answers}")
    finally:
        await runtime.close()

The controller receives an ArtLaunchContext once, while the top-level rollout runs in one process on each host. The context owns the attached host mesh and builds a homogeneous typed cluster without exposing provider environment variables. Both functions must be installed or synchronized at the same import paths on every node; ART sends verified import references and never ships opaque closures. The dummy TrainableModel is serialized for the rollout contract but never loaded, so this validates the package, host admission, process placement, public trajectory types, and cleanup without a GPU or inference server.

The distributed service APIs are opt-in. Existing single-node programs continue to construct and use LocalBackend exactly as before:

from art.local import LocalBackend

backend = LocalBackend()

SkyPilot

Start with examples/multinode/skypilot.yaml. It is an intentionally CPU-only two-node smoke that runs all three bounded rollouts on each host without reserving training GPUs or provisioning the managed vLLM runtime.

For source-based GPU training, replace its resources and setup with the desired topology and run the CUDA-detecting cluster setup:

resources:
  accelerators: H200:8

setup: |
  set -euo pipefail
  INSTALL_MULTINODE=true bash scripts/setup.sh

run: |
  set -euo pipefail
  export NCCL_NET=IB
  exec .venv/bin/art-monarch skypilot \
    --program train.py:main

set -euo pipefail is required because SkyPilot runs multiline setup under Bash without enabling fail-fast behavior. For source development, scripts/setup.sh selects the CUDA-matched root and private trainer locks and builds HybridEP. It does not install system packages. The image or cluster bootstrap must provide the NVIDIA driver and toolkit, native build tools, NCCL network transport, MOFED/RDMA devices, and the required kernel modules. The CPU example only syncs the root distributed extra.

Setup is cluster provisioning, not service startup. art-monarch, trainer actors, and managed vLLM processes never invoke these shell scripts. Source checkouts launch the already-built vllm_runtime/.venv; release wheels may materialize their bundled, locked vLLM environment into a content-addressed cache on first use.

Any GPU workload spanning hosts must set one explicit NCCL network contract in its ClusterSpec, for example nccl_transport=NcclTransportSpec(net_name="IB"), and set NCCL_NET to that exact registered name on every node. IB covers built-in InfiniBand/RoCE; external network plugins use their registered NCCL name. Before model allocation, ART runs a small collective in both the trainer and managed-vLLM environments and requires each rank to report that exact selected module. It never retries with Socket. Deployment qualification remains responsible for all-GPU bandwidth, GPU Direct RDMA, HCA, and GID validation.

Cross-host HybridEP uses NixlTransportSpec(). If metadata_store is omitted, the controller starts a checksum-pinned etcd process, publishes its routable endpoint, health-checks it from every host, and owns its cleanup. An explicitly managed endpoint remains supported.

If ART_VLLM_RUNTIME_BIN is set, it must point directly to a standard .venv/bin/art-vllm-runtime-server executable. ART derives the matching Python, runtime root, environment, and working directory from that path so the preflight cannot certify a different runtime. Arbitrary command wrappers fail closed.

For a published release wheel, install the profile matching the host CUDA toolkit. For CUDA 12:

setup: |
  set -euo pipefail
  uv venv --python 3.12 --seed .venv
  .venv/bin/pip install \
    --extra-index-url https://download.pytorch.org/whl/cu128 \
    "openpipe-art[megatron]==VERSION"

For CUDA 13, use openpipe-art[megatron-cu130]==VERSION and the PyTorch cu130 index. These commands install ART, Monarch, and the CUDA-specific NIXL wheel. The example uses the supported image’s uv installation to provision Python 3.12. The first trainer launch materializes the pinned Megatron environment, builds source-only CUDA components such as CUDA 12 Apex and ART’s HybridEP when needed, and reuses the immutable result on later launches. NIXL and its UCX GDA plugin come from the official relocatable wheel; ART bundles only the matching pinned headers needed to build HybridEP.

Release wheels use content-addressed managed Megatron and vLLM runtime bundles. Only wheels built with scripts/build_package.py contain those bundles. The install profile provides uv; first use needs package-index access unless the node-local cache was prewarmed.

examples/multinode/skypilot_training.yaml runs a real two-host DP2 SFT step from the published wheel. It synchronizes only the user program, not an ART checkout. Point ART_SHARED_ROOT at an existing path mounted on every host, then launch:

sky launch -c art-multinode-training \
  --env ART_SHARED_ROOT=/mnt/shared/art-multinode-release \
  examples/multinode/skypilot_training.yaml

Launch it from the project root:

sky launch -c art-multinode examples/multinode/skypilot.yaml

One task rank runs on each allocated node. Every rank owns one Monarch worker subprocess; rank 0 also attaches the host mesh and runs the controller program. Rank-0 program completion or failure closes the lifecycle sockets and releases the peer task ranks. No manual SSH or per-node command is required.

Each task invocation owns fresh worker loops and terminates them after the host mesh shuts down. A later sky exec starts new loops; ART does not reattach a second controller to completed workers.

Ctrl-C disconnects SkyPilot log streaming; it does not stop the remote job. Check the queue and cancel explicitly when needed:

sky queue art-multinode
sky cancel art-multinode JOB_ID

Only after the previous job is terminal, reuse an existing cluster without rerunning setup:

sky exec art-multinode examples/multinode/skypilot.yaml

sky exec synchronizes the workdir before scheduling, so running it while the previous job is live can change files under that job. Use sky launch instead when setup, mounts, the image, SkyPilot config, a wheel, pyproject.toml, or a lockfile changed. Setting num_nodes: 1 uses the same controller on one node.

For a local process that explicitly wants the same Monarch service APIs, ART can own one loopback worker directly:

.venv/bin/art-monarch local \
  --program examples/multinode/program.py:main \
  --port 0 \
  --startup-timeout 90

Port 0 selects a fresh loopback port. ArtRuntime.start_local(...) is the equivalent library API. It accepts the same one-host compiled topology used by multi-node code and owns the worker for the runtime lifetime.

SkyPilot provides SKYPILOT_NODE_RANK, SKYPILOT_NODE_IPS, and SKYPILOT_NUM_NODES; ART validates and translates them internally. Port 22222 is the Monarch worker port and 22223 is its job-lifecycle port. Pass --port N to reserve N and N + 1 instead. These ports must be reachable between allocated nodes but must not be publicly exposed: ART’s Monarch runtime uses unauthenticated trust_all_connections transport.

Existing SSH hosts

For preallocated machines, start and own all workers from one controller command:

.venv/bin/art-monarch ssh \
  --host gpu-a=10.0.0.10 \
  --host gpu-b=10.0.0.11 \
  --python /shared/project/.venv/bin/python \
  --program /shared/project/train.py:main

Each value is SSH_TARGET=WORKER_HOST. Omit =WORKER_HOST when the SSH target is also the private address to which Monarch should bind. The controller must have both passwordless SSH access to every SSH_TARGET and a direct trusted private or VPN route to every WORKER_HOST:N. SSH options such as ProxyJump or --ssh-arg=-F affect only launch and stop commands; they do not tunnel Monarch traffic. SSH mode uses only worker port N, not SkyPilot’s lifecycle port N + 1.

The selected Python executable and source script must exist at the same paths on every host. ART uses non-interactive SSH, verifies that each launch-specific worker PID owns its listener, and monitors each foreground SSH process for the controller lifetime. A pre-existing listener is a hard error rather than a worker to reattach. SIGTERM and SIGHUP trigger bounded remote cleanup before the controller exits.

The lower-level worker and controller subcommands remain available for schedulers or process supervisors that own worker lifecycle themselves. Those supervisors must replace worker loops before a subsequent controller attach.