This guide walks through setting up a local real-time AI video pipeline using ComfyUI and ComfyStream. You’ll learn how to process live video with AI models, using depth map generation as an example.

​
Prerequisites

  • A RunPod account with access to GPU instances

  • Basic familiarity with terminal commands

  • SSH key pair for secure connections

  • Node.js and npm installed locally

​
Technical Architecture

Local Machine                     RunPod Instance
┌─────────────┐    SSH Tunnel    ┌─────────────┐
│ Browser UI  │◄──────────────►  │  ComfyUI    │
│ (localhost) │                  │             │
└─────────────┘                  │  ComfyStream│
                                 └─────────────┘

​
Step 1: Set Up ComfyUI Environment

  1. Launch a RunPod instance with ComfyUI-Launcher
# On RunPod instance
cd /workspace
git clone https://github.com/ComfyWorkflows/ComfyUI-Launcher.git
cd ComfyUI-Launcher
./setup.sh
  1. Configure workspace directories
# Directories will be created automatically
/workspace/comfyui_launcher_models    # Shared models
/workspace/comfyui_launcher_projects  # Workspace projects

Note: Install at least one model through the ComfyUI manager GUI to enable the “Load Checkpoint” node.

​
Step 2: Install ComfyStream

  1. Set up Conda environment
cd /workspace
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
eval "$(/root/miniconda3/bin/conda shell.bash hook)"

# Create and activate environment
conda create -n comfystream python=3.11
conda activate comfystream

# Install PyTorch
conda install pytorch torchvision -c pytorch
  1. Install ComfyStream
cd /workspace
git clone https://github.com/yondonfu/comfystream.git
cd comfystream
git fetch
git checkout pin-opentelemetry
pip install .

# Install tensor utilities
cp -r nodes/tensor_utils /workspace/comfyui_launcher_projects/{workspace}/comfyui/custom_nodes

​
Step 3: Install DepthAnything TensorRT Node

  1. Clone and install the node
cd /workspace/comfyui_launcher_projects/{workspace}/comfyui/custom_nodes
git clone https://github.com/yuvraj108c/ComfyUI-Depth-Anything-Tensorrt.git
cd ComfyUI-Depth-Anything-Tensorrt
pip install -r requirements.txt
  1. Build TensorRT engine
wget https://huggingface.co/yuvraj108c/Depth-Anything-2-Onnx/resolve/main/depth_anything_v2_vitb.onnx?download=true
mv depth_anything_v2_vitb.onnx?download=true depth_anything_v2_vitb.onnx

# Update export_trt.py with:
# onnx_path = "./depth_anything_v2_vitb.onnx"
# trt_path = "depth_anything_v2_vitb-fp16.engine"

python export_trt.py

mkdir -p /workspace/comfyui_launcher_models/tensorrt/depth-anything/
mv depth_anything_v2_vitb-fp16.engine /workspace/comfyui_launcher_models/tensorrt/depth-anything/

​
Step 4: Configure Network Tunneling

  1. Set up UDP forwarding on the remote server
# On RunPod instance
socat TCP4-LISTEN:4321,fork UDP4:127.0.0.1:5678
  1. Create SSH tunnel from local machine
# On local machine
ssh -L 4321:localhost:4321 {remote_user}@{remote_host}
socat UDP4-LISTEN:1234,fork TCP4:127.0.0.1:4321

​
Step 5: Launch ComfyStream Server

  1. Install dependencies and initialize workspace
cd /workspace/comfystream
pip install -r requirements.txt
python install.py --workspace /workspace/comfyui_launcher_projects/{workspace}/comfyui
  1. Start the server
python server/app.py --workspace /workspace/comfyui_launcher_projects/{workspace}/comfyui --media-ports=5678

​
Step 6: Run the Frontend

  1. Set up the local development environment
# On local machine
git clone https://github.com/yondonfu/comfystream.git
cd comfystream/ui
npm install --legacy-peer-deps
npm run dev
  1. Access the interface

  • Open localhost:3000 in your browser

  • Set stream URL to http://127.0.0.1:8888

  • Select the “depth anything” workflow

​
Testing Your Setup

  1. Open the ComfyUI interface and verify the DepthAnything node appears

  2. Test with a static image first to ensure model loading works

  3. Switch to live video input through the ComfyStream interface

​
Troubleshooting

  • If custom nodes don’t appear, restart ComfyUI and refresh the browser

  • Check RunPod container logs for node installation issues

  • Verify SSH tunnels are active using netstat or similar tools

​
Next Steps

  • Experiment with different ComfyUI workflows

  • Explore other real-time AI models

  • Join the Livepeer Real-time AI Video Showcase waitlist for updates

Performance recommendationsGet Pipeline