This guide walks through setting up a local real-time AI video pipeline using ComfyUI and ComfyStream. You’ll learn how to process live video with AI models, using depth map generation as an example.
Prerequisites
-
A RunPod account with access to GPU instances
-
Basic familiarity with terminal commands
-
SSH key pair for secure connections
-
Node.js and npm installed locally
Technical Architecture
Local Machine RunPod Instance
┌─────────────┐ SSH Tunnel ┌─────────────┐
│ Browser UI │◄──────────────► │ ComfyUI │
│ (localhost) │ │ │
└─────────────┘ │ ComfyStream│
└─────────────┘
Step 1: Set Up ComfyUI Environment
- Launch a RunPod instance with ComfyUI-Launcher
# On RunPod instance
cd /workspace
git clone https://github.com/ComfyWorkflows/ComfyUI-Launcher.git
cd ComfyUI-Launcher
./setup.sh
- Configure workspace directories
# Directories will be created automatically
/workspace/comfyui_launcher_models # Shared models
/workspace/comfyui_launcher_projects # Workspace projects
Note: Install at least one model through the ComfyUI manager GUI to enable the “Load Checkpoint” node.
Step 2: Install ComfyStream
- Set up Conda environment
cd /workspace
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
eval "$(/root/miniconda3/bin/conda shell.bash hook)"
# Create and activate environment
conda create -n comfystream python=3.11
conda activate comfystream
# Install PyTorch
conda install pytorch torchvision -c pytorch
- Install ComfyStream
cd /workspace
git clone https://github.com/yondonfu/comfystream.git
cd comfystream
git fetch
git checkout pin-opentelemetry
pip install .
# Install tensor utilities
cp -r nodes/tensor_utils /workspace/comfyui_launcher_projects/{workspace}/comfyui/custom_nodes
Step 3: Install DepthAnything TensorRT Node
- Clone and install the node
cd /workspace/comfyui_launcher_projects/{workspace}/comfyui/custom_nodes
git clone https://github.com/yuvraj108c/ComfyUI-Depth-Anything-Tensorrt.git
cd ComfyUI-Depth-Anything-Tensorrt
pip install -r requirements.txt
- Build TensorRT engine
wget https://huggingface.co/yuvraj108c/Depth-Anything-2-Onnx/resolve/main/depth_anything_v2_vitb.onnx?download=true
mv depth_anything_v2_vitb.onnx?download=true depth_anything_v2_vitb.onnx
# Update export_trt.py with:
# onnx_path = "./depth_anything_v2_vitb.onnx"
# trt_path = "depth_anything_v2_vitb-fp16.engine"
python export_trt.py
mkdir -p /workspace/comfyui_launcher_models/tensorrt/depth-anything/
mv depth_anything_v2_vitb-fp16.engine /workspace/comfyui_launcher_models/tensorrt/depth-anything/
- Set up UDP forwarding on the remote server
# On RunPod instance
socat TCP4-LISTEN:4321,fork UDP4:127.0.0.1:5678
- Create SSH tunnel from local machine
# On local machine
ssh -L 4321:localhost:4321 {remote_user}@{remote_host}
socat UDP4-LISTEN:1234,fork TCP4:127.0.0.1:4321
Step 5: Launch ComfyStream Server
- Install dependencies and initialize workspace
cd /workspace/comfystream
pip install -r requirements.txt
python install.py --workspace /workspace/comfyui_launcher_projects/{workspace}/comfyui
- Start the server
python server/app.py --workspace /workspace/comfyui_launcher_projects/{workspace}/comfyui --media-ports=5678
Step 6: Run the Frontend
- Set up the local development environment
# On local machine
git clone https://github.com/yondonfu/comfystream.git
cd comfystream/ui
npm install --legacy-peer-deps
npm run dev
- Access the interface
-
Open localhost:3000
in your browser
-
Set stream URL to http://127.0.0.1:8888
-
Select the “depth anything” workflow
Testing Your Setup
-
Open the ComfyUI interface and verify the DepthAnything node appears
-
Test with a static image first to ensure model loading works
-
Switch to live video input through the ComfyStream interface
Troubleshooting
-
If custom nodes don’t appear, restart ComfyUI and refresh the browser
-
Check RunPod container logs for node installation issues
-
Verify SSH tunnels are active using netstat
or similar tools
Next Steps
-
Experiment with different ComfyUI workflows
-
Explore other real-time AI models
-
Join the Livepeer Real-time AI Video Showcase waitlist for updates