Available ComfyUI nodes

This guide covers the available ComfyUI nodes for creating real-time video pipelines.

This guide covers the available nodes and requirements for creating real-time video pipelines using ComfyUI with Livepeer.

Video Input/Output Nodes - REQUIRED FOR ALL PIPELINES

ComfyStream

• Github Link
• Input:
  • Video stream URL or device ID
  • Optional configuration parameters
• Output:
  • RGB frame tensor (3, H, W)
  • Frame metadata (timestamp, index)
• Performance Requirements:
  • Frame processing time: < 5ms
  • VRAM usage: < 500MB
  • Buffer size: ≤ 2 frames
  • Supported formats: RTMP, WebRTC, V4L2
• Best Practices:
  • Set fixed frame rate

Analysis Nodes

Depth Anything TensorRT

• Github Link
• Input: RGB frame (3, H, W)
• Output: Depth map (1, H, W)
• Performance Requirements:
  • Inference time: < 20ms
  • VRAM usage: 2GB
  • Batch size: 1
• Best Practices:
  • Place early in workflow
  • Cache results for static scenes
  • Use lowest viable resolution

Segment Anything 2

• (Github Link)[https://github.com/kijai/ComfyUI-segment-anything-2]
• Input: RGB frame (3, H, W)
• Output: Segmentation mask (1, H, W)
• Performance Requirements:
  • Inference time: < 30ms
  • VRAM usage: 3GB
  • Batch size: 1
• Best Practices:
  • Cache static masks
  • Use mask erosion for stability
  • Implement confidence thresholding

Florence2

• Github Link
• Input: RGB frame (3, H, W)
• Output: Feature vector (1, 512)
• Performance Requirements:
  • Inference time: < 15ms
  • VRAM usage: 1GB
  • Batch size: 1
• Best Practices:
  • Cache embeddings for known references
  • Use cosine similarity for matching
  • Implement feature vector normalization

Generation and Control Nodes

LivePortraitKJ

• Github Link
• Input:
  • Source image (3, H, W)
  • Driving frame (3, H, W)
• Output: Animated frame (3, H, W)
• Performance Requirements:
  • Inference time: < 50ms
  • VRAM usage: 4GB
  • Batch size: 1
• Best Practices:
  • Pre-process source images
  • Implement motion smoothing
  • Cache facial landmarks

ComfyUI Diffusers

• (Github Link)[https://github.com/Limitex/ComfyUI-Diffusers]
• Input:
  • Conditioning tensor
  • Latent tensor
• Output: Generated frame (3, H, W)
• Performance Requirements:
  • Inference time: < 50ms
  • VRAM usage: 4GB
  • Maximum steps: 20
• Best Practices:
  • Use TensorRT optimization
  • Implement denoising strength control
  • Cache conditioning tensors

Supporting Nodes

K Sampler

• Input:
  • Latent tensor
  • Conditioning
• Output: Sampled latent
• Performance Requirements:
  • Maximum steps: 20
  • VRAM usage: 2GB
  • Scheduler: euler_ancestral
• Best Practices:
  • Use adaptive step sizing
  • Cache conditioning tensors

Prompt Control

• Input: Text prompts
• Output: Conditioning tensors
• Performance Requirements:
  • Processing time: < 5ms
  • VRAM usage: minimal
• Best Practices:
  • Cache common prompts
  • Use consistent style tokens
  • Implement prompt weighting

VAE

• Input: Latent tensor
• Output: RGB frame
• Performance Requirements:
  • Inference time: < 10ms
  • VRAM usage: 1GB
  • Tile size: 512
• Best Practices:
  • Use tiling for large frames
  • Implement half-precision
  • Cache common latents

IPAdapter

• Input:
  • Reference image
  • Target tensor
• Output: Conditioned tensor
• Performance Requirements:
  • Inference time: < 20ms
  • VRAM usage: 2GB
  • Reference resolution: ≤ 512x512
• Best Practices:
  • Cache reference embeddings
  • Use consistent weights
  • Implement cross-attention

Cache Nodes

• Input: Any tensor
• Output: Cached tensor
• Performance Requirements:
  • Access time: < 1ms
  • Maximum size: 2GB
  • Cache type: GPU
• Best Practices:
  • Implement LRU eviction
  • Monitor cache pressure
  • Clear on scene changes

ControlNet

• Input:
  • Control signal
  • Target tensor
• Output: Controlled tensor
• Performance Requirements:
  • Inference time: < 30ms
  • VRAM usage: 2GB
  • Resolution: ≤ 512
• Best Practices:
  • Use adaptive conditioning
  • Implement strength scheduling
  • Cache control signals

Default Nodes

All default nodes that ship with ComfyUI are available. The list below is subject to change.

• AlignYourStepsScheduler
• BasicGuider
• BasicScheduler
• BetaSamplingScheduler
• Canny
• CFGGuider
• CheckpointLoader
• CheckpointLoaderSimple
• CheckpointSave
• CLIPAdd
• CLIPAttentionMultiply
• CLIPLoader
• CLIPMergeSimple
• CLIPSave
• CLIPSetLastLayer
• CLIPSubtract
• CLIPTextEncode
• CLIPTextEncodeControlnet
• CLIPTextEncodeFlux
• CLIPTextEncodeHunyuanDiT
• CLIPTextEncodeSD3
• CLIPTextEncodeSDXL
• CLIPTextEncodeSDXLRefiner
• CLIPVisionEncode
• CLIPVisionLoader
• ConditioningAverage
• ConditioningCombine
• ConditioningConcat
• ConditioningSetArea
• ConditioningSetAreaPercentage
• ConditioningSetAreaStrength
• ConditioningSetMask
• ConditioningSetTimestepRange
• ConditioningZeroOut
• ControlNetApply
• ControlNetApplyAdvanced
• ControlNetApplySD3
• ControlNetInpaintingAliMamaApply
• ControlNetLoader
• CropMask
• DifferentialDiffusion
• DiffControlNetLoader
• DiffusersLoader
• DisableNoise
• DualCFGGuider
• DualCLIPLoader
• EmptyImage
• EmptyLatentAudio
• EmptyLatentImage
• EmptyMochiLatentVideo
• EmptySD3LatentImage
• ExponentialScheduler
• FeatherMask
• FlipSigmas
• FluxGuidance
• FreeU
• FreeU_V2
• GITSScheduler
• GLIGENLoader
• GLIGENTextBoxApply
• GrowMask
• HypernetworkLoader
• HyperTile
• ImageBatch
• ImageBlend
• ImageBlur
• ImageCompositeMasked
• ImageColorToMask
• ImageCrop
• ImageFromBatch
• ImageInvert
• ImageOnlyCheckpointLoader
• ImageOnlyCheckpointSave
• ImagePadForOutpaint
• ImageQuantize
• ImageScale
• ImageScaleBy
• ImageScaleToTotalPixels
• ImageSharpen
• ImageToMask
• ImageUpscaleWithModel
• InpaintModelConditioning
• InstructPixToPixConditioning
• InvertMask
• JoinImageWithAlpha
• KarrasScheduler
• KSampler
• KSamplerAdvanced
• KSamplerSelect
• LaplaceScheduler
• LatentAdd
• LatentApplyOperation
• LatentApplyOperationCFG
• LatentBatch
• LatentBatchSeedBehavior
• LatentBlend
• LatentComposite
• LatentCompositeMasked
• LatentCrop
• LatentFlip
• LatentFromBatch
• LatentInterpolate
• LatentMultiply
• LatentOperationSharpen
• LatentOperationTonemapReinhard
• LatentRotate
• LatentSubtract
• LatentUpscale
• LatentUpscaleBy
• LoadAudio
• LoadImage
• LoadImageMask
• LoadLatent
• LoraLoader
• LoraLoaderModelOnly
• LoraSave
• MaskComposite
• MaskToImage
• ModelAdd
• ModelMergeBlocks
• ModelMergeFlux1
• ModelMergeSD1
• ModelMergeSD2
• ModelMergeSD35_Large
• ModelMergeSD3_2B
• ModelMergeSDXL
• ModelMergeSimple
• ModelSamplingAuraFlow
• ModelSamplingContinuousEDM
• ModelSamplingContinuousV
• ModelSamplingDiscrete
• ModelSamplingFlux
• ModelSamplingSD3
• ModelSamplingStableCascade
• ModelSave
• ModelSubtract
• Morphology
• PatchModelAddDownscale
• PerpNeg
• PerpNegGuider
• PerturbedAttentionGuidance
• PhotoMakerEncode
• PhotoMakerLoader
• PolyexponentialScheduler
• PorterDuffImageComposite
• PreviewAudio
• PreviewImage
• RandomNoise
• RebatchImages
• RebatchLatents
• RepeatImageBatch
• RepeatLatentBatch
• RescaleCFG
• SamplerCustom
• SamplerCustomAdvanced
• SamplerDPMAdaptative
• SamplerDPMPP_2M_SDE
• SamplerDPMPP_2S_Ancestral
• SamplerDPMPP_3M_SDE
• SamplerDPMPP_SDE
• SamplerEulerAncestral
• SamplerEulerAncestralCFGPP
• SamplerEulerCFGpp
• SamplerLCMUpscale
• SamplerLMS
• SaveAnimatedPNG
• SaveAnimatedWEBP
• SaveAudio
• SaveImage
• SaveLatent
• SD_4XUpscale_Conditioning
• SDTurboScheduler
• SelfAttentionGuidance
• SetLatentNoiseMask
• SetUnionControlNetType
• SkipLayerGuidanceSD3
• SolidMask
• SplitImageWithAlpha
• SplitSigmas
• SplitSigmasDenoise
• StableCascade_EmptyLatentImage
• StableCascade_StageB_Conditioning
• StableCascade_StageC_VAEEncode
• StableCascade_SuperResolutionControlnet
• StableZero123_Conditioning
• StableZero123_Conditioning_Batched
• StyleModelApply
• StyleModelLoader
• SV3D_Conditioning
• SVD_img2vid_Conditioning
• ThresholdMask
• TomePatchModel
• TorchCompileModel
• TripleCLIPLoader
• unCLIPCheckpointLoader
• unCLIPConditioning
• UNETLoader
• UNetCrossAttentionMultiply
• UNetSelfAttentionMultiply
• UNetTemporalAttentionMultiply
• UpscaleModelLoader
• VAEDecode
• VAEDecodeAudio
• VAEDecodeTiled
• VAEEncode
• VAEEncodeAudio
• VAEEncodeForInpaint
• VAEEncodeTiled
• VAESave
• VideoLinearCFGGuidance
• VideoTriangleCFGGuidance
• VPScheduler
• WebcamCapture