wavespeed-ai/flux-controlnet-union-pro-2.0

FLUX.1-dev-ControlNet-Union-Pro-2.0 is an enhanced unified ControlNet for FLUX.1-dev model released by Shakker Labs. This version offers significant improvements over the previous Pro version with better performance and control capabilities.

Key Improvements (vs Pro 1.0)

• Smaller Model Size: Removed mode embedding for reduced memory footprint
• Enhanced Control: Improved Canny and Pose control with better aesthetics
• New Soft Edge Support: Added AnylineDetector-based soft edge control
• Streamlined Architecture: Simplified from 12 modes to 5 optimized modes

Technical Specifications

• Architecture: 6 double blocks + 0 single blocks (mode embedding removed)
• Training: 300k steps on 20M high-quality general and human images
• Resolution: 512x512 training resolution
• Precision: BFloat16
• Batch Size: 128
• Learning Rate: 2e-5
• Guidance Range: [1, 7] uniformly sampled
• Text Drop Ratio: 0.20

Supported Control Modes

1. Canny Edge Detection

• Detector: cv2.Canny
• Recommended Settings: conditioning_scale=0.7, guidance_end=0.8
• Use Case: Precise edge-based control for structural guidance

2. Soft Edge Detection

• Detector: AnylineDetector
• Recommended Settings: conditioning_scale=0.7, guidance_end=0.8
• Use Case: Softer, more natural edge detection for artistic control

3. Depth Control

• Detector: depth-anything
• Recommended Settings: conditioning_scale=0.8, guidance_end=0.8
• Use Case: 3D depth-aware image generation

4. Human Pose Control

• Detector: DWPose
• Recommended Settings: conditioning_scale=0.9, guidance_end=0.65
• Use Case: Precise human pose and body structure control

5. Grayscale Control

• Detector: cv2.cvtColor
• Recommended Settings: conditioning_scale=0.9, guidance_end=0.8
• Use Case: Grayscale-to-color generation with structural preservation

Usage Guidelines

• Detailed Prompts: Use detailed text prompts for better stability
• Multi-Condition Support: Can be combined with other ControlNets
• Parameter Tuning: Adjust conditioning_scale and control_guidance_end for optimal results
• Quality Input: Higher quality control images produce better results

Performance Optimizations

Our WavespeedAI implementation includes:

• Memory Optimization: Efficient GPU memory management for the streamlined architecture
• Pipeline Acceleration: Optimized inference pipeline leveraging the simplified model structure
• Dynamic Batching: Intelligent batching for improved throughput
• Model Compilation: XeLerate-powered model compilation for faster inference

Professional Applications

• Architectural Visualization: Depth and edge control for building renders
• Character Design: Pose control for consistent character positioning
• Art Direction: Soft edge control for concept development
• Photography Enhancement: Grayscale colorization and structure preservation
• Digital Art Creation: Combined control modes for artistic workflows

Limitations

• Control Quality Dependency: Output quality depends on control image precision
• Prompt Sensitivity: Results are influenced by both control inputs and text prompts
• Removed Modes: No longer supports Tile mode (removed in Pro 2.0)
• Memory Requirements: Still requires significant GPU memory for high-resolution outputs

This model represents the state-of-the-art in unified ControlNet technology, offering professional-grade control with improved efficiency and quality.