ColonNeRF: High-fidelity neural reconstruction of long colonoscopy

Yufei Shi1,4,* Beijia Lu1,* Jia-Wei Liu2,† Ming Li3 Si Yong Yeo4 Mike Zheng Shou1,✉
1Show Lab, National University of Singapore    2Institute of Data Science, National University of Singapore    3Northwestern University    4MedVisAI Lab, Lee Kong Chian School of Medicine, Nanyang Technological University
📄 Paper 📚 arXiv 🎥 Video 📊 Data 🌐 Original Project Page
Video demonstration of ColonNeRF's high-fidelity neural reconstruction capabilities

Abstract

ColonNeRF introduces a novel approach to high-fidelity neural reconstruction of long colonoscopy sequences using Neural Radiance Fields (NeRF). Traditional colonoscopy reconstruction methods face significant challenges due to the complex geometric structures of the colon, dissimilarity among colon segments, and sparse viewpoint constraints.

Our method leverages Neural Radiance Fields to achieve comprehensive 3D reconstruction of the entire colon structure from endoscopic video sequences. The approach incorporates region division and integration modules, multi-level fusion techniques, and semantic consistency-guided pose densification to overcome the inherent challenges in colonoscopy reconstruction.

ColonNeRF demonstrates superior performance on benchmark datasets, achieving significant improvements in reconstruction quality and providing detailed geometric and texture representation of colon structures for enhanced medical diagnosis and surgical planning.

Key Contributions

  • Novel NeRF-based Framework: First neural radiance field approach specifically designed for comprehensive colonoscopy reconstruction
  • Piecewise Colon Reconstruction: Innovative region division and integration strategy to handle dissimilar colon segments
  • Progressive Geometry Modeling: Multi-level fusion module for handling complex geometric structures
  • DensiNet Integration: Semantic consistency-guided pose densification for improved reconstruction quality

Method Overview

ColonNeRF employs a Neural Radiance Field (NeRF) architecture specifically adapted for the unique challenges of colonoscopy reconstruction:

ColonNeRF Architecture Overview
Figure 1. Overview of ColonNeRF. The architecture comprises a region division module, depicted in the upper section, where orange areas illustrate transition zones between adjacent regions. Each region includes a core area (red) and an adjacent transition zone, processed through various sparsity levels to produce coarse, medium, and fine data. These data feed into a multi-level fusion module, with each stage containing a DensiNet module for data augmentation. Within DensiNet module, we input the helix rotating pose, original pose, and spinning around pose into the MipNeRF to optimize intestinal geometry learning. A DINO-ViT module is included for supervised training. Following processing through this module, final color, density, and transparency are determined, and the region integration module executes information filtering, fusion, and rendering across all blocks.

The framework incorporates the following key components:

  • Region Division Module: Segments the colon into manageable regions to handle structural dissimilarity
  • Integration Framework: Combines multiple region reconstructions into a coherent 3D model
  • Multi-level Fusion: Addresses complex geometric structures through progressive modeling
  • DensiNet: Densifies camera poses using semantic consistency guidance
Intestinal Seamless Integration Module
Figure 2. Detailed depiction of our intestinal seamless integration module. The module first evaluates the distance from the line connecting the centers of two blocks to the target view. Blocks exceeding the specified distance threshold, represented by the red area, are filtered out. The remaining blocks undergo visibility prediction, with blocks demonstrating visibility below a certain threshold excluded. The final remaining blocks are seamlessly integrated using Inverse Distance Weighting (IDW), producing our final results.

The framework addresses three key challenges in colonoscopy reconstruction: dissimilarity among colon segments, complex geometric structures, and sparse viewpoint constraints. By leveraging the implicit representation capabilities of NeRF, ColonNeRF achieves high-fidelity reconstruction of the entire colon structure.

Results & Performance

ColonNeRF demonstrates state-of-the-art performance on colonoscopy reconstruction benchmarks:

  • SimCol-to-3D Dataset: LPIPS-ALEX scores improved by 67%-85% compared to existing methods
  • Texture Quality: Significant enhancement in texture clarity and detail preservation
  • Geometric Accuracy: Superior reconstruction of complex colon geometric structures
  • Long-sequence Handling: Robust performance on extended colonoscopy sequences
ColonNeRF Comparative Results
Figure 3. Novel view synthesis RGB and depth results of different methods, including five different baselines on the three datasets. The 1, 4, 7 rows display the depth images, and 2, 5, 8 rows display the corresponding RGB images. The 3, 6, 9 lines show the enlarged details of the rendered image. ColonNeRF consistently outperforms baseline methods by providing reliably constructed details, textures, and a better understanding of geometry.
ColonNeRF Additional Comparative Results
Figure 4. Novel view synthesis RGB and depth results of different methods, including five different baselines on the three different datasets Sigmoid, Cecum, and transcending colon. The 1, 4, 7 rows display the depth images, and 2, 5, 8 rows display the corresponding RGB images. The 3, 6, and 9 lines show the enlarged details of the rendered image. ColonNeRF consistently outperforms baseline methods by providing reliably constructed details, textures, and a better understanding of geometry.

The method's ability to achieve high-fidelity reconstruction of long colonoscopy sequences makes it particularly valuable for medical applications requiring detailed 3D visualization of colon structures for diagnosis and surgical planning.

Technical Innovation

ColonNeRF introduces several technical innovations to address the unique challenges of colonoscopy reconstruction:

  • Adaptive Region Processing: Handles varying colon segment characteristics through intelligent region division
  • Semantic Pose Densification: Improves viewpoint coverage using semantic consistency guidance
  • Progressive Reconstruction: Multi-level approach to handle complex anatomical structures
  • Integration Strategy: Seamless combination of region-based reconstructions into unified models

Impact & Applications

ColonNeRF represents a significant advancement in medical imaging and 3D reconstruction, with applications including:

  • Medical Diagnosis: Enhanced visualization for improved polyp detection and diagnosis
  • Surgical Planning: Detailed 3D models for pre-operative planning and simulation
  • Medical Education: High-fidelity 3D reconstructions for training and educational purposes
  • Clinical Documentation: Comprehensive 3D records of colonoscopy examinations

Citation

@article{shi2025colonnerf,
  title={ColonNeRF: High-fidelity neural reconstruction of long colonoscopy},
  author={Shi, Yufei and Lu, Beijia and Liu, Jia-Wei and Li, Ming and Yeo, Si Yong and Shou, Mike Zheng},
  journal={Neurocomputing},
  pages={131445},
  year={2025},
  publisher={Elsevier}
}

Published in Neurocomputing 2025