VIL advances computer vision and deep learning for visual perception, multimodal 3D understanding, autonomous systems, and efficient image/video intelligence.
Our work connects visual perception, efficient learning, video intelligence, and 3D scene understanding for real-world systems.
Recent research spans real-time semantic segmentation, universal image segmentation, multimodal 3D detection, and video object detection.
CGSeg is a one-stage framework for open-vocabulary semantic segmentation that leverages pretrained vision-language models to segment arbitrary categories specified by text prompts. It introduces a gated fusion module to adaptively refine image and text features before constructing the cost volume, and a cross-aggregation module to enhance interactions between spatial and class representations. These designs improve image-text alignment and segmentation accuracy without relying on complex decoders. CGSeg achieves strong performance on standard benchmarks including ADE20K, Pascal Context, and Pascal VOC.
DynaFusion is a difficulty-aware multimodal framework for 3D semantic segmentation that fuses LiDAR and RGB information for autonomous driving. It selectively refines challenging and uncertain points using saliency-guided fusion, deformable attention, and local context modeling, allowing computation to focus on the most informative regions. This design improves segmentation accuracy while maintaining an efficient accuracy–latency trade-off.
EMSFormer is an efficient Transformer-based framework designed for real-time semantic segmentation. It introduces multi-scale single-head attention to capture rich contextual information with lower computational cost, and cross-resolution single-head attention to efficiently fuse global and high-resolution features. With its dual-resolution architecture, EMSFormer achieves a strong balance between segmentation accuracy and inference speed on ADE20K, Cityscapes, and CamVid.
AFA is a lightweight framework for universal image segmentation that supports multiple segmentation tasks within a unified model. It introduces a learnable task token to adaptively capture task-specific characteristics and an adaptive feature alignment module to efficiently bridge visual and textual representations. This design improves multi-task segmentation performance without relying on large modality encoders or costly alignment strategies, making it well suited for efficient and resource-constrained applications.
BEVFusion-DHIP is a multimodal 3D object detection framework designed to reduce false negatives in autonomous driving, particularly for small, distant, and occluded objects. It introduces dual hard instance probing to progressively refine challenging objects in both LiDAR and image features, together with a deformable attention fusion network to improve spatial alignment and interaction between modalities. The framework achieves strong performance on the nuScenes benchmark, improving both detection accuracy and robustness over the BEVFusion baseline.
EQO is an efficient multimodal framework for universal image segmentation that supports semantic, instance, and panoptic segmentation within a unified architecture. It simplifies text-based query optimization by encoding inter-class and inter-task information into a compact textual representation, while introducing an attention-based contrastive loss to improve query learning through one-to-many matching. This design reduces computational complexity while achieving strong performance across multiple segmentation tasks on ADE20K.
This project introduces a lightweight semantic layer for Transformer-based semantic segmentation to better capture local details and semantic context during feature encoding. The proposed module uses multi-scale depthwise convolutions and is integrated into different stages of a hierarchical Transformer backbone, improving the representation of fine-grained structures and small objects. It can be combined with common segmentation frameworks such as Semantic-FPN and Mask Transformers, achieving strong performance on ADE20K and Cityscapes.
VoTr-DADA is a Transformer-based 3D object detection framework that enhances voxel feature learning with density-aware deformable attention. Instead of relying on a predefined receptive field, the proposed DADA module uses voxel density information to identify important regions and adaptively sample informative features. This design provides more flexible long-range modeling while maintaining efficient inference, achieving improved detection performance on the KITTI and Waymo Open datasets.
This project enhances Mask2Former with auxiliary convolutional branches to improve local feature learning and small-object segmentation. During training, the auxiliary branches capture dense local details that complement the global representations learned by Transformers. Since these branches are removed during inference, the model gains improved segmentation accuracy without introducing additional inference cost. The proposed approach achieves strong performance on ADE20K and Cityscapes.
This project develops an event-aware video object detection framework for challenging conditions such as occlusion, large motion, and aspect-ratio changes. It combines an event-aware ConvLSTM to highlight difficult temporal events with an object relation module that selectively aggregates useful features from reference frames. By exploiting temporal context more effectively, the framework improves object detection robustness in complex video scenarios.
This project develops an efficient object detection framework for autonomous driving with a focus on multi-class and multi-scale objects in complex street scenes. It introduces location-aware deformable convolution to adaptively capture informative contextual features and backward attention filtering to enhance useful high-resolution features while suppressing distracting low-level information. The proposed approach improves detection accuracy across varying object scales while maintaining fast inference.
This project develops a causal online video object detection framework that uses only the current frame and preceding frames, making it suitable for real-time applications. It combines optical-flow-based feature warping for short-term temporal context with a temporal ConvLSTM for long-term feature propagation. By integrating both short- and long-term temporal information, the framework achieves accurate and efficient video object detection without relying on future frames.
This project improves one-stage object detection by introducing a weakly supervised auxiliary multi-label segmentation branch during training. The auxiliary task helps the detector learn richer spatial and semantic information, particularly for handling occluded objects, without requiring expensive pixel-level annotations. Since the segmentation branch is removed during inference, the method improves detection accuracy without adding computational cost and can be integrated with popular detectors such as RetinaNet, YOLO, and SSD.
This project develops a two-path ConvLSTM pyramid network to improve video object detection under drastic object scale changes caused by camera motion. The framework propagates temporal context through both top-down and bottom-up pathways, allowing information to flow across multiple feature scales over time. By combining multi-scale temporal modeling with feature pyramids, the method improves robustness to changing object sizes and can be integrated with various still-image object detectors.
This project develops a semantic segmentation framework that combines multi-scale contextual modeling with attention-guided feature fusion. It introduces a Mixed Spatial Pyramid Pooling (MSPP) module to capture dense contextual information at multiple scales and a Global-Attention Fusion (GAF) module to refine low-level features using global semantic guidance. The framework improves object detail recovery and segmentation accuracy while maintaining a simple and effective architecture.
VIL collaborates with research institutes and industry partners on visual intelligence, autonomous driving, video systems, and embedded AI.
Development of an embedded AI model for road surface condition analysis
Total budget: $1,625,700
Software and hardware development of cooperative autonomous driving control platform for commercial special and work-assist vehicles
Camera-based artificial intelligence system for autonomous cars
Enhancement of 3D model reconstruction technology,
Stereo-based pedestrian detection and tracking algorithms for advanced driver assistant systems,
Power-optimized distributed video codec design for wireless video sensor networks,
Hardware/software co-design of low-complexity video coding algorithms for wireless video surveillance,
Earlier work established a foundation in 3D reconstruction, video coding, wireless multimedia, and robust visual communication.
Based on the popular two-stage detector (Faster R-CNN) and one-stage detector (SSD), our research focuses on improving the detection accuracy and the processing speed using spatial and temporal context information. The spatial context information extraction is carried out using spatial recurrent neural network and location-aware deformable convolution. Short-term and long-term temporal context information is modeled by optical flow and convolutional LSTM.
Our research speeds up the detection process by utilizing stereo camera and depth information to detect the ground plane where pedestrians would normally stand upon. Thus, the ROI search range is greatly reduced, and the detection can be performed at real-time speed. Our research also features depth map estimation (stereo matching) based on modified Census transform and mutual information which is fast and robust against the brightness inconsistency in the outdoor environment. The pedestrian detection system is implemented on the low-profile mini-PC and tested in the Chicago urban area.
3D reconstruction like KinectFusion has been a well-established area of study in the field of robotics and computer vision. The objective is to recreate a real-world scene, and it has applications in areas like augmented reality (AR), robotic teleoperation, medical analysis, video games, etc. We focus on improving the performance of 3D reconstruction for challenging tasks such as moving object segmentation, tracking fast camera motion, modeling complex environments, etc.
DVC is commonly referred to as the reverse paradigm of conventional video coding where the complexity is transferred from the encoder to the decoder. The target applications are video surveillance cameras, video conferencing and remote video sensors which need to perform video encoding on resource constrained devices.
Multiple description coding in DVC is a an effort to reduce the effects of channel losses in DVC By intentionally introducing redundancy into the encoded bitstreams, the ability of the codec to withstand channel losses are increased.
Multiview video coding (MVC) is a hot research topic these owing to the advances in processing power and algorithms to process multiview video content. The lab focuses on all aspects of MVC starting from Depth extraction from a pair of stereoscopic images. Multiview video coding focusing on compression of depth maps, layered depth images and layered depth videos. View synthesis at arbitrary camera locations using concepts of depth image based rendering.
Anomaly detection in video is a very important topic that the lab is conducting research into. Using concepts from machine learning, pattern recognition and stochastic processes, research is being conducted to extract useful features from video. Such research will be very useful to law enforcement authorities for automatic detection of anomalies in surveillance cameras distributed around a city.
Research is being conducted into improvement of H.264/AVC, joint source-channel coding, rate control and error control, error concealment, video distortion estimation and quality assessment.
Quality of Service provision for video transmission in wireless networks, multi-path transport over wireless ad-hoc networks, multimedia transport protocols.
Application-layer techniques for adaptive video transmission over wireless networks, adaptive cross-layer error protection, adaptive link-layer techniques, power-distortion optimized routing and scheduling.
For research collaboration and student opportunities, contact Prof. Joohee Kim.