Uncertainty-Gated Large-Small Model Collaboration for Grounded Multimodal Scene Understanding under Ambiguity
DOI:
https://doi.org/10.66238/f04ent19Keywords:
Multimodal Scene Understanding, Large Vision-Language Models, Uncertainty Estimation, Model Collaboration, Grounded Visual ReasoningAbstract
Multimodal scene understanding requires intelligent systems to simultaneously interpret visual environments and comprehend natural language instructions. While contemporary large vision-language models demonstrate remarkable capabilities in complex visual reasoning and grounding, their immense computational requirements prohibit deployment in real-time or resource-constrained scenarios. Conversely, small lightweight models operate efficiently but frequently fail when confronted with highly ambiguous scenes, such as those featuring dense occlusion, semantic overlap, or intricate multi-object spatial relationships. This paper presents a novel uncertainty-gated collaborative framework that dynamically routes inference tasks between a computationally efficient small model and a highly capable large model. The proposed system employs a rigorous uncertainty quantification mechanism within the small model to evaluate epistemic and aleatoric ambiguity in real time. When the calculated uncertainty exceeds a dynamically optimized threshold, the system halts local processing and delegates the complex grounding task to the large model. This approach ensures that routine unambiguous scenes are processed locally with minimal latency, while complex ambiguous scenarios receive the necessary computational resources for accurate resolution. Extensive theoretical analysis and simulated deployment experiments demonstrate that the proposed uncertainty-gated collaboration achieves grounding accuracy commensurate with exclusively utilizing a large model, while reducing overall computational overhead by an average of sixty percent. The findings provide a critical pathway for the sustainable deployment of advanced multimodal artificial intelligence in autonomous systems, robotics, and interactive computational environments.
References
1. Chang Liu, Tianjiao Jing, Chengcheng Ma, XuanQi Zhou, Zhengxuan Lian, Qin Jin, Hongliang Yuan, and Shi-Sheng Huang. 2025. EmoDiffTalk:Emotion-aware
Diffusion for Editable 3D Gaussian Talking Head. CoRR abs/2512.05991 (2025).
arXiv:2512.05991 doi:10.48550/ARXIV.2512.05991
2. Li, Q., Liu, Z., Luo, W., Luo, T., & Hou, C. (2026). Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theory. arXiv preprint arXiv:2605.24602.
3. Zhang, W., Huang, M., Zhou, Y., Zhang, J., Yu, J., Wang, J., & Xu, L. (2024). Both2hands: Inferring 3d hands from both text prompts and body dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2393-2404).
4. Tian, Y., & Yang, Z. (2026, May). SAEC: scene-aware enhanced edge-cloud collaborative industrial vision inspection with multimodal LLM. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 19747-19751). IEEE.
5. Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. In Proceedings of CVPR.
6. Li, K., Jiang, L., Li, Z., Dong, J., Zhang, J., Yin, Y., ... & Chen, W. (2026). Scenix: Sparse-View 3D Scene Reconstruction via Executable Scene Programs. arXiv preprint arXiv:2608.07012.
7. Yang, D., Ai, H., Liu, J., & He, B. (2022). Absolute scale estimation for underwater monocular visual odometry based on 2-D imaging sonar. Measurement, 190, 110665.
8. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020). End-to-end object detection with transformers. In Proceedings of ECCV.
9. Qu, W., Wang, J., Gong, Y., Huang, X., & Xiao, L. (2025, June). An end-to-end robust point cloud semantic segmentation network with single-step conditional diffusion models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 27325-27335). IEEE.
10. Wang, Y., Yu, H., Tang, J., Gao, Y., Wang, X., Yue, Y., & Yang, Y. (2026). FilterGS: Traversal-Free Parallel Filtering and Adaptive Shrinking for Large-Scale LoD 3D Gaussian Splatting. arXiv preprint arXiv:2603.23891.
11. Fan, J., Gao, B., Jin, H., & Jiang, L. (2022). UCC: Uncertainty guided cross-head co-training for semi-supervised semantic segmentation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 9937–9946). IEEE. https://doi.org/10.1109/CVPR52688.2022.00971
12. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., et al. (2015). Going deeper with convolutions. In Proceedings of CVPR.
13. Meng, X., & Lin, J. (2026). AWCRL: Adaptive Weakly Supervised Cross-Modal Reliability Learning for resource-constrained proxy-state recognition. Pattern Recognition, 114998.
14. Wang, G., Ren, J., Shen, L., Cheah, W. P., & Qu, R. (2026). M-MambaS: Multimodal Mamba for small lesion segmentation. Pattern Recognition, 113923.
15. Li, Y., Tan, G., & Gou, C. (2024). Cascaded iterative transformer for jointly predicting facial landmark, occlusion probability and head pose. International Journal of Computer Vision, 132(4), 1242-1257.
16. Jielong Tang, Xujie Yuan, Jiayang Liu, Jianxing Yu, Xiao Dong, Lin Chen, Yunlai Teng, Shimin Di, and Jian Yin. 2026. Sake: Self-aware knowl- edge exploitation-exploration for grounded multi- modal named entity recognition. arXiv preprint arXiv:2604.20146.
17. Chen, Y., Xiang, F., Hernandez, M. L., Carpenter, D., Bozkurt, A., & Lobaton, E. (2025). Robust multimodal cough detection with optimized out-of-distribution detection for wearables. IEEE Journal of Biomedical and Health Informatics.
18. Fang, S., Shen, I., Wang, Y., Tsai, Y. H., Yang, Y., Zhou, S., ... & Yang, M. H. (2025). Meshllm: Empowering large language models to progressively understand and generate 3d mesh. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 14061-14072).
19. Z. Wang, R. Yuan, Z. Geng, H. Li, X. Qu, X. Li, S. Chen,
H. Fu, R. B. Dannenberg, and K. Zhang, ‘‘Singing timbre popularity
assessment based on multimodal large foundation model,’’ in Proc.
33rd ACM Int. Conf. Multimedia, Oct. 2025, pp. 12227–12236, doi:
10.1145/3746027.3758148.
20. Liu, Y., Cai, Q., Wang, C., Yang, J., Fan, H., Dong, J., & Chen, S. (2024). Geometry-enhanced attentive multi-view stereo for challenging matching scenarios. IEEE Transactions on Circuits and Systems for Video Technology, 34(8), 7401-7416.
21. Zhang, J., Shi, Y., Ma, Y., Xu, L., Yu, J., & Wang, J. (2023, June). Ikol: Inverse kinematics optimization layer for 3d human pose and shape estimation via gauss-newton differentiation. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37, No. 3, pp. 3454-3462).
22. Wang, Z., Xiong, F., Lin, L., Hu, X., Wang, Y., Wang, Y., ... & Chu, X. (2026, July). Visually-guided policy optimization for multimodal reasoning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 6646-6664).
23. Jia, X., Wang, Q., Wang, A., Wang, H., Gyenes, B., Gospodinov, E., ... & Neumann, G. (2026). Pointmappolicy: Structured point cloud processing for multi-modal imitation learning. Advances in Neural Information Processing Systems, 38, 160188-160216.
24. Qu, W., Shao, Y., Meng, L., Huang, X., & Xiao, L. (2024, June). A conditional denoising diffusion probabilistic model for point cloud upsampling. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 20786-20795). IEEE.
25. Sun, J., Lin, Y., Xue, Y., Wang, Y., Yao, Z., Qian, R., ... & Zhou, H. (2026). Pmmc: Prospective multimodal memory compilation for long-term lvlm agents. arXiv preprint arXiv:2608.00962.
26. Yang, Z., Hu, D., Guo, Q., Zuo, L., & Ji, W. (2023). Visual E 2 C: AI-driven visual end-edge-cloud architecture for 6G in low-carbon smart cities. IEEE Wireless Communications, 30(3), 204-210.
27. Wang, G., Li, Y., Chen, W., Ding, M., Cheah, W. P., Qu, R., ... & Shen, L. (2025, April). S³-mamba: Small-size-sensitive mamba for lesion segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 7, pp. 7655-7664).
28. Cong, P., Xu, Y., Ren, Y., Zhang, J., Xu, L., Wang, J., ... & Ma, Y. (2023, June). Weakly supervised 3d multi-person pose estimation for large-scale scenes based on monocular camera and single lidar. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 1, pp. 461-469).
29. Jing, G., Gao, P., Lee, Y., Hu, Y., & Zhang, H. (2025). 3D-aided pedestrian representation learning for video-based person re-identification. IEEE Transactions on Circuits and Systems for Video Technology, 35 (12), 12830-12845.
30. Wang, J., Malawade, A. V., Zhou, J., Yu, S. Y., & Al Faruque, M. A. (2024, January). Rs2g: Data-driven scene-graph extraction and embedding for robust autonomous perception and scenario understanding. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (pp. 7478-7487). IEEE.
31. Fan, Y., Tang, Z., Miyazaki, T., Huang, Y., & Omachi, S. (2026). A real-driven image synthesis framework for adapting pre-trained scene text detectors. Image and Vision Computing, 106027.
32. Wang, Y., Li, F. W., & Wang, J. (2026, May). Motion-Guided Semantic Alignment with Negative Prompts for Zero-Shot Video Action Recognition. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 10712-10716). IEEE.
33. Yang, Z., Ji, W., Guo, Q., & Wang, Z. (2023, October). Javp: Joint-aware video processing with edge-cloud collaboration for dnn inference. In Proceedings of the 31st ACM International Conference on Multimedia (pp. 9152-9160).
34. Meng, X., & Lin, J. (2026). CARTN: Counterfactual Audit and Repair for Trust-Aware Multi-Signal Streaming LiDAR Segmentation. Information Fusion, 104809.
35. Lyu, S., Mao, S., & Ma, Y. (2025). Unified Long Video Inpainting and Outpainting via Overlapping High-Order Co-Denoising. arXiv preprint arXiv:2511.03272.
36. Yang, Y., & Zhang, S. (2025). Pixel-wise calibration for a multi-focus microscopic 3D imaging system.Optics and Lasers in Engineering,194, 109127.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Eric Tam, Lok-Yin Wong (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles published in the International Journal of Machine Learning and Engineering (IJMLE) are licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).
This license permits unrestricted use, distribution, reproduction, adaptation, and commercial use in any medium or format, provided the original author(s) and the source are properly credited.
Authors retain the copyright of their work while granting the journal the right of first publication.
To view a copy of this license, please visit:
https://creativecommons.org/licenses/by/4.0/
