The objective of this project is to develop principled mathematical, computational and systems components in order to construct the next generation of autonomous vehicles capable of integrated visual perception (scene reconstruction and recognition) and action (planning and navigation) based on computer vision, machine learning, and optimal control techniques. A central contribution of this work is the development of fully trainable, large scale semantic architectures based on deep neural networks that enable the complete, end-to-end, training of the geometric, categorization and navigation parameters of the model in a single optimization process. By integrating and advancing components within computer vision, machine learning and optimal control, we will be able to develop perceptual robotics systems that can semantically map, navigate, and interact with an unknown environment. For demonstration, we will develop an autonomous system for the visual inspection of a forest using small UAVs (quadcopters), including classifying different types of trees, estimating their age and counting their numbers based on geometric and semantic information, as well as avoiding or following people. The demonstrator is interesting in its own right, but represents only a testbed for the methodology developed in the project, which is applicable broadly, to autonomous vehicles, humanoid robots, surveillance and security, or flexible inspection in general.

This project is a collaboration between the “Simion Stoilow” Mathematics Institute of the Romanian Academy (IMAR) and the Image Processing and Pattern Recognition Research Center (IPPRRC) at the Technical University of Cluj-Napoca. Funding was provided by the Romanian Ministry of Education and Research through research grant PN-III-P4-ID-PCCF-2016-0180.

Technical University of Cluj-Napoca Team

Team Leader: Prof. Dr. Sergiu Nedevschi

Prof. Dr. Florin Oniga

Dr. Robert Varga

Dr. Vlad Miclea

Andra Petrovai

Bianca-Cerasela-Zelia Blaga

Horatiu Florea

Datasets

As part of our work, we have developed and released several datasets for supporting research in deep learning solutions for visual perception tasks in the aerial domain.

WildUAV - Monocular UAV Dataset for Depth Estimation Tasks: Aerial RGB images, depth and accurate positioning, updated with semantic segmentation

Forest Inspection Dataset for Aerial Semantic Segmentation and Depth Estimation: Synthetic aerial RGB images, depth, positioning and semantic segmentation

Videos, Demos

Publications

2022

A. Petrovai, S. Nedevschi, “Exploiting Pseudo Labels in a Self-Supervised Learning Framework for Improved Monocular Depth Estimation”, 2022 Conference on Computer Vision and Pattern Recognition (CVPR). 19-24 June 2022, New Orleans, SUA - A* conference

A. Petrovai, S. Nedevschi, “Semantic Cameras for 360-degree Environment Perception in Automated Urban Driving”. IEEE Transactions on Intelligent Transportation Systems (Early Access), March 2022 (IF 6.319)

A. Petrovai, S. Nedevschi. “Fast Panoptic Segmentation with Soft Attention Embeddings”. SENSORS, January 2022 (IF: 3.57)

A. Petrovai, S. Nedevschi, “Time-Space Transformers for Video Panoptic Segmentation” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022, pp. 925-934

V. C. Miclea and S. Nedevschi, “Monocular Depth Estimation With Improved Long-Range Accuracy for UAV Environment Perception,” in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-15, 2022, Art no. 5602215, doi: 10.1109/TGRS.2021.3060513. (Q1, IF 5.6)

2021

R. Brehar, M.P. Muresan, M. Tiberiu, C. Vancea, N. Mihai, S. Nedevschi, “Pedestrian Street-Cross Action Recognition in Monocular Far Infrared Sequences”, IEEE ACCESS, Vol. 9, pp. 74302-74324, JUN 2021, DOI:10.1109/ACCESS.2021.3080822. (Q2, IF 3.367)

M.P. Muresan, S. Nedevschi, R. Danescu, “Robust Data Association using Fusion of Data-Driven and Engineered Features for Real Time Pedestrian Tracking in Thermal Images”, SENSORS, Vol. 21 Issue 23, AN 8005, NOV 2021, DOI: 10.3390/s21238005. (Q1, IF 3.567)

Video B.C.Z. Blaga, S. Nedevschi, “Weakly Supervised Semantic Segmentation Learning on UAV Video Sequences”, EUSIPCO 2021, Dublin, Ireland.

Git H. Florea, V.C. Miclea, S. Nedevschi, “WildUAV: Monocular UAV Dataset for Depth Estimation Tasks”, in Proceedings of 17th 2021 IEEE International Conference Intelligent Computer Communication and Processing (ICCP 2021).

S. Deac, C. Vancea, S. Nedevschi, “MVGNet: 3D object detection using multi-volume grid representation in urban traffic scenarios”, in Proceedings of 17th 2021 IEEE International Conference Intelligent Computer Communication and Processing (ICCP 2021), 27-20 October, 2021.

B. Maxim, S. Nedevschi, “OccTransformers: Learning occupancy using attention”, in Proceedings of 17th 2021 IEEE International Conference Intelligent Computer Communication and Processing (ICCP 2021).

B. Maxim, S. Nedevschi, “A survey on the current state of the art on deep learning 3D reconstruction”, in Proceedings of 17th 2021 IEEE International Conference Intelligent Computer Communication and Processing (ICCP 2021).

M.P. Muresan, R. Marchis, S. Nedevschi, R. Danescu “Stereo and Mono Depth Estimation Fusion for an Improved and Fault Tolerant 3D Reconstruction”, in Proceedings of 17th 2021 IEEE International Conference Intelligent Computer Communication and Processing (ICCP 2021).

2020

VC. Miclea, S. Nedevschi, Real-Time Semantic Segmentation-Based Stereo Reconstruction, IEEE Transactions on Intelligent Transportation Systems, vol. 21, no. 4, pp. 1514-1524, APR 2020, IF 6.319

VC. Miclea, S. Nedevschi, Semi-Global Optimization for Classification-Based Monocular Depth Estimation, Proceedings of 2020 IEEE International Conference on Intelligent Robots and Systems (IROS2020), October 25-29, 2020, Las Vegas, SUA

VC. Miclea, S. Nedevschi , A unified method for improving long-range accuracy of stereo and monoculardepth estimation algorithms, Proceedings of 2020 IEEE Intelligent Vehicles Symposium (IV2020), October19–November 13, 2020, Las Vegas, SUA.

A. Petrovai, S. Nedevschi, Real-Time Panoptic Segmentation with Prototype Masks for Automated Driving, Proceedings of 2020 IEEE Intelligent Vehicles Symposium (IV2020), October 19–November 13, 2020, Las Vegas, SUA.

R. Beche, S. Nedevschi, Narrowing the semantic gap between real and synthetic data, Proceedings of 2020 IEEE International Conference on Intelligent Computer Communication and Processing (ICCP2020),September 3-5, 2020, Cluj-Napoca, Romania.

V. Lup, S. Nedevschi, Video Semantic Segmentation leveraging Dense Optical Flow, Proceedings of 2020 IEEE International Conference on Intelligent Computer Communication and Processing (ICCP2020),September 3-5, 2020, Cluj-Napoca, Romania.

B. Maxim, S. Nedevschi, Efficient spatio-temporal point convolution, Proceedings of 2020 IEEE International Conference on Intelligent Computer Communication and Processing (ICCP2020), September 3-5, 2020, Cluj-Napoca, Romania.

S. Baciu, F. Oniga, S. Nedevschi, Semantic 3D Obstacle Detection Using an Enhanced Probabilistic Voxel Octree Representation, Proceedings of 2020 IEEE International Conference on Intelligent Computer Communication and Processing (ICCP2020), September 3-5, 2020, Cluj-Napoca, Romania.

B. C. Z. Blaga, S. Nedevschi, A Critical Evaluation of Aerial Data Datasets for Semantic Segmentation, Proceedings of 2020 IEEE International Conference on Intelligent Computer Communication and Processing (ICCP2020), September 3-5, 2020, Cluj-Napoca, Romania.

C. Golban, P Cobarzan, S. Nedevschi, A comparison study on replacing stereo disparity with LiDAR in visualodometry methods, Proceedings of 2020 IEEE International Conference on Intelligent Computer Communication and Processing (ICCP2020), September 3-5, 2020, Cluj-Napoca, Romania.

2019

A. Petrovai, S. Nedevschi, “Efficient instance and semantic segmentation for automated driving”, Proceeding of 2019 IEEE Intelligent Vehicles Symposium (IV 2019), Paris; France; 9 - 12 June, 2019, pp. 2575-2581

A. Petrovai, S. Nedevschi, ”Multi-Task Network for Panoptic Segmentation in Automated Driving”, Proceeding of 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zeeland, 26-30 October,2019

A. Baraian, S. Nedevschi, „Improved 3D Perception based on Color Monocular Camera for MAV exploiting Image Semantic Segmentation”, Proceeding of 2019 IEEE Intelligent Computer Communication and Processing (ICCP), 2019, pp. 295-301

B. C. Z. Blaga, S. Nedevschi, „Semantic Segmentation Learning for Autonomous UAVs using Simulators and Real Data”, Proceeding of 2019 IEEE Intelligent Computer Communication and Processing (ICCP), 2019, pp. 303-310

S. Capalnean, F. Oniga, R. Danescu, “Obstacle Detection Using a Voxel Octree Representation”, Proceeding of 2019 IEEE Intelligent Computer Communication and Processing (ICCP), 2019

V. Miclea, S. Nedevschi, ”Real-Time Semantic Segmentation-Based Depth Up Sampling Using Deep Learning”, Proceedings of 2018 IEEE Intelligent Vehicles Symposium (IV), Changzhou, China, June 26-30, 2018, pp. 300-306

2018

A.D. Costea, A. Petrovai, S. Nedevschi, ”Fusion Scheme for Semantic and Instance-Level Segmentation”, Proceedings of 2018 IEEE Intelligent Transportation Systems Conference (ITSC), Maui, Hawaii, USA, November 4-7, 2018, pp. 3469-3475.

V. Miclea, S. Nedevschi, L. Miclea, ”Real-Time Stereo Reconstruction Failure Detection and Correction Using Deep Learning”, Proceedings of 2018 IEEE Intelligent Transportation Systems Conference (ITSC), Maui, Hawaii, USA, November 4-7, 2018, pp. 1095-1102.