Since the past few years, video stitching has been around us and we have been seeing accelerated demand for the stitching algorithms (Image Stitching, Video Stitching). Nowadays, stitching algorithms are clearly associated with the products users interact in their daily lives e.g. creating a panoramic view from mobile cameras, video conferencing solutions, creating 360 degrees / surround-view, automobile assistance, wide FOV for surveillance videos, etc. Much ink has been spilled on video stitching which is one of the oldest and widely used algorithms running the technological ecosystem but do you know that artificial intelligence and machine learning (AI/ML) have deep implications on the said technology?  The advancement of  AI/ML has given a whole new dimension to Video stitching and its applications. Let’s understand how Video stitching has evolved over the years, major challenges faced by the developers, and the various aspects of video stitching in terms of its applications leveraging the latest AI-infused technologies.

What are the long term challenges for Video Stitching?

In this section, I would like to share from my experience some of the challenges that a developer faces while implementing Video Stitching.

Overlapping Regions with Low Texture

The appearance of low texture overlapping regions in the images that need to be stitched is one of the major challenges of video stitching. This leads to inaccurate results where we won’t get the exact match for overlapping regions.

Wide Baseline with Large Parallax 

Despite using professional technologies, people tend to capture practical images in a very casual way using their own portable devices such as mobile phones or digital cameras which results in some unequal distortions that lead to issues like a wide baseline and large parallax. 

In my opinion, the standard proposed methods for video stitching only work on standard datasets i.e. natural baseline, little or no parallax. Whereas, the above-mentioned issues generally instigate blurring and ghosting in the image, inaccurate transformations of the images.

For all the developers who are facing these stubborn issues, AI/ML has come out as a guiding light. Innovative technologies like Machine learning, Deep learning, Artificial intelligence are creating opportunities for such kind of issues. Predominantly, AI/ML plays an important role in the performance and the quality of the video stitching to avoid the overlapped regions. Based on the common view covered in all the cameras, the AI/ML-based video stitching algorithms reduce the load on the hardware employed, and video features tuning helps to provide a very accurate and surreal experience of multiple wide-angle views on a single screen. It gives a smooth and accurate transition that one can’t make out the difference between the views along the stitching lines.

Video Stitching Applications with AI/ML – THE NEW NORMAL

Let’s understand the impact of Video Stitching + AI/ML on some well-known applications

Video Conferencing Solutions

Exponential growth in the telecom sector and the advent of 5G capabilities in the mobile network accelerated the video streaming capabilities & options. Therefore, to have seamless user experience with multiple views stitched for a panoramic view, 4K and 8K video streaming options are becoming the mainstream. High resolution streaming options increase the demand for high resolution stitched images that directly affects more on the bit rate front, and this problem can be alleviated using artificial intelligence. AI integrated Video conferencing solutions also help in saving bit rate and maintaining the same quality for a particular resolution.

Using traditional methods for video stitching increases the chances of having an inaccurate transformation and blurriness in the image. This is the reason that video stitching and conferencing solutions have left behind the traditional methods which only use image processing and video analytics algorithms. Nowadays, keeping the best stitching performance is a major concern and that can be achieved by post-processing algorithms integrated with some latest and innovative technologies such as Artificial Intelligence and Machine Learning. Let’s discuss the challenges for video stitching in video conferencing solutions and how we can overcome them.

Challenges:

  1. The deviation in the views along the line of stitching.
  2. The Performance impact on the hardware owing to the increase in video resolution up to 5.7k which is slowly becoming mainstream in 2020 and beyond.
  3. An increase in the use of other AI/ML use-cases that can parallelly work on the stitched output can affect the accuracy of these use-cases. 
  4. Accuracy of the AI/ML algorithms along the lines where the object of interest is divided between 2 views and the quality of stitching for accurate object/person detection.

New Normal

  1. Using AI/ML algorithms that help in stitching only the necessary views from all the cameras can negate the performance impact to an extent.
  2. A video stitching algorithm with AI creates an accurate single stitched output image without any unusual demarcations along the stitched lines.
  3. Using AI/ML in video conferencing helps in covering more regions of interest (ROI) which brings in the usage of the higher resolution of images/videos, followed by an improvement in the perspective of covering a large scale on a single large screen.

360 Degree / Panoramic View Solutions 

Slowly and gradually around the world, all the vehicles sold are manufactured with surround-view cameras owing to the safety they provide especially in bigger trucks & commercial vehicles. Video stitching also helps in traffic management and maintaining safe driving standards for the better safety of all the occupants in the vehicle. This is basically a four-camera solution that helps in generating a 360-degree panoramic view around the vehicle and reduces the blind spot in order to maintain safe driving. 

As the size of the vehicle increases, e.g, a trailer or a caravan the number of cameras used also increases, which brings in making use of AI/ML-based stitching to cover the full area around the vehicle. Innovative technologies like AI/ML also improves the performance, accuracy of the stitching algorithms which helps in detecting obstacles around the vehicle in life-critical scenarios with efficient results. 

Challenges:

  1. 360 Degree and Panoramic view solutions have always been inhabited by the number of cameras deployed. As the number of cameras goes up, so do the stitching lines which increases the negative user experience.
  2. High-resolution images/videos for panoramas always bring in high memory bandwidth requirements.
  3. If we don’t make use of AI/ML, 360-degree views bring us in the territory of using high-end CPUs and GPUs for the stitching to be performed.


New Normal

  1. Employing AI/ML algorithms in video stitching helps in having lesser misalignment along the stitching lines which in turn improving the quality of the panoramic/360-degree videos.
  2. In order to remove the significant amount of data bandwidth requirements, AI/ML helps in removing the redundant parts in all the views of multiple cameras.
  3. AI algorithms help in reducing the horsepower requirements on the hardware (CPU and GPU) side, which helps to perform panoramic and 360 views on low-end hardware as well.

Integration of AI Capabilities with Video Stitching- Benefits

  • Less performance Impact on the CPU and GPU
  • Data consumption or stress on the memory bandwidth is less
  • More realistic user experience with an accurate representation of all the views of the different cameras
  • More precise surveillance experience in safety, crowd control, and object/person tracking 

Lately, Video stitching has gained considerable grounds in securing vast regions of space like a cricket match in a stadium where it can help in covering all the views of more than 1000 cameras deployed in the stadium. This hugely improves the user experience and helps in having more eye on the security aspect as well where you can have multiple views of say stairways stitched together on a single big screen.

How does VVDN help OEMs with Video stitching solutions?

VVDN has deep expertise in the design, development, and manufacturing of next-gen technological solutions such as video conferencing, Surveillance cameras, 360-degree panoramic view, surround-view The presence of a video calibration lab for accurate stitching & tuning of the AI/ML-based stitching algorithms and usage of Image/Video features based model training enables us to deliver best in class results. VVDN with its reusable frameworks and IPs can help OEMs with effective and accurate video stitching solutions in a FASTER TIME TO MARKET. 

Click here to read more about VVDN’s camera expertise or for any such requirement feel free to contact: info@vvdntech.com

Sujith Sivanandan

Author

Sujith Sivanandan

Principal Engineer (SW) - Vision