Open Visual Cloud

ID 674059
Updated 9/7/2021
Version 21.6
Public

author-image

By

Open Visual Cloud

Overview

The Open Visual Cloud is a set of open source software stacks (with full end-to-end sample pipelines) for media, analytics, graphics, and immersive media, optimized for cloud native deployment on commercial-off-the-shelf x86 CPU architecture. The open source software stacks presented for the Open Visual Cloud are provided under the FFmpeg and GStreamer frameworks as ready-to-use Docker* images and Dockerfiles, which can help to implement cutting edge Visual Cloud services for advanced video distribution and processing. The goal of the Open Visual Cloud is to unleash innovation, simplify development, and accelerate time to market for visual cloud services by providing open source, interoperable, high-performance building blocks and containerized reference pipelines.

Five Services Flowchart for Visual Cloud

Open Visual Cloud Services

The Open Visual Cloud enables five major services, each providing a set of related visual cloud use cases. The Open Visual Cloud services are: Media Processing and Delivery, Media Analytics, Immersive Media, Cloud Graphics, and Cloud Gaming.  

Open Visual Cloud pipeline

Contending with the onslaught of new visual workloads will require more nimble, scalable, virtualized infrastructures. The visual cloud requires companies to have the capability of shifting workloads to the network edge when appropriate; and a collection of tools, software, and hardware components to support individual use cases fluidly. 

Advanced network technologies and cloud architectures are essential for agile distribution of visual cloud services. Examples include: 

  • Increasing flexibility and optimizing processing: Virtualization and software-defined infrastructure (SDI) make it easier to balance workloads on available resources. The open source Scalable Video Technology is an example of using software to optimize processing on x86 machines. 
  • Scaling compute, storage, and network resources: Dynamic elasticity is a major advantage when contending with visual cloud workloads. Adopting a modern cloud infrastructure powered by a new generation of scalable processors increases resource availability substantially. 
  • Enhancing development processes: High performance, open sourced media, AI, and graphics software components, along with sample reference pipelines that demonstrate how to construct new Visual Cloud services quickly and easily support rapid application development and help reduce time to market for new revenue.  
  • Deploying purpose-built solutions for select use cases and edge computing: Specialized hardware—including discrete graphics processing units (GPUs), integrated GPUs, field programmable gate arrays (FPGAs), and video processing units (VPUs)—can boost performance for select applications in which a single, targeted workload must be handled on a large-scale basis.    
  • Implementing modern cloud architectures: For performing operations on large, complex data files and delivering elastic processing power to efficiently handle ebbs and flows in intensive computing operations, modern cloud architectures hold the key to effective workload distribution. 

Building Blocks 

There are 4 groups of open source building blocks for most visual cloud workloads: encode, decode, inference, and render.  

Visual cloud services are defined by the order in which these open source building blocks are placed in a workflow pipeline. For example, a simple “media processing and delivery service” can be thought of as consisting of decode + encode. Insertion of an inference building block such as OpenVINO™ (decode + inference + encode) would result in a “media analytics” service relevant for such things as digital security and surveillance or user generated content ad-insertion use cases where intelligent content analysis is required. 

Visual Cloud Pipeline with Blocks

Intel is contributing to each core building block with new and existing projects and enhanced performance. After observing that encode is a required building block across all the visual cloud services, Intel released several Scalable Video Technology (SVT) encoder core libraries, along with interoperability with x264 and x265 decoders, to support the ecosystem's needs. Additionally, the OpenVINO™ toolkit and the Intel® Rendering Framework make up the inference and render blocks respectively.  

Encode 

At the most basic, encode is compressing video data to reduce it in size. Since the visual cloud is run on video data, Encode becomes one of the key building blocks developers will use in constructing most visual cloud services and pipelines. As of 2019, AV1 has emerged as a new entrant now commercially viable, thanks to SVT-AV1.  

There are a variety of individual open source ingredients that make up each Open Visual Cloud Building Block. Some of these ingredients include: 

  • Scalable Video Technology (SVT) Video encoding technology optimized for x86 processors. Supported codecs include HEVC, VP9, and AV1. 
  • FFmpeg - FFmpeg is a open source project consisting of a vast software suite of libraries and programs for handling video, audio, and other multimedia files and streams. 
  • x265 - x265 is a H.265 / HEVC video encoder application library, designed to encode video or images into an H.265 / HEVC encoded bitstream. 
  • x264 - x264 is an open-source software library and a command-line utility developed by VideoLAN for encoding video streams into the H.264/MPEG-4 AVC format. 
  • Open WebRTC Toolkit – Open WebRTC Toolkit is an open source real-time media delivery framework, which includes comprehensive media processing functions on video and audio streams. 

Decode

Decode is defined as uncompressing encoded video data. Decode goes hand-in-hand with Encode, as once your video has been encoded, it needs to be decoded to process or view on a screen. Decode technology can be either hardware or software-based. As new HD video file codecs hit the market, we will see a greater adoption of  these new codecs into hardware such as TVs, video cameras, smartphones, and others. As of early 2019, some of the open source decoders include: 

  • VLC Media Player - VLC is a free and open source cross-platform multimedia player and framework that plays most multimedia files. 
  • dav1d - dav1d is a new AV1 cross-platform decoder, which is open-source, and focused on speed and correctness. 
  • Mozilla Firefox - Firefox is an open source web browser that released AV1 support in Firefox 65 in Jan of 2019. 
  • Android Q - Google* has announced that Android* Q will support AV1 decode. 
  • Chrome - Google's Chrome/Chromium 70 web-browser is now shipping with AV1 video decoding support 

Inference

An inference building block analyzes video content data. Inference is used by Artificial Intelligence (AI) to perform many tasks, from facial recognition to ad insertion, smart city use cases such as street corner traffic management, based on deep learning neural networks.  

As video become more ubiquitous, the need to analyze what is shown in the video becomes more and more important. Using AI models, it is possible to train applications to search for specific patterns within a video (a person, a vehicle, brand logo, etc.) and act on what it finds. 

Individual open source ingredients that make up the Inference building block can be found in the OpenVINO™ toolkit, and include: 

  • Deep Learning Deployment Toolkit (DLDT):  A key component of the OpenVINO™ toolkit, DLDT provides a model optimizer and inference engine that supports Intel CPU and GPU (Intel® Processor Graphics) and heterogeneous plugins. 
  • OpenVINO - OpenVINO™, short for Open Visual Inference and Neural network Optimization, is a toolkit that provides developers with improved neural network performance on a variety of hardware (CPU, GPU, FPGA, VPU) and helps them further unlock cost-effective, real-time vision applications. 
  • Open Model Zoo - Pre-trained deep learning models and samples for use in OpenVINO™. 

Render

Video rendering is the process by which a computer processes information from a coded data source and uses that information to produce and display an image. This is usually in the context of creating a video animation or visualizing a large data set. 

Individual open source ingredients for the Render building block are found in the Intel® Rendering Framework, which includes: 

End-to-end Reference Pipelines for Visual Cloud Services

The Open Visual Cloud Building Blocks support industry standard open source frameworks such as FFMPEG, GStreamer, OpenVINO™ toolkit, TensorFlow*, Caffe*, Apache MXNet*, and more. With the ease of interoperability between software and hardware building blocks, it is easy for developers to create flexible architecture designs for their applications or services. By using pre-defined server configuration files and open source building blocks, developers can easily create pipelines for Open Visual Cloud Services.

Why Do We Need the Open Visual Cloud?

A software-based approach to visual cloud processing offers flexibility and the opportunity for new revenue channels without having to rework the physical architecture or update the hardware. Running visual compute workloads on pure software architectures gives companies the flexibility of making improvements to media processing and delivery, as well as customizations to the visual quality of processed data. To maximize the business value of infrastructure investments, companies need servers flexible enough to accommodate a diverse range of both compute and visual workloads.

By using available open source software components with highly permissive licensing terms, developers are able to achieve rapid innovation and circumvent having to use a proprietary hardware platform subject to vendor lock-in and a greatly reduced range of supported applications. With a modular plug-in approach, the Open Visual Cloud will support hardware accelerated functionality and seamless migration, if needed, using the same software architecture. 

Values to Developers

  • Ready-to-use docker images: Domain specific software stacks provided as Docker images ready to pull, and Dockerfiles for customization.
    Explore more container solutions on the Intel® oneContainer Portal
  • FFmpeg/GStreamer high-level software: Building blocks you already familiar with, unified under well-known frameworks, for quick app development.
  • Optimized software, add HW acceleration easily: Develop with optimized software with easy switching to hardware acceleration by changing Docker layers.
  • E2E tested and patched: We have done the hard work to ensure the software works end to end.
  • E2E sample for innovation: Many samples to showcase what is possible and use the software stacks we provide.  
  • Open recipes for non-docker apps: Not ready to adopt Docker? Dockerfiles shows all the ingredients and secrets. Read our open recipe book.

News & Announcements

Open Visual Cloud 21.6.1 Release (Aug 9, 2021)

Open Visual Cloud 21.6.1 Released  -  https://github.com/OpenVisualCloud/Dockerfiles/releases/tag/v21.6.1
•    OpenVINO™/dldt template bug fixes
•    Enable SVT HEVC with FLV support by patching FFmpeg

Open Visual Cloud 21.6 Release (July 26, 2021)

Open Visual Cloud 21.6 released  - https://github.com/OpenVisualCloud/Dockerfiles/releases/tag/v21.6

Docker File Upgrade of components

  • Enable Ice Lake Platform with Xeon/SG1/QAT images validated on ICX systems 
  • Introducig QAT/NGINX_SW Image optimized for ICX systems with AVX512 
  • Inroduce vmaf images for video quality assessment on Xeon/XeonE3/SG1 platforms 
  • Retire graphics images 
  • Upgrade Gstreamer framework to 1.19.1 
  • Upgrade SVT-HEVC to v1.5.1 
  • Upgrade SVT-AV1 to v0.8.7 
  • Upgrade SVT-VP9 to v0.2.2 
  • Upgrade fdk-acc to 2.0.2 
  • Upgrade libx265to 3.4 
  • Upgrade libvpx to 1.10.0 
  • Upgrade libogg to 1.3.5 
  • README Changes 
  • Bug Fixes   

 

Learn More

Blogs

Whitepapers

Videos

Podcasts

Use Cases

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.