Smart VR – Live Streaming of Immersive Media Reference Implementation

Overview

This Reference Implementation (RI) provides a smart VR solution leveraging edge platforms for media ingestion from multiple streams, media processing (360 view construction, 360 frames analytics), and interactive media distribution based on Field of View (FoV) request.

To run the reference implementation, you will need to first download and install the Intel® Smart Edge Open Developer Experience Kit.

Once you have installed the Intel® Smart Edge Open Developer Experience Kit, select Configure & Download to download the reference implementation and the software listed below.  

Configure & Download

• Time to Complete: Approximately 1 hour
• Programming Language: Python*, C++, Golang
• Available Software: 
  • Intel® Smart Edge Open version 21.12
  • Intel® Collaboration Suite for WebRTC (Intel® CS for WebRTC) with Open WebRTC Toolkit (OWT)
  • Intel® Advanced 360 Video Library
  • Intel® Distribution of OpenVINO™ toolkit
  • Scalable Video Technology for High Efficiency Video Coding (SVT-HEVC) 

Target System Requirements

Intel® Smart Edge Open Nodes

• One of the following processors:
  • Intel® Xeon® scalable processor
  • Intel® Xeon® processor D
• At least 64 GB RAM
• At least 128 logic cores (CPU)
• At least 512 GB hard drive
• An Internet connection
• Ubuntu* 20.04 LTS Server
• IP camera or pre-recorded video(s)

How It Works

This reference implementation is an Omnidirectional MediA Format (OMAF) immersive (VR) media application run on cloud infrastructure for end-to-end (E2E) live streaming and supports file content VR streaming. It runs on WebRTC stack with Linux* (Ubuntu* 20.04). The Intel® Smart Edge Open platform infrastructure is used to deploy the application pods.  

Includes the following main functionality systems:  

• Media Ingestion Function (MIF): Captures stream from 4k camera, 360 stitching, high- and low-resolution encoding.  
• Media Distribution Function (MDF): Performs tile re-packing and distributes the content.  
• Media Control Function (MCF): Responsible for scheduling and managing network units, signaling transmission between MIF and MDF.  
• Media Analytic Function (MAF): Enables the media analytic pipeline using the OpenVINO™ model. This media analytics acceleration building block allows CPU and GPU/VPU acceleration for frame analytics. 
• Client (Linux/Android*): Does Supplemental Enhancement Information (SEI) parsing, decoding, low- and high-resolution tiles re-ordering, FoV, rendering, etc.

Intel® Smart Edge Open 21.12 or above supports on-premise cluster deployment with MIF, MAF and MDF on the same host. MCF control function is located on the cloud cluster to provide centralized control function and web UI. 

L3 switch will be supported in Real 360 Camera mode.

Figure 1: Architecture Diagram

Get Started

Prerequisites

Ensure that the following conditions are met properly for a smooth installation process.

1. Hardware Requirements
   Make sure you have a fresh Ubuntu 20.04 LTS Server installation with the hardware specified in the Target System Requirements section.
2. To run the reference implementation, you will need to first download and install the Intel® Smart Edge Open Developer Experience Kit.

Step 1: Install the Reference Implementation

Select Configure & Download to download the reference implementation and then follow the steps below to install it.  

Configure & Download

1. Make sure that the Target System Requirements are met properly before proceeding further.  
   • For single-device mode, only one machine is needed. (Both controller and edge node will be on the same device.) 
   • NOTE: Multi-device mode is not supported in the current release. 
2. Create a user account for Intel® Smart Edge Open: 

   sudo adduser "smartedge-open"
   sudo passwd "smartedge-open"
   echo "smartedge-open ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/smartedge-open 

3. Open a new terminal as smartedge-open user:  

   su - smartedge-open

4. Generate an SSH key under smartedge-open user:

   ssh-keygen
   ssh-copy-id smartedge-open@target_server_IP 

5. Move the downloaded zip package to the /home/smartedge-open folder: 

   sudo mv <path-of-downloaded-directory>/Smart_VR_Live_Streaming_of_Immersive_Media.zip /home/smartedge-open

6. Go to the /home/smartedge-open directory using the following command and unzip the RI:

   cd /home/smartedge-open 
   unzip Smart_VR_Live_Streaming_of_Immersive_Media.zip 

7. Go to the Smart_VR_Live_Streaming_of_Immersive_Media/ directory: 

   cd Smart_VR_Live_Streaming_of_Immersive_Media /

8. Change permission of the executable edgesoftware file: 

   chmod 755 edgesoftware 

9. Run the command below to install the Reference Implementation:  

   ./edgesoftware install

10. During the installation, you will be prompted for the Product Key. The Product Key is contained in the email you received from Intel confirming your download.  
    NOTE: Installation logs are available at: /var/log/esb-cli/Immersive_Media_<version>/Immersive_Media/install.log

    Figure 2. Product Key

     

11. When the installation is complete, you see the message Installation of package complete and the installation status for each module.  

    NOTE: This step may take about 5 minutes, depending on your network connection.
     

    

    Figure 3. Installation Complete

     

If Intel® Smart Edge Open is installed, running the following command should show output similar to the image below. All the pods should be either in the running or completed stage.  

kubectl get pod -A

Figure 4. Pod Status

Run the Application

1. Check the Smart_VR_Live_Streaming_of_Immersive_Media pod with the command:

kubectl get pod -n smartedge-apps

You will see output similar to: 

Figure 5. Application Status

2. In a browser, go to https://<SmartEdge-node-ip>:31004/ to verify the Smart_VR_Live_Streaming_of_Immersive_Media app result.

Follow the steps below to verify the certificate: 

1. Select Click this for testing certificate and refresh.
 

Figure 6. Test Certificate

2. Select Proceed to... 
 

Figure 7. Test Certificate - Proceed

3. You will see output similar to:
 

Figure 8. Test Certificate - Complete

4. Visit URL to allow camera and microphone of the device: https://\<SmartEdge-node-ip\>:31004/.

Figure 9. Allow camera and microphone

5. Close the tab and open a new one with https://<SmartEdge-node-ip>:31004/?publish=false.

6. Select Analytics Demo.

Figure 10. Click Analytics Demo

You will see the analytic web:

Figure 11. Analytic Web

• Select 62xxxxxxxxxxx-common in the video from drop-down, and then click the startAnalytics button.
• Select 9D1Axxxxxxxxx-common in the subscribe video drop-down, and then click the subscribe button.

You will see the final result:

Figure 12. Application Video Output

NOTE: When you finish viewing the video or someone reopens a page to check the video, click the stopAnalytics button to stop analytics.

Uninstall the Application

1. List the installed RI module with the command:

   ./edgesoftware list

   You will see output similar to: 

   

   Figure 11. Edgesoftware List

    

2. Uninstall the RI module using the <module-id> from the previous step in the following command: 

   ./edgesoftware uninstall <module-id>

   You will see output similar to: 

   

   Figure 12. Uninstall RI

3. Run the commands below to make sure the block-all-ingres network policy is deleted after uninstalling

   $ kubectl get netpol
   NAME                POD-SELECTOR   AGE
   block-all-ingress   <none>         58d

Post deletion only the below network policy should be present.

$ kubectl get netpol -n smartedge-apps
NAME                         POD-SELECTOR   AGE
deny-from-other-namespaces   <none>         18h

Node Feature Discovery (NFD) Feature

Immersive Media uses Intel® Distribution of OpenVINO™ toolkit which is optimized for Intel® processors that support special instructions like AVX512VNNI for optimized performance. The deployment of this application will require the node with this feature supported on the node along with Ubuntu 20.04 OS. This NFD features ensures to deploy the application on the node supported with these features. NFD is installed by Intel® Smart Edge Open Developer Experience Kit and running as two pods on Intel® Smart Edge Open.

$ kubectl get pods -A | grep node-feature-discovery

smartedge-system         nfd-release-node-feature-discovery-master-7f4ffd8f6b-ckwrx   1/1     Running   6 (25h ago)      58d
smartedge-system         nfd-release-node-feature-discovery-worker-m69h7              1/1     Running   64 (25h ago)     58d

Immersive Media pod scheduled and running successfully on Intel® Smart Edge Open node based on hardware capabilities of Intel® Xeon® Scalable server.

$ kubectl get pods -n smartedge-apps | grep im360

im360-mcf-85744f5b5d-nnrk6    1/1     Running   0          5m26s
im360-mdf-7d8455677f-fpkjl    1/1     Running   0          5m26s
im360-mif-79674f8978-m52bf    1/1     Running   0          5m26s
im360-mqtt-6b6cdd7cfc-8dmcs   1/1     Running   0          5m26s

The following output shows a description of im360 pod, which shows that it is running successfully with the NFD feature.

$ kubectl describe pods im360-mcf-85744f5b5d-nnrk6 -n smartedge-apps
... ...
Node-Selectors:              feature.node.kubernetes.io/cpu-cpuid.AVX2=true
                             feature.node.kubernetes.io/cpu-cpuid.AVX512VNNI=true
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  32m   default-scheduler  Successfully assigned smartedge-apps/im360-mcf-c9587755c-9dkds to ubuntu252
  Normal  Pulled     32m   kubelet            Container image "openvisualcloud/xeon-ubuntu2004-smartedge-im360:2.9.3" already present on machine
  Normal  Created    32m   kubelet            Created container im360-mcf
  Normal  Started    32m   kubelet            Started container im360-mcf

Public Helm Registry for Helm Charts

Installation of Immersive Media Reference Implementation Intel® Smart Edge Open is accomplished using Helm charts. Earlier Helm charts used to be a part of Reference Implementation installation package. Now a global Helm repo is issued, so that Reference Implementation helm charts can be accessible from private and public network. This will speed up and ease the process of introducing updates and their integration with the Reference Implementation.

Summary and Next Steps

This reference implementation successfully demonstrates the following features: 

• Video analytics pipeline based on Intel® Distribution of OpenVINO™ toolkit and Intel® Deep Learning Streamer (Intel® DL Streamer). 
• Media streams can be forwarded and analyzed through distributed pipeline. 
• K8s* based deployment (single node, two pods, multi-containers) on Intel® Smart Edge Open 21.12. 
• Enhanced Android* player receives and processes analytics metadata. 
• Web UI for simulator.
• For Developers and end users: 
  • Work on real time video stitching for 4k/8k after VR camera input.
  • Work on H.265 OMAF encoding format support, which supports high and low-definition mixing.
  • Work on FOV field of view support for dynamically switching to HD.
     

As a next step, try one of the following: 

• Support the multi-clusters deployment. 
• Media analytic pipeline qualities’ optimization.

Learn More

To continue learning, see the following guides and software resources:

• Intel® Smart Edge Open Developer Experience Kit Architecture

Troubleshooting

Installation Confirmation

Check the log of im360-mif to see if the stream is added successfully: 

kubectl logs <im360-deployment-PodName> -n smartedge-apps -c im360-mif 

Uninstall Confirmation

Execute the command helm list to check whether there is im360-app or kubectl get pod -n smartedge-apps to check whether there are related pods.

Subscribed but Video is not displayed

If you have clicked the Subscribe button but no video is displayed, disable the firewall.

## Please turn off the firewall of your target server
sudo ufw disable

Video not displayed in browser

If you cannot see the video in the browser, you may need to update the browser version. Refer to the table below for supported browsers on different operating systems. 

You must grant the necessary permission by selecting Continue allowing <url> to access your camera and microphone.

Internal Server Error

If there is a error in the analytics id field, Internal Server Error, complete the steps as shown in the following screen:

Make sure there is only one xxx-common item and another three common items.

Network Policy

The Smart_VR_Live_Streaming_of_Immersive_Media application uses the host network, which will use the host's port 31004, 31080. (Port 31004 is for the view side; the port 31080 is for the testing certification.)

Support Forum 

If you're unable to resolve your issues, contact the Support Forum.  

To attach the installation logs with your issue, execute the command below to consolidate a list of the log files in tar.gz compressed format, e.g., Smart_VR_Live_Streaming_of_Immersive_Media.tar.gz. 

tar -czvf Smart_VR_Live_Streaming_of_Immersive_Media.tar.gz /var/log/esb-cli/Immersive_Media_3.0.0/immersive_media/