Video and Vision Processing Suite Intel® FPGA IP User Guide

Download PDF
ID 683329
Date 9/30/2021
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
About the Video and Vision Processing Suite Getting Started with the Video and Vision Processing IPs Video and Vision Processing IP Interfaces Video and Vision Processing IP Registers Protocol Converter Intel® FPGA IP 3D LUT Intel® FPGA IP Tone Mapping Operator Intel® FPGA IP Warp Intel® FPGA IP Document Revision History for Video and Vision Processing Suite User Guide
About the Video and Vision Processing Suite x
Device Family Support
Getting Started with the Video and Vision Processing IPs x
Video and Vision Processing IP Time-out Behavior Generating a Video and Vision Processing IP Simulating the Video and Vision Processing IPs
Protocol Converter Intel® FPGA IP x
About the Protocol Converter Protocol Converter Intel® FPGA IP Parameters Protocol Converter Intel® FPGA IP Functional Description Protocol Converter Intel® FPGA IP Interfaces Protocol Converter Intel® FPGA IP Registers
About the Protocol Converter x
Protocol Converter Release Information
3D LUT Intel® FPGA IP x
About the 3D LUT Intel® FPGA IP 3D LUT IP Parameters 3D LUT IP Block Description 3D LUT IP Registers 3D LUT IP Software API
About the 3D LUT Intel® FPGA IP x
3D LUT IP Features 3D LUT IP Release Information 3D LUT IP Performance IP and Resource Information
3D LUT IP Block Description x
3D LUT IP Interfaces 3D LUT IP Latency
Tone Mapping Operator Intel® FPGA IP x
About the Tone Mapping Operator IP TMO IP Parameters TMO IP Block Description TMO IP Registers TMO IP Software API
About the Tone Mapping Operator IP x
TMO IP Features TMO IP Release Information TMO IP Performance and Resource Utilization
TMO IP Block Description x
TMO IP Interfaces TMO IP Latency
Warp Intel® FPGA IP x
About the Warp IP Warp IP Parameters Warp IP Block Description Warp IP Registers Warp IP Software API
About the Warp IP x
Warp IP Features Warp IP Release Information Warp IP Performance and Resource Utilization
Warp IP Block Description x
Coefficient Tables Warp Mesh Interpolation Output Pixel Interpolation and Filtering Blank Skip Regions Warp IP Interfaces Warp IP Latency External Memory for Warp IP
Warp IP Software API x
Using Warp IP Software Warp IP Software Code Examples
About the Video and Vision Processing Suite
Getting Started with the Video and Vision Processing IPs
Video and Vision Processing IP Interfaces
Video and Vision Processing IP Registers
Protocol Converter Intel® FPGA IP
3D LUT Intel® FPGA IP
Tone Mapping Operator Intel® FPGA IP
Warp Intel® FPGA IP
Document Revision History for Video and Vision Processing Suite User Guide

Warp IP Block Description

The Warp IP accepts RGB or YUV format video input from its Intel FPGA streaming video interface and directly stores the video data in external memory. The video input process has a pool of four frame buffers that it uses to store the incoming video data. The IP accesses the buffers in a cyclic order.

Coefficient Tables

The IP processes the buffered input video by the configured number of warp engines to apply the required warp. Three coefficient tables control the warp engines to define the warp that the IP applies. External memory stores the three coefficient tables that are generated using the Warp IP software API.

The IP defines the required warp with a backward mapping from the output to the input pixel positions. It represents the warp as a subsampled mesh that defines the mapping in 8x8 regions. For output pixel mappings within the 8x8 positions, the warp engine applies bilinear interpolation.

The IP writes back the resultant warped image to external memory in to one of two output video buffers. The IP writes to these dual output buffers alternately.

The video output process generates an RGB or YUV format Intel FPGA streaming video by reading the warped image data from external memory.

Figure 12. Warp IP block diagram

The figure shows a high-level block diagram for the Warp IP with its connection to external memory.
Figure 13. Warp image transform examplesFrom left: arbitrary warp, four corner warp

Each engine within the Warp IP has read access to its own set of three coefficient tables that define and control the image transform that the IP applies. The three different tables are:

  • Mesh coefficients that defines the output to input pixel transform
  • Fetch coefficients that control the loading of the input image into the cache memory within the engine(s).
  • Filter coefficients that control the mapping from the cache memory as the IP generates the interpolated or filtered output pixels.

The format of the mesh coefficients is different to the mesh data that the IP provides to the software API. The Software API uses 32-bit signed integers for the mesh values; the Warp IP uses a 16-bit offset binary format.

The IP needs just the mesh data to define the warp. The software API uses this mesh data to generate the required coefficient tables.

Warp Mesh Interpolation

The IP defines the warp transform using an 8x8 subsampled mesh. This mesh defines the mapping from the output pixel positions to the corresponding input pixel positions. The 8x8 subsampled mesh requires that only the mappings for the following output pixel positions are defined: 

(0,0), (8,0), (16,0) … (W, 0)
(0,8), (8,8), (16,8) … (W, 8)
.
(0,H), (8, H), (16, H) … (W, H)

where W=8*ceil(image width/8) and H=8*ceil(image height/8)

To generate the output pixel positions that lie in between these 8x8 positions, the Warp IP uses bilinear interpolation .

Output Pixel Interpolation and Filtering

The IP generates output pixels with the pixel data from the associated input pixel positions as defined by the warp that the IP applies. The IP generates output pixel values with a bicubic interpolation calculation using a 4x4 kernel of the associated input pixel values.

The weightings for the interpolation over the 4x4 kernel are a bicubic function and a variable low pass filtering function. The software API automatically applies the degree of low pass filtering, which it bases on the amount of downscaling that results for that particular region of the warp.

Blank Skip Regions

When you configure the Warp IP to substantially downscale regions of an image, large areas of the output image can map to points outside the input image. These unmapped regions result in the IP producing black.

Because these regions in the output image do not require any processing of the input image by the Warp IP, for efficiency the IP skips the processing associated with these regions. This skipping process is setup automatically by the software API which determines, from the desired warp mapping, which regions you may program to skip.

Level Two Title