FPGA AI Suite: IP Reference Manual

Download PDF
ID 768974
Date 9/06/2024
Public
Document Table of Contents
Document Table of Contents x
1. FPGA AI Suite IP Reference Manual 2. About the FPGA AI Suite IP 3. FPGA AI Suite IP Generation Utility 4. FPGA AI Suite Ahead-of-Time Splitter Utility 5. CSR Map and Descriptor Queue A. FPGA AI Suite IP Reference Manual Archives B. FPGA AI Suite IP Reference Manual Document Revision History
2. About the FPGA AI Suite IP x
2.1. Supported Models 2.2. Model Performance 2.3. Software Emulation of the FPGA AI Suite IP 2.4. FPGA AI Suite Layer / Primitive Ranges 2.5. FPGA AI Suite IP Block Configuration 2.6. IP Block Interfaces 2.7. Feature Input and Output Streaming 2.8. DDR-Free Operation
2.1. Supported Models x
2.1.1. MobileNet V2 differences between Caffe and TensorFlow models
2.2. Model Performance x
Details - FPGA AI Suite 2024.2 public/mobilenet-v1-1.0-224 public/mobilenet-v2 public/mobilenet-v2-1.4-224 public/mobilenet-v3-large-1.0-224-tf public/resnet-50-tf Resnet50 v1 (Caffe) intel/unet-camvid-onnx-0001 public/yolo-v3-tf public/yolo-v3-tiny-tf public/yolo-v8-nano detection public/yolo-v8-nano classification public/squeezenet1.1 public/i3d_rgb_tf 2.2.1. Throughput on the MobileNetV1 model (and other very fast models)
2.5. FPGA AI Suite IP Block Configuration x
2.5.1. Architecture Description File Format for Instance Parameterization 2.5.2. Architecture Description File Parameters
2.5.2. Architecture Description File Parameters x
2.5.2.1. Parameter Group: Global Parameters 2.5.2.2. Parameter Group: activation 2.5.2.3. Parameter Group: pe_array 2.5.2.4. Parameter Group: pool 2.5.2.5. Parameter Group: depthwise 2.5.2.6. Module: softmax 2.5.2.7. Parameter Group: dma 2.5.2.8. Parameter Group: xbar 2.5.2.9. Parameter Group: filter_scratchpad 2.5.2.10. Parameter Group: input_stream_interface 2.5.2.11. Parameter Group: output_stream_interface 2.5.2.12. Parameter Group: config_network 2.5.2.13. (Early Access) Parameter Group: layout_transform_params
2.6. IP Block Interfaces x
2.6.1. Clock and Reset 2.6.2. AXI Interfaces 2.6.3. AXI Interface Clock and Reset 2.6.4. Input Feature Tensor In-Memory Format 2.6.5. Output Tensor In-Memory Format
2.6.4. Input Feature Tensor In-Memory Format x
2.6.4.1. Multiple Input Graphs 2.6.4.2. Input Folding 2.6.4.3. Input Scale and Shift 2.6.4.4. Input Transform Mapping 2.6.4.5. Input Layout Transform Hardware
2.7. Feature Input and Output Streaming x
2.7.1. Input Streaming 2.7.2. Output Streaming
3. FPGA AI Suite IP Generation Utility x
3.1. IP Generation Utility Execution Flows 3.2. IP Generation Utility Inputs 3.3. IP Generation Utility Outputs 3.4. IP Generation Utility Command Line Options
3.4. IP Generation Utility Command Line Options x
3.4.1. The --flow create_ip Flow 3.4.2. The --flow add_arch Flow 3.4.3. The --flow list Flow 3.4.4. The --flow remove_arch Flow
4. FPGA AI Suite Ahead-of-Time Splitter Utility x
4.1. Files Generated by the FPGA AI Suite Ahead-of-Time (AOT) Splitter Utility 4.2. Building the FPGA AI Suite Ahead-of-Time (AOT) Splitter Utility 4.3. Running the FPGA AI Suite Ahead-of-Time (AOT) Splitter Utility 4.4. FPGA AI Suite Ahead-of-Time (AOT) Splitter Utility Example Application
5. CSR Map and Descriptor Queue x
5.1. Discovery ROM 5.2. Interrupt Control 5.3. DMA Descriptor Queue 5.4. DMA Control Registers 5.5. Performance Registers 5.6. Debug Network Registers 5.7. DMA License Register
1. FPGA AI Suite IP Reference Manual
2. About the FPGA AI Suite IP
3. FPGA AI Suite IP Generation Utility
4. FPGA AI Suite Ahead-of-Time Splitter Utility
5. CSR Map and Descriptor Queue
A. FPGA AI Suite IP Reference Manual Archives
B. FPGA AI Suite IP Reference Manual Document Revision History

2.2. Model Performance

The performance estimator tool (described in the FPGA AI Suite Compiler Reference Manual ) assumes the following fMAX values for FPGA devices:
  • Arria® 10: 265 MHz
  • Agilex™ 7: 400 MHz
These assumptions are reasonable and conservative for the standard speed bin. As shown by the results in this section, the achieved fMAX of the example design typically exceeds these assumptions.

The performance results for the designs that follow were achieved using the dla_build_example_design.py script that is included with the FPGA AI Suite. The script uses a standard (-2) speed bin with a single seed and uses high-effort compiler settings.

The runtime hosts used for determining the performance results are as follows:
  • Agilex™ 7 runtime host: SLES12 host on an Intel® Xeon® processor E5-1650 @ 3.5 GHz.
This design uses a dedicated DDR interface for the IP. The batch size is 1. Performance varies based on the clock speed, the DDR latency and bandwidth, and, depending on the graph, the host CPU speed.
The dla_build_example_design.py script includes the following two .qsf lines to enable non-default Quartus® Prime options during design compilation: 
set_global_assignment -name ALLOW_SHIFT_REGISTER_MERGING_ACROSS_HIERARCHIES ALWAYS
set_global_assignment -name DISABLE_REGISTER_MERGING_ACROSS_HIERARCHIES OFF

The architectures in the tables that follow are in the $COREDLA_ROOT/example_architectures/ directory. Review the README file in that directory for information about each architecture.

Details - FPGA AI Suite 2024.2

Architecture fMAX ALMs DSPs M20Ks Registers
AGX7_FP16_Generic 616 MHz 32.5 k 186 501 100 k
AGX7_FP16_Performance 600 MHz 103. k 1162 1533 346 k
AGX7_Small_NoSoftmax 612 MHz 16.7 k 80 296 54 k
AGX7_Small_Softmax 610 MHz 18.3 k 90 304 57 k
AGX7_Generic 600 MHz 38.6 k 202 778 126 k
AGX7_Performance 585 MHz 70.5 k 650 1278 209 k
AGX7_Performance_Giant 537 MHz 127.9 k 1546 2371 372 k

public/mobilenet-v1-1.0-224

Architecture ALMs DSPs DDR 1

[MB/s]

 Throughput

[fps]

 Top-1

[%]

 Top-5

[%]
AGX7_FP16_Generic 32.5 k 186 2325 176 71.2 89.5
AGX7_FP16_Performance 103. k 1162 8845 555 71.2 89.5
AGX7_Small_NoSoftmax 16.7 k 80 2774 168 70.9 89.6
AGX7_Small_Softmax 18.3 k 90 2765 167 70.9 89.5
AGX7_Generic 38.6 k 202 3247 250 70.9 89.5
AGX7_Performance 70.5 k 650 6231 397 70.9 89.5
AGX7_Performance_Giant 127.9 k 1546 4755 785 70.9 89.6

public/mobilenet-v2

Architecture ALMs DSPs DDR1

[MB/s]

 Throughput

[fps]

 Top-1

[%]

 Top-5

[%]
AGX7_FP16_Generic 32.5 k 186 3720 151 71.8 89.6
AGX7_FP16_Performance 103. k 1162 6979 374 71.8 89.6
AGX7_Small_NoSoftmax 16.7 k 80 4527 139 71.6 89.6
AGX7_Small_Softmax 18.3 k 90 4532 139 71.8 89.4
AGX7_Generic 38.6 k 202 2635 197 71.8 89.4
AGX7_Performance 70.5 k 650 5804 278 71.7 89.4
AGX7_Performance_Giant 127.9 k 1546 4242 720 71.7 89.4

public/mobilenet-v2-1.4-224

Architecture ALMs DSPs DDR1

[MB/s]

 Throughput

[fps]

 Top-1

[%]

 Top-5

[%]
AGX7_FP16_Generic 32.5 k 186 4194 125 74.8 91.9
AGX7_FP16_Performance 103. k 1162 8833 294 74.8 91.9
AGX7_Generic 38.6 k 202 4074 147 74.7 91.8
AGX7_Performance 70.5 k 650 6881 229 74.7 91.8
AGX7_Performance_Giant 127.9 k 1546 5729 644 74.6 91.8

public/mobilenet-v3-large-1.0-224-tf

Architecture ALMs DSPs DDR1

[MB/s]

 Throughput

[fps]

 Top-1

[%]

 Top-5

[%]
AGX7_FP16_Generic 32.5 k 186 3831 171 75.8 92.1
AGX7_FP16_Performance 103. k 1162 11087 237 75.8 92.1
AGX7_Generic 38.6 k 202 4420 176 72.3 90.7
AGX7_Performance 70.5 k 650 9165 193 72.1 90.5
AGX7_Performance_Giant 127.9 k 1546 7637 319 72.4 90.4

public/resnet-50-tf

Architecture ALMs DSPs DDR1

[MB/s]

 Throughput

[fps]

 Top-1

[%]

 Top-5

[%]
AGX7_FP16_Generic 32.5 k 186 3081 32 76.8 92.9
AGX7_FP16_Performance 103. k 1162 11626 164 76.8 92.9
AGX7_Small_NoSoftmax 16.7 k 80 5950 28 77.0 92.9
AGX7_Small_Softmax 18.3 k 90 5931 28 77.1 92.9
AGX7_Generic 38.6 k 202 4205 60 77.1 92.9
AGX7_Performance 70.5 k 650 10180 144 76.9 92.9
AGX7_Performance_Giant 127.9 k 1546 7838 230 76.9 92.8

Resnet50 v1 (Caffe)

Architecture ALMs DSPs DDR1

[MB/s]

 Throughput

[fps]

 Top-1

[%]

 Top-5

[%]
AGX7_FP16_Generic 32.5 k 186 2894 39 74.4 91.4
AGX7_FP16_Performance 103. k 1162 11970 193 74.4 91.4
AGX7_Small_NoSoftmax 16.7 k 80 4171 37 74.1 91.4
AGX7_Small_Softmax 18.3 k 90 4156 37 74.2 91.3
AGX7_Generic 38.6 k 202 4490 73 74.2 91.3
AGX7_Performance 70.5 k 650 10214 164 74.0 91.4
AGX7_Performance_Giant 127.9 k 1546 7595 245 74.1 91.4

intel/unet-camvid-onnx-0001

Architecture ALMs DSPs DDR1

[MB/s]

 Throughput

[fps]
AGX7_FP16_Generic 32.5 k 186 828 1.09
AGX7_FP16_Performance 103. k 1162 4298 7.14
AGX7_Small_NoSoftmax 16.7 k 80 1121 1.08
AGX7_Small_Softmax 18.3 k 90 1114 1.08
AGX7_Generic 38.6 k 202 1262 2.05
AGX7_Performance 70.5 k 650 3691 6.28
AGX7_Performance_Giant 127.9 k 1546 4198 9.06

public/yolo-v3-tf

Architecture ALMs DSPs DDR1

[MB/s]

 Throughput

[fps]

 Detection mAP @0.5 Detection mAP @0.5:0.95
AGX7_FP16_Generic 32.5 k 186 1422 4.2 62.27 31.58
AGX7_FP16_Performance 103. k 1162 6284 27.6 62.25 31.58
AGX7_Generic 38.6 k 202 1795 7.8 62.28 31.49
AGX7_Performance 70.5 k 650 2662 11.6 62.22 31.47
AGX7_Performance_Giant 127.9 k 1546 4918 30.0 62.25 31.46

public/yolo-v3-tiny-tf

Architecture ALMs DSPs DDR1

[MB/s]

 Throughput

[fps]

 Detection mAP @0.5 Detection mAP @0.5:0.95
AGX7_FP16_Generic 32.5 k 186 1073 37 35.79 14.77
AGX7_FP16_Performance 103. k 1162 4567 113 35.81 14.78
AGX7_Generic 38.6 k 202 1969 66 35.76 14.74
AGX7_Performance 70.5 k 650 1604 40 35.73 14.72
AGX7_Performance_Giant 127.9 k 1546 2980 64 35.81 14.75

public/yolo-v8-nano detection

Architecture ALMs DSPs DDR1

[MB/s]

 Throughput

[fps]

 Detection mAP @0.5 Detection mAP @0.5:0.95
AGX7_FP16_Performance 103. k 1162 6154 96 51.15 36.52
AGX7_Generic 38.6 k 202 1942 40 51.14 36.50
AGX7_Performance 70.5 k 650 2135 34 51.10 36.48

public/yolo-v8-nano classification

Architecture ALMs DSPs DDR1

[MB/s]

 Throughput

[fps]

 Top-1

[%]

 Top-5

[%]
AGX7_FP16_Performance 103. k 1162 4761 636 67.92 87.72
AGX7_Generic 38.6 k 202 1628 280 67.96 87.86
AGX7_Performance 70.5 k 650 1232 164 67.72 87.72

public/squeezenet1.1

Architecture ALMs DSPs DDR1

[MB/s]

 Throughput

[fps]

 Top-1

[%]

 Top-5

[%]
AGX7_FP16_Generic 32.5 k 186 649 225 58.5 81.1
AGX7_FP16_Performance 103. k 1162 4468 898 58.5 81.1
AGX7_Small_NoSoftmax 16.7 k 80 924 220 58.5 81.0
AGX7_Small_Softmax 18.3 k 90 920 219 58.5 81.0
AGX7_Generic 38.6 k 202 1713 532 58.5 81.0
AGX7_Performance 70.5 k 650 2155 432 58.4 81.0
AGX7_Performance_Giant 127.9 k 1546 2767 724 58.3 81.1

public/i3d_rgb_tf

Architecture ALMs DSPs DDR1

[MB/s]

 Throughput

[fps]

 Top-1

[%]

 Top-5

[%]
AGX7_FP16_Generic 32.5 k 186 449 0.62 65.79 82.89
AGX7_FP16_Performance 103. k 1162 2393 3.87 65.79 82.89
AGX7_Small_NoSoftmax 16.7 k 80 488 0.58 65.35 83.11
AGX7_Small_Softmax 18.3 k 90 487 0.57 65.57 83.11
AGX7_Generic 38.6 k 202 728 1.33 65.57 83.11
AGX7_Performance 70.5 k 650 2341 3.78 65.13 83.11
AGX7_Performance_Giant 127.9 k 1546 2571 4.20 65.79 82.89

Section Content
Throughput on the MobileNetV1 model (and other very fast models)

* DDR is estimated minimum average read + write (that is, read + write require at least this much bandwidth on average). Peak bandwidth is higher.
Level Two Title