GoPro Fusion* software not only allows consumers to create innovative 360-degree action videos of their latest exploits on land, sea, and air, now users can extract a standard 1080p video from that 5.2K feed. This technique goes by many names: GoPro* dubbed it "OverCapture—The Art of Editing Differently," while Intel account manager Jean-Pierre Navarro calls it "The Magic Window." Whatever the name, you now have the ability to create a seamless, spherical, 360-degree image and reframe it into a standard 2D format. To do so, you'll need some serious computing capabilities, but the good news is that Intel and GoPro have collaborated to allow the software to take full advantage of modern multicore processors.
"GoPro Fusion Studio software takes your 360-degree video edits to the max! You can easily create incredible 360-degree videos, but you can also utilize the amazing OverCapture experience to reframe just what you need from captured content."
Alexandre Jenny, GoPro senior director for Immersive Media Solutions;
In this document, you will learn about some of the GoPro Fusion software capabilities such as image stabilization, image stitching, and OverCapture, and understand at a high level how the CPU and graphics processing unit (GPU) optimizations performed together with Intel's Core and Visual Computing Group, resulting in a better-balanced computing load.
Figure 1. GoPro Fusion* V1.2 software includes plugins for Adobe Premiere Pro* and Adobe After Effects* to edit 360-degree content (align, reframe, add 2d layers).
About the System
GoPro has long been a leader in capturing high-quality action videos, and the new GoPro Fusion camera has even more to offer. Its powerful dual-lens system is designed to be the ultimate capture device for compelling content, creating video in 5.2K (5120 × 2560) resolution at 30 frames per second (fps).
Figure 2. The new GoPro Fusion* features two lenses to capture 360-degree action videos.
The increase in video resolution means more demand for processing power to turn that raw footage into watchable video, which is where Intel's collaboration with GoPro Fusion Studio* makes a big difference. Steps include stitching together images from the two feeds, extracting the desired 1080p view or the full 5.2K stream, editing, rendering, compressing, and uploading for sharing. Even the newest PC systems can strain to complete those tasks if the software isn't optimized to share the load between the cores and the GPU.
"If you want to work at the highest resolution, it's going to be too crazy to do anything on a PC you bought three or more years ago."
Jean-Pierre Navarro, Intel account manager
For example, stitching is a balance of CPU and GPU usage. Depending on the quality you stitch and application of stabilization and parallax correction (using the D.WARP algorithm, explained below), you will stress your CPU and GPU at different levels.
Fortunately, Intel and GoPro* teamed up to make the publishing stage faster and easier for consumers with the right system. Using Intel® Quick Sync Video for decoding the two input videos, performance testing showed significant improvements on laptops using Intel® Graphics Technology.
When users are back home on their home PC, they can perform advanced film editing using plugins to additional software titles such as Adobe After Effects* or MAGIX Movie Edit Pro*. To understand how GoPro Fusion's free plugin interacts with Adobe Premiere Pro*, check out this video. And the GoPro Fusion VR Plugins are available.
Demanding Users, Demanding Specs
The GoPro Fusion camera uses two lenses located on the front and the rear of the camera, slightly offset; each record nearly 4K of resolution in JPEG, MP4, and RAW formats onto two different microSD, microSDHC, or microSDXC cards. With overlap and black space around the circular image projected onto the rectangular image sensor, the two feeds can combine into a 5.2K 360 video. GoPro recommends identical Class 10 or higher memory cards, with a maximum capacity of 128 GB per slot.
Output codec options include H.264 for downsizing footage to 4K, or CPU-demanding CineForm 422 and Apple ProRes 422 for 5.2K. Available presets tune output for different purposes—editing, Facebook*, Vimeo*, and YouTube*. There are several options for video resolution—5.2K, 4K, 3K, and 2K. Users can also choose between stereo or 360-degree directional audio.
Clearly, users intending to produce and upload top-quality action videos want more than consumer-level specs to handle the raw video. They'll need plenty of RAM, and big hard drives, too. Every minute of stitched CineForm 422 footage at 5.2K takes up about 4.5 GB of space. Even still images in 18-megapixel resolution take up plenty of room.
Rendering speeds for a 360-degree clip depend primarily on how powerful your PC is. Users are taking it upon themselves to benchmark the software and report the results on social media. One user remarked that it took more than seven hours to process a five-minute 5.2K clip on their 2011 MacBook* Pro. Once they upgraded, the time was cut dramatically.
Alexandre Jenny, Senior Director Immersive Media Solutions at GoPro
Intel's Navarro noted that the relationship between GoPro and Intel has been developing for about ten years. "We've been working intensively with GoPro to tap into the CPU and the GPU both," Navarro said. "Consumers require a very easy and fast workflow to be able to capture and post quickly. We worked with GoPro to understand all phases of the work. The software is now heavily multithreaded and vectorized because the tasks involved are very CPU-intensive; the latest Fusion software now also includes GPU optimization using Intel Quick Sync Video for decoding input videos."
The teams regularly communicated, with GoPro seeking technical information on multicore processing and Intel providing specifications, training, and software assistance. The coordination has paid off for GoPro—according to reports, from the alpha phase through final release, rendering speed increased significantly.
Consumers benefit from these optimizations in many ways, including experiencing a faster user interface and witnessing real-time previsualization during rendering. To fully appreciate the optimizations, users should adhere to GoPro Fusion's recommended system requirements:
CPU—Minimum: Intel® Core™ i5 processor. Recommended: Intel® Core™ i7 processor or better.
GPU—Minimum: Iris® Pro graphics 6200 integrated graphics card supporting OpenGL* 4.1 or later. Recommended: Dedicated GPU such as AMD Radeon* R9 290 with 4 GB RAM, NVIDIA GeForce* GTX 760 graphics card with 4 GB RAM, or better.
Video RAM—Minimum: 4 GB RAM.
Video Stabilization Through Hardware and Software
Image stabilization for video recording refers to a family of techniques that reduce jitter while recording a moving image. Without compensating for pan and tilt, a visible frame-to-frame jitter can render the video unsuitable for sharing, so there are technologies implemented in the lens itself or in the optical path, before the sensor converts the image into digital information.
GoPro Fusion has standard gyro-based stabilization in the hardware, which ensures smooth source video and makes it possible to hold the camera at any angle. Inside GoPro Fusion Studio, digital image stabilization occurs as a post-processing step, after content is recorded. This has the advantage of directing more computing power to the problem; images can be analyzed both before and after a particular frame, extensively using Intel® Advanced Vector Extensions 2 (Intel® AVX2) and multiple cores.
Product reviewers have been positive about this software feature. PC Magazine said Fusion Studio delivered "…footage that is incredibly smooth and steady. It even passed our torture test, delivering stable video when mounted to the vibrating hood of a farm tractor. Other cameras we put to the same test … delivered noticeably shaky results."
Stitching Together Images Creates Seamless View
GoPro Fusion users have the ability to capture more video than they actually need, as the two offset cameras can pull in a 360-degree image that is a round fish-eye view of everything captured (below).
Figure 3. The typical "fish-eye" view of a portion of 360-degree capture.
Imagine a skateboarder setting up for a shoot in a park; rather than installing multiple cameras on tripods around the venue, a user could set the GoPro Fusion camera up on a tripod somewhere near the middle of the scene, then get coverage of much of the area, as shown below.
Figure 4. One GoPro Fusion* camera can cover an extensive area.
This would enable the user to perform tricks and stunts around the tripod, capturing the moves on both camera feeds. Back in the studio—or even still at the park, using a powerful laptop—the resulting video feeds must first be "stitched" together in the software to remove duplicated data and provide a seamless view. This is a compute-intensive operation—the software interprets pixels, matches shading, and merges the edges—which requires syncing the time stamps and using powerful algorithms. Refer to figure five for a representation of this step.
Figure 5. Two high-quality feeds are stitched together into a single, 360-degree video.
Stitch quality is important to compensate for parallax and exposure differences between the lenses in many scenarios, such as when a close object appears in both views, and the sun is high in the sky. If the skies are cloudy, without extremes in brightness, exposures will be better balanced, but the GoPro software can also handle bright skies.
Processing power for stitching quality also comes into play to create what GoPro calls "Angel View". When used with a sufficiently slim monopod, the Fusion camera remains in the view while the monopod is removed from the scene. This occurs when the two feed images are stitched together. Basically, Angel View automatically erases the pole from the frame, making it look like the camera is floating in the shot or was removed completely (see figure six).
Stitching the images together is a highly compute-intensive process, which requires lining up each image, pixel by pixel using several advanced algorithms, according to GoPro's Alexandre Jenny. "Our proprietary D.WARP algorithm transforms the fish-eye images into the spherical space … minimizing visual artifacts," he said. By transforming the image dynamically and in a time-coherent manner, D.WARP's content-aware stitching is the first of its kind.
Once the image warping has been corrected with D.WARP, the blending algorithm is applied to merge several source pixels, creating a more equalized image. This stage adjusts for exposure and contrast differences and is also compute-intensive.
GoPro and Intel software engineers spent considerable time optimizing the stitching portion of the workflow. The Intel team worked with their GoPro counterparts to use Intel® Graphics Performance Analyzers (Intel® GPA) Monitor, a client-monitor application that enables Intel GPA System Analyzer to get an overview of the software platform usage and tune its CPU- and GPU-concurrent processing. Intel engineers also extensively used Intel® VTune™ Amplifier to get a fine-grained analysis of the critical path in the software to enhance it with better multithreading and use of Intel AVX2.
Figure 6. "Angel View" refers to the ability to strip out the camera and monopod during stitching.
OverCapture: The Art of Editing Differently
Once the complete 360-degree view is available, users can then select which clip of the 5.2K video they want to track using GoPro VR plugins and the video editor of their choice.
Visualizing OverCapture.
Figure 7. Users can select from one of the grids in the 360-degree view to extract their preferred stream.
Once the desired portion (or portions, if video editors want to use multiple angles) is extracted, the video can be put together with Adobe Premiere Pro and Adobe After Effects.
Intel's Jean-Pierre Navarro believes anyone working with their GoPro footage will want the newest possible system. "Consumers producing professional-grade videos will probably want to upgrade their PC to get the most out of their files," he said. "If you want to work at the highest resolution, it's going to be too crazy to do anything on a PC you bought three or more years ago. You're going to spend too much time, and it will be too hard to do anything fun with OverCapture, for example."
Navarro said the engineers from Intel and GoPro tested some system-stressing scenarios using sixth, seventh, and eighth-generation CPUs in various form factors, desktops, and laptops. "If you compare desktop to desktop with eighth and ninth-generation CPUs, you see significant performance improvements," he said. Even laptop systems with Intel integrated graphics will benefit from the optimization work. "We worked with GoPro to optimize their code together, suggested ways to optimize workflows on the CPU and the GPU, and it really paid off for consumers."
GoPro Fusion Studio has the flexibility to be used on a notebook or a desktop, depending on the edits you want to achieve. Cool features such as tackling stitching issues with D.WARP parallax compensation at 5.2K, video transcoding, or stabilization are more compute-intensive, but the Intel® Core™ i7 processor and Intel® Core™ i9 processor families provide the power for you to create and shape your projects in the best possible conditions, to the highest possible standard.