Workload Consolidation in Industrial IoT

Using Virtualization Technologies to Transform Operations for a More Efficient Industry

Executive Summary

The world’s highly competitive business environment drives companies to optimize operations wherever possible to reduce costs, increase efficiencies, satisfy customers, and innovate new products and services. Historically, across automated manufacturing floors, myriads of individually computerized and de-centralized workloads, such as images, video, network data, and others, drove machine operations with different levels of control that enabled manufacturers to scale. With the Internet of Things (IoT) and digitalization transforming industrial operations, more and more of these controls are now being centralized onto a common compute platform. As manufactures seek direct communication across different equipment in the factory, such as PLC, HMI, PAC, and etc., new standards and frameworks such as Time Sensitive Networking (TSN) for OT-IT convergence, functional safety for predictable fail-safe operations, deep learning models for better data analytics, and others, are gaining stronger traction in the Industrial IoT (IIoT) ecosystem. With such business drivers and the need to stay competitive, companies seek new ways to condense technologies and workload, processing into as few physical components as possible. Such industry requirements form the basis of industry 4.0 modernization and give way to the principles of workload consolidation.

Workload consolidation unites multiple computerized operations onto fewer platforms, replacing separate purpose-built hardware machines with a smaller foundation of general purpose compute technologies. Reducing hardware infrastructure and utilizing more of existing resources helps optimize operations. Such an approach to IT and industrial automation can reduce capital costs (CapEx) and increase efficiencies and simplify operations to reduce operational costs (OpEx).

This paper underlines the tenets of workload consolidation, Intel® technology offerings, architectural considerations, and recommendations for customers as they analyze operational efficiencies in their factories. As requirements vary by customers, workload consolidation projects would vary for different approaches. For example, a food processing plant would greatly differ from a car manufacturing facility. However, the principles for workload consolidation overall remain similar.

Workload Consolidation in IoT

Industrial machines for several decades have had built-in, proprietary digital systems that are as physically isolated within the machine as they are logically. They communicate with each other over digital links—also often proprietary. PLCs, HMIs, industrial PCs (IPCs), and PACs run fixed- function applications, calling on other subsystems to move material, interface with operators, connect to servers, protect employees, and execute a function. Such a siloed and disparate architecture results in high maintenance costs, increased complexity, and difficulty in scaling to accommodate growth. Consolidating workloads onto fewer platforms help address these issues.

Enabling Technologies – Virtualization and Containerization

Unlike running multiple user applications on a single, multi- tasking operating system (OS), workload consolidation creates separate, isolated environments, each with its own OS, on a single, powerful platform. Two key technologies that enable workload consolidation are virtualization and containerization.

As shown in Figure 1, virtualization¹ creates isolated environments (called Virtual Machines—VM) that access the underlying hardware through a central layer called a hypervisor. Each VM runs as though it were the only environment on the platform with its OS and virtualized network and storage resources. Containerization² creates isolated processes in user space (at the application layer) that share the same kernel and non-virtualized hardware resources through a central application (e.g., Docker*). Each method has its own strengths and applicable use cases.

While containers are gaining traction in IIoT, virtualization is often the method of choice in industrial automation and IT infrastructures, where complete isolation is often paramount to operations.

Another parallel concept that also helps to optimize manufacturing infrastructure is the process of creating a digital twin. A digital twin is a virtual model of a process, product, or service. This pairing of the virtual and physical worlds allows analysis of data and monitoring of systems to head off problems before they even occur. Digital twin technology is different from virtualization, as virtualization essentially drives more throughput from an asset, whereas a digital twin optimizes a process or an asset. Both virtualization and digital twin technology drive efficiencies in the manufacturing process, but in very different ways.

A Modern Industrial IoT Model

Modernized IIoT infrastructure is software-defined and driven. It leverages workload consolidation to decouple the hardware from software and drive operational functionality into the applications instead of dedicated hardware. Each software-defined environment operates as a standalone, isolated subsystem running its applications, while the common underlying hardware provides the needed resources for all operations. The application communicates with dedicated hardware, such as servos and sensors, over a standardized interface, such as USB or Ethernet. For real- time critical operations, different components inside the networking infrastructure of the factory, such as bridges, endpoints, and etc., likely conform to emerging standards, such as TSN or OPC-UA for interoperability. For low cost compute upgrades, many manufacturers also look at various accelerators or FPGA’s for dedicated computing resources.

To support flexibility and interoperability, this type of infrastructure needs to be built on a common, open standards- based, virtualized platform capable of consolidating, running, and managing any type of industrial or compute-heavy workload. Such an approach can extend from the operations center across the plant to the edge systems, such as mills, robots, and packaging machines (Figure 2).

Figure 1. Differences between virtualization and containerization.

Figure 2. Workload Consolidation at the Edge.

Primary Benefits and the Ecosystem

This approach to industrial operations helps reduce costs, increase operational efficiencies, and enable agility because of the following:

• Consolidation eliminates many dedicated, costly hardware components that require unique spares for repair.
• Platform standardization allows leveraging off-the-shelf components, thereby increasing the range of hardware choices, reducing total cost of ownership (TCO), and preventing vendor lock-in around proprietary hardware.
• A simpler overall architecture means fewer components that can fail and faster repair times.
• Functional upgrades can be completed either partly, or even entirely, with new software that can be pushed to the system by the manufacturer, nearly seamlessly and invisible to the plant.

Early virtualization deployments of over a decade ago taxed system performance, and solution providers offered little support for a software-defined infrastructure. But, advances in multi-core CPUs, real-time operating systems (RTOS), lightweight hypervisors, hardware-assisted virtualization, and an evolving, modernized solution stack for IIoT

(Table 1) have significantly boosted the responsiveness of virtualized systems and enabled performant solutions. Equipment developers can now consolidate deterministic and non-deterministic applications onto a single board, creating a credible path to streamlined operations, improved productivity, and reduced cost and complexity (Table 1).

Table 1. Conceptualized ecosystem-supported workload consolidation stack built on existing components.

Workload Requirements Influencing Consolidation

When approaching workload consolidation, engineers must take several factors into consideration to design a platform that helps ensure operational success.

Worst-Case Execution Time (WCET)

WCET defines the longest execution time that is acceptable for the workload   to be successful. Different functional processes have various WCETs. Here are some to consider:

• For interactive workloads (e.g., HMI), the WCET is simply an observed tolerable wait time for the application to respond.
• Non-interactive workloads range from periodically scheduled batch jobs, such as data transformation and transfer from source to destination, to applications that control the functioning of manufacturing tools on the shop floor.
• Batch jobs typically define their WCET as the duration beyond which they would interfere with other jobs or would simply fail to complete before the next cycle starts.
• Control applications typically have stringent time determinism specifications and require WCET to not exceed a very tight window that may span only microseconds, (such as functional safety). Such control applications require the utmost attention when planning and testing for workload consolidation.

Bandwidth Capability

Individual workloads have peak bandwidth requirements during their life cycles. Bandwidth requirements rise and fall over the life cycle, but never exceed the specified peak bandwidth. Some bandwidth considerations include:

• The platform’s bandwidth capabilities are specified by the platform provider.
• If the workloads are not time division multiplexed (TDM), bandwidth needs can peak at the same time. The platform needs to support the sum of the peak bandwidths for all workloads being consolidated.
• If the workloads are TDM, the platform may be able to support all the workloads, even though the sum of their peak bandwidths exceeds its bandwidth capability.

Logical Isolation

Workloads may have a need for total logical isolation (partitioning) from each other. This guarantees that no other workloads on the system will interfere with its execution and address space. Partitioning requirements play an important role in determining how much consolidation is possible on a given platform. Virtualization adds a certain overhead on the platform. It also limits what underlying resources are accessible by each workload to ensure one does not trespass certain boundaries of another.

The platform’s architecture can offer unique virtualization technologies (VT) to mitigate limitations. A platform’s integrated VT become critical to help support partitioning.

Determinism

Workloads with deterministic functions (e.g., functional safety) impose strict execution requirements.

Such workloads need to have temporal isolation so that other workloads do not interfere with their execution deadlines. This is typically accomplished through a combination of one or more capabilities, such as time-determinism features of the underlying platform, VT support, etc. If the workloads are staggered in time for execution, the platform may be able to achieve better consolidation through TDM. Careful attention must be given to the temporal requirements of the individual workloads for successful consolidation.

Platform Requirements

The platform must accommodate the storage and power requirements of all workloads being consolidated. For platforms that intelligently manage power consumption, engineers should validate that such a feature does not interfere with the WCET of the individual workloads.

The above characteristics are just some that engineers should consider during consolidation design. All designs require comprehensive test and validation coverage to ensure that no parameters of the workload requirements, the platform features, and the environment in which the platform executes are violated.

Enabling Workload Consolidation Through Virtualization

Workload consolidation is typically accomplished by virtualization. In the industrial automation application space, the International Society of Automation’s ISA-95 model is the standard that drives consolidation for integrating enterprise and production control systems.³

Virtualization and Hypervisors

As described above, virtualization partitions the host platform into multiple software-defined, isolated environments with shared resources managed by a hypervisor. Several hypervisors are available on the market—both open source and commercial. Hypervisors are characterized as Type 1 and Type 2.

• Type 1, also called bare metal or native hypervisors, provide their own operating systems for booting, running the hardware, and connecting to the network. Popular Type 1 hypervisors include Wind River Hypervisor*, Microsoft Hyper-V*, and VMware ESXi*.
• Type 2, or hosted hypervisors, run on top of an operating system installed on the platform. Once the operating system is running, the hosted hypervisor launches. Popular Type 2 hypervisors include VMware Workstation*, VirtualBox*, and Parallels*, which emulates a Windows* operating system while running on a Mac*-based computer.

Table 2. Key differences between Type 1 and Type 2 hypervisors.

	Type 1 Hypervisor (Bare Metal or Native)	Type 2 Hypervisor
Installation	Works directly on the hardware of the host and can	Installed on an operating system and supports other monitor operating systems that run above the hypervisor operating systems above it
OS	Completely independent of the OS	Completely dependent on the host OS for its operations
Memory	Small: its main task is sharing and managing hardware resources between different operating systems	Bigger memory footprint: while having a base OS allows better specification of policies, any problems in the base OS affect the entire system, even if the hypervisor running above the base OS is secure
Advantage	Any problems in one VM or guest OS do not impact the OS running on the hypervisor Allows for much better real-time (RT) turnaround	In a scenario where a customer has a legacy OS running a proprietary embedded application but wants to co-host a newer application that the legacy OS cannot support, one can host a newer OS that supports the newer application in the legacy OS environment
Typical Layers

Table 3. Requirements of embedded industrial hypervisors.

Hypervisors in the Industrial Segment

Industrial applications typically require an RTOS, sophisticated networking, and security against cyber-attacks and malware (Table 2). A hypervisor is the ideal way to separate systems and maintain security. For deterministic- seeking embedded applications, Type 1 hypervisors make the most sense for the majority of applications.

Hardware-Assisted Virtualization

Hypervisors have matured greatly over the years, offering important features that enhance performance, isolation, and security. Because virtualization is a transformational technology for a software-defined infrastructure, processors have also evolved to support virtualization directly in silicon. In Intel® processors, the feature set is called Intel® VT. This technology is a portfolio of silicon capabilities that assist the virtualization software.

Engineers should carefully consider the specific needs of the workloads to be consolidated, the hypervisor that will support consolidation, and the hardware and silicon features of the platform needed to successfully support the virtualized environments.

Table 4. Ecosystem virtualization solutions.

Intel® Virtualization Technology (Intel® VT) Portfolio

Intel® VT eliminates performance overheads occurring in cache, I/O, and memory and by improving security. Providing hardware assistance to the virtualization software, Intel® VT helps reduce the size, cost, and complexity of virtualization. A significant number of hypervisor vendors, solution developers, and users leverage and have enabled Intel® VT in their offerings.

Intel® VT assists virtualization across multiple components of a system architecture.

CPU virtualization features enable faithful abstraction to a VM of an Intel® CPU’s full capabilities. CPU virtualization features of Intel® VT include the following:

• Intel® processors with Intel® Mesh Architecture enhance performance compared to older “ring” processor architectures by adding new pathways to eliminate bottlenecks between the CPU cores, cache, memory, and I/O.
• Processors equipped with mode-based execution control (MBE) virtualization (e.g., Intel® Xeon® Scalable processors) provide an extra layer of protection from malware attacks in a virtualized environment. MBE enables hypervisors to more reliably verify and enforce the integrity of kernel- level code.
• Timestamp-counter scaling (TSC) virtualization allows virtual machines to move across CPUs operating at different base frequencies, enabling workload optimization in hybrid cloud environments.

Memory virtualization features allow abstraction isolation and monitoring of memory on a per-VM basis. These features also make live migration of VMs possible, add to fault tolerance, and enhance security. Example features include direct memory access (DMA) remapping and extended page tables (EPT), including their extensions: accessed and dirty bits, and fast switching of EPT contexts.

I/O virtualization features facilitate offloading of multi- core packet processing to network adapters, as well as direct assignment of VM to virtual functions, including disk I/O. These features include Virtual Machine Device Queues (VMDQ), Single Root I/O Virtualization (SR-IOV, also a PCI-SIG standard), and Intel® Data Direct I/O Technology enhancements (Intel® DDIO).

Intel® Graphics Virtualization Technology (Intel® GVT) allows VMs to have full and/or shared assignment of the graphics processing unit (GPU) and the video transcode accelerator engines integrated in Intel® System on Chip (SoC) products. It enables usages such as remote workstations, desktop-as-a-service (DaaS), media streaming, and online gaming.

Figure 3. Wind River* Titanium Control

Recommended Ecosystem Providers

The key requirements discussed above should be carefully considered with all other aspects of a consolidation project. Intel® Xeon® processors, Intel® Core™ processors, and Intel Atom® processors integrate Intel VT to help support any consolidated platform. Based on an internal assessment conducted by Intel's Industrial Solutions Division (ISD), Intel recommends the following options (Table 3) for customers in need of an existing workload consolidation solution. See each provider’s website for additional details.

Wind River* Systems

A global leader in delivering software for intelligent connected systems, Wind River offers a comprehensive, end-to-end portfolio of solutions ideally suited to address the emerging needs of IoT, from the secure and managed intelligent devices at the edge, to the gateway, into the critical network infrastructure, and up into the cloud. Wind River technology is found in nearly 2 billion devices and is backed by world-class professional services and award-winning customer support.⁴

Wind River* Titanium Cloud

Wind River Titanium Cloud—consisting of Titanium Control, Titanium Edge, Titanium Edge SX, and Titanium Core— provides an application-ready software platform that runs virtual functions with high reliability and is built to support the intensive performance, reliability, and security requirements of demanding computing and communications networks.⁵ Titanium Control (Figure 4) is an on-premises cloud infrastructure platform that delivers the uptime and performance needed for industrial applications and control services at any scale.⁶

Project ACRN*

ACRN* is an open source, flexible, lightweight reference hypervisor designed for the unique needs of IIoT (Figure 5). It is built for real-time and safety-criticality and optimized to streamline embedded development. More information on Project ACRN can be found at ACRN.

Figure 4. Project ACRN*

Intel® Xeon® Scalable Processors for Workload Consolidation

Intel’s portfolio of processors all support Intel® VT—Intel Xeon Scalable processors, Intel® Core™ processors, and Intel Atom® processors. The latest Intel Xeon Scalable processors provide a foundation that creates an evolutionary leap in agility and scalability. Intel Xeon Scalable processors are highly configurable and scalable products from Intel and offer customers the ability to create multi-socket customized solutions. Products with Intel Xeon Scalable processors feature Intel® Ultra Path Interconnect (Intel® UPI), which offers up to three Intel® UPI channels. This improves inter-CPU bandwidth for I/O-intensive workloads and offers the perfect balance between improved throughput and energy efficiency.

Intel Xeon processor Scalable family also offer comprehensive security features. Intel® QuickAssist Technology (Intel® QAT) comes integrated into these products. This acceleration technology improves performance across symmetric encryption and authentication, asymmetric encryption, digital signatures, RSA, DH, and ECC, and lossless data compression. Intel® Key Protection Technology (Intel® KPT) is enabled by Intel® QAT to help secure private key management. Together, these technologies provide near-zero encryption overhead to enable higher performance on all secure data transactions.

Intel Xeon processor Scalable family also offer advanced network capabilities, such as integrated Intel® Ethernet with scalable iWARP (Internet Wide-area Remote Direct Memory Access Protocol). These features provide up to four 10 Gbps high- speed Ethernet ports for high data throughput and low- latency workloads.

The next generation Intel Xeon Scalable processor, codenamed Cascade Lake, provides for a richer set of features for Artificial Intelligence for different kinds of workloads. With Cascade Lake, Intel has enabled support for Vector Neural Network Instructions (VNNI) set that allows for deep learning and enhanced neural networking performance.

These advances drive operational efficiencies that lead to improved TCO and higher productivity for users and are ideal products for consolidating different controls inside a factory. Systems built on Intel Xeon processor Scalable family are designed to deliver agile services and reduce TCO up to 65 percent due to lower software and OS licensing fees, and acquisition, maintenance, and infrastructure costs.⁷ Some implementations have enabled 4.2 times more VMs per server compared to earlier generations, allowing IT to increase their consolidation of more services on less hardware.⁸

Workload Consolidation in Industrial Segments

Workload consolidation takes different forms and satisfy different customer needs in a variety of segments. The drivers for virtualization are dependent on multi-facet use cases and potentially engage different eco-system partners. Intel is committed to drive horizontal value proposition in this effort through silicon features as well as through subsidiaries like Wind River.

Following are few examples on Intel’s engagement.

Rockwell Automation*

The Allen-Bradley CompactLogix 5480 Controller* based on an Intel® Core™ i7 processor hosts Logix industrial control and Windows® 10 IoT Enterprise operating systems in parallel.

It consolidates PLC, HMI, analytics, and user applications (Windows) into one system. It offers machine and line control, analytics, data concentrator, integrated control, and visualization functions that were typically based on individual devices. This example demonstrates the consolidation of physical controls for driving efficiency on the factory floor.

Beckhoff Automation*

Beckhoff Automation, an industry leader in implementing automation systems for industry and building, worked with Husky*, a global supplier of injection molding systems to the plastic manufacturing industry, on consolidating multiple control systems. This led to creation of Husky’s HyPET* systems—a solution for a range of packaging applications, including carbonated soft drinks.

Using its TwinCAT* software technology, Beckhoff created a PC-based control system (equipped with Intel® Core™ i7 processors) through which it can consolidate many conventional controls (such as PLC, motion control, hydraulics, robotics, and operator interface) into one powerful controller.

Conclusion

Industrial manufacturing customers are facing challenges in terms of managing costs, optimizing efficiencies, and improving system maintenance. Workload consolidation in the IIoT means fewer unique devices to manage and maintain, allowing for a smaller footprint with diminished system and environmental complexity.

Today, a typical factory might have multiple, different controls and systems throughout the plant running on several technologies. Such a distributed architecture causes scalability and operational challenges as more factory components at different levels of hierarchy are required to communicate with each other. Workload consolidation offers an opportunity for customers to move to more robust and modular architectures by replacing small processors with Intel processors.

As the industry moves towards greater automation with an increased emphasis on edge compute, security, scalability, and lower costs, trends towards workload consolidation will continue, and Intel’s products and IoT ecosystem partners will become increasingly important. Intel Xeon Scalable processors offer an ideal set of features to satisfy such requirements.

Intel is actively enabling multiple original design manufacturers (ODMs) in their workload consolidation projects. Intel is also supporting the developer ecosystem by creating various developer kits with pre-integrated hardware and software components to showcase various end customer use cases, such as predictive maintenance, machine vision, etc.

To learn more about Intel® technologies around workload consolidation solutions, visit Intel® Developer Zone.

1. Virtualization
2. Container (containerization or container-based virtualization)
3. Software-defined infrastructure in industrial IoT: How it works
4. Wind River*
5. Wind River Titanium Cloud
6. Wind River Titanium Control
7. Up to 65% lower 4-year TCO estimate example based on equivalent rack performance using VMware ESXi* virtualized consolidation workload comparing 20 installed 2-socket servers with Intel® Xeon® processor E5-2690 (formerly named “Sandy Bridge-EP”) running VMware ESXi* 6.0 GA using Guest OS RHEL* 6.4 compared at a total cost of $919,362 to 5 new Intel® Xeon® Platinum 8180 (formerly named "Skylake") running VMware ESXi* 6.0 U3 GA using Guest OS RHEL* 6 64 bit at a total cost of $320,879 including basic acquisition. Server pricing assumptions based on current OEM retail published pricing for Broadwell based servers – subject to change based on actual pricing of systems offered. Product Brief: Intel® Xeon® Scalable Platform
8. Up to 4.2 times more VMs based on server virtualization consolidation workload: Based on Intel® internal estimates 1-Node, 2 x Intel® Xeon® processor E5-2690 on Romley-EP with 256 GB Total Memory on VMware ESXi* 6.0 GA using Guest OS RHEL* 6.4, glassfish3.1.2.2, postgresql9.2. Data Source: Request Number: 1718, Benchmark: server virtualization consolidation, Score: 377.6 @ 21 VMs vs. 1-Node, 2 x Intel® Xeon® Platinum 8180 processor on Wolf Pass SKX with 768 GB Total Memory on VMware ESXi6.0 U3 GA using Guest OS RHEL* 6 64 bit. Data Source: Request Number: 2563, Benchmark: server virtualization consolidation, Score: 1580 @ 90 VMs. Higher is better. Product Brief: Intel® Xeon® Scalable Platform

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com

Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.

Estimated results were obtained prior to implementation of recent software patches and firmware updates intended to address exploits referred to as “Spectre” and “Meltdown.” Implementation of these updates may make these results inapplicable to your device or system.

Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction.