XuCode: An Innovative Technology for Implementing Complex Instruction Flows

Those who have read Volume 3D of the Intel Architecture Software Developers Manual, may have noticed that the instruction set added to support Intel® Software Guard Extensions (Intel® SGX) is quite involved. At the time Intel® SGX was in development, Intel understood that implementing such a complex set of instructions in our traditional manner would impact our ability to deliver many of the features of Intel SGX.

To tackle this problem, Intel created an innovative new technology that would allow us to rapidly develop the broad set of features required for Intel SGX. Intel decided to use this technology, called XuCode (pronounced zoo-code), to deliver parts of the Intel SGX implementation. This article describes XuCode and how it is used to implement Intel SGX.  

What is XuCode?

To understand what XuCode is, first we need to understand a little about microcode (sometimes written μcode). In a Complex Instruction Set (CISC) machine like x86, the stream of instructions read from memory are decoded into small operations known as micro-ops (μops). For some x86 instructions there is a simple 1:1 mapping to a μop, and in some cases the instruction maps to a sequence of μop instructions. The collection of μop sequences is what is often referred to as microcode. The amount of resources that can be dedicated to these microcode sequences in any given product is finite due to space and performance/design requirements. 

XuCode is implemented as a variant of 64-bit mode code, running from protected system memory, using a special execution mode of the CPU. It is authenticated and loaded as part of a microcode update and is installed into a Processor Reserved Memory (PRM) range, typically allocated by system firmware. The memory range itself is protected from software and direct memory accesses by the Processor Reserved Memory Range Registers (PRMRRs). XuCode has its own set of instructions based mostly on the 64-bit Instruction Set, removing some unnecessary instructions, and adding a limited number of additional XuCode-only instructions and model specific registers (MSRs) to assist with the implementation of Intel SGX.

Instantiating XuCode

XuCode is established in PRM as part of applying a microcode update early in the BIOS boot sequence, after the PRM has been configured and locked. During this sequence a module known as MCHECK, which is similar to the Authenticated Code Modules (ACMs) used by Intel® Trusted Execution Technology (Intel® TXT), implements a set of actions. Namely MCHECK:

• Establishes that the CPU configuration is properly set.
• Performs platform-specific Intel SGX initialization operations.
• Programs per-boot keys into the memory encryption engines designed to protect the PRM.
• Programs per-boot link keys into the CPU interconnect encryption engines designed to protect XuCode and Intel SGX related memory coherency traffic.
• Performs some memory initializations necessary before XuCode is installed.
• Passes Intel SGX-related information to XuCode via a dedicated page in the PRM.

Memory configuration checks undertaken by MCHECK include but are not limited to specific memory configuration choices which are allowable. In single socket systems where there is a single memory controller, the ability for the untrusted platform firmware to create aliases¹ in memory is limited, but still possible. In our Intel® Xeon® series servers where multiple Processor Reserved Memory (PRM) ranges are supported, not only does MCHECK have to check each socket’s view of the memory map, but MCHECK also checks that each socket view is consistent with the other sockets in the platform.

Upon successful completion of MCHECK, microcode decrypts and authenticates that the XuCode image in the microcode update is from Intel and then installs it in the memory space prepared by MCHECK. Microcode then invokes the initialization routines of the installed XuCode.

Defense in Depth

Additional hardware protections designed to protect XuCode are implemented. The XuCode memory range within the PRM (a.k.a. XuRAM) is split into code and data regions, and the CPU enforces that XuCode can only execute from the XuCode code region and cannot write to that region. 

By design, XuCode is deprivileged so that it cannot access microcode nor can it access internal register state and SRAMs. XuCode also does not issue special bus cycles that allow it to communicate with the trusted platform module (TPM) about locality events.

When programming XuCode, we treat it as an embedded environment. As such, all memory allocations in XuCode are static at design time, and there is no dynamic allocation of memory that can cause use-after-free weaknesses².  The Intel SGX instructions make extensive use of keyed symmetric cryptography, in particular, AES-GCM and AES-CMAC so traditional (i.e. non-transient) side-channel attacks are mitigated in the course of reviews of our implementations. Transient execution attacks are mitigated by either adjusting certain configuration features as we enter XuCode mode, or by adding the necessary transient execution defenses to our XuCode codebase.

Implementing SGX

The implementation of SGX is comprised of several major components:

• Isolation – changes to the memory access control semantics to prevent non-enclave software from accessing enclave memory.
• SGX memory management – capabilities to enable the operating system to manage the secure memory resources used to support Intel SGX operations and support debugging.
• Enclave operations – capabilities to enter/exit an enclave, create various enclave specific keys, and locally attest.
• Key Derivation and TCB Recovery – ability to derive new keys when the TCB of the platform is updated.
• Physical Memory Protections – cryptographic protections for memory as it leaves the CPU package and is placed in DRAM and when it travels across memory coherency busses.

In implementing these different Intel SGX components, we use a mixture of Hardware circuits, Microcode MCHECK and XuCode in combination as a defense-in-depth approach. The table below highlights the areas where we chose which method to implement the various aspects of Intel SGX functionality.

For Intel SGX Trusted Compute Base (TCB) recovery purposes, MCHECK and the first microcode update applied directly out of a CPU reset share the same Intel® SGX Security Version Number (SGX SVN).

Summary

XuCode is one of many important innovations that allow Intel the flexibility to more swiftly deliver complex features. To date, we have only used XuCode to implement parts of the Intel SGX instruction set and the infrastructure to set it up. 

Visit the Intel SGX home page for more information about Intel SGX.

Read the Intel Security Development Lifecycle whitepaper to learn about Intel’s Security Development Lifecycle (SDL), a set of processes that implement security principles and privacy tenets into Intel product development. 

Footnotes

1.  A memory alias is where more than one system address can map to a single physical DRAM address. These can be used to subvert security provided by range registers.
2. Use-after-free weaknesses can occur after incorrect cleanup of pointers to dynamic memory during program operation.

References

1. Intel® 64 and IA-32 Architectures Software Developer's Manual Volume 3C: System Programming Guide, Part 3
2. Intel Security Development Lifecycle

Software Security Guidance Home | Advisory Guidance | Technical Documentation | Best Practices | Resources