OP-TEE is short for Open Portable Trusted Execution Environment, an open source-TEE solution originally based on Arm* TrustZone* technology.
As a TEE solution, OP-TEE is designed as a companion to a non-secure Linux*-flavor operating system to provide isolation from non-secure operating systems. Code and data loaded into OP-TEE can be protected for confidentiality and integrity via platform security technology.
Figure 1: OP-TEE* for Intel® Architecture. Source: Intel
Figure 1 shows the OP-TEE architecture that provides the underlying hardware isolation between the Rich Execution Environment (REE) and OP-TEE. Native client applications in REE use the Global Platform TEE client APIs through TEE driver to access trusted applications (TA) running on the TEE side. TA serves this security-related request then returns to the REE side. The hypervisor, based on Intel® Virtualization Technology (Intel® VT), is responsible for world switch and isolation. There’s also a daemon called tee-supplicant running on the REE side for secure storage, TA-to-TA communication, and other remote security services. It interacts with the storage driver and TEE driver in Linux kernel space for all kinds of remote procedure call (RPC) services like secure storage operations or TA dynamic loading and so on. On TEE side, the OP-TEE OS stack includes the GlobalPlatform TEE Internal APIs along with other native APIs to access kernel resources.
As described above, in addition to the small footprint and good portability, OP-TEE is gaining traction thanks to the GlobalPlatform API support it provides. OP-TEE implements TEE Internal Core API v1.1.x which is the API exposed to Trusted Applications and the TEE Client API v1.0, the API handling communications from a TEE from REE.
To meet customer requirements for deploying OP-TEE on the Intel platform and provide more alternatives, we’ve enabled OP-TEE on the Intel Platform. Figure two below gives an overview; we’ll take a closer look at the evolution and OP-TEE solutions on the Intel platform in the following sections. The OP-TEE IA source code is available on GitHub*.
Figure 2: Roadmap of OP-TEE on Intel Platform. All graphics: Jingdong Lu.
One Virtual Machine, Two Worlds
This is the first solution adopted enabling OP-TEE on the Intel platform. On top of an Intel® Kernel Guard Technology (iKGT) hypervisor, one VM is split into a non-secure world and secure world as shown in Figure 3. iKGT is a lightweight type-1 hypervisor, open sourced by Intel.
Figure 3: A “One VM, Two World” solution for OP-TEE
Memory isolation is achieved by configuring extended page tables appropriately, so Linux can’t access OP-TEE memory. However, there’s one block of shared memory that can be accessed by both Linux and OP-TEE to exchange parameters and data.
World switch is done by the “VMCALL” command, which can be issued from either Linux or OP-TEE. The next step is to issue “VM exit” to the iKGT hypervisor, responsible for the detailed world switching such as VMCSs/GPRs in preparation for saving or restoring VM context, performing “VM entry” to the other world and so on.
Both REE and TEE software environments will maintain API level compatibility with the Arm TrustZone environment. Only the Linux driver of OP-TEE and OP-TEE OS core are architecture dependent.
A Custom Two-VM Solution
This solution is based on ACRN,* a flexible, lightweight reference hypervisor open sourced by Intel. As shown in Figure 4, two VMS run on top of the ACRN hypervisor. VM1 is Linux and VM2 is OP-TEE.
Figure 4: Custom two VM solution for OP-TEE
ACRN creates an isolated memory region for TEE and a communication path for the two VMs to interact. To emulate the behavior of Arm Trusted Firmware, there are some customizations in ACRN hypervisor implementation including CPU scheduling, inter-VM communication, secure interrupt delivery, non-secure interrupt delivery and memory protection based on Intel® VT.
To enhance security, both ACRN and OP-TEE run in an encrypted memory region that protects confidentiality, integrity and replay.
Running OP-TEE as TD guest
OP-TEE can run as standalone VM based on the most popular Quick EMUlators (QEMU*) and Kernel-based Virtual Machines (KVM). This comes in handy for confidential computing, to easily deploy OP-TEE or opt for a more generic solution. However, KVM works inside the Linux kernel, bringing with it a large trusted computing base (TCB) that must also include the host and hypervisor. To reduce the TCB as much as possible, OP-TEE can be run in a secure, standalone VM.
Intel® Trust Domain Extensions (Intel® TDX) introduces new architectural elements to help deploy hardware-isolated virtual machines called Trust Domains (TDs) based on Intel® Total Memory Encryption - Multi-Key (Intel® TME) and a new CPU execution mode called Secure-Arbitration Mode (SEAM). OP-TEE runs as a TD guest, satisfying secure standalone VM requirements. In this scenario, both host and hypervisor can be excluded from the TCB.
Figure 5 shows how OP-TEE runs as TD guest based on QEMU/KVM.
Figure 5: Architecture of OP-TEE running as TD gues
If OP-TEE only needs to run in a normal standalone VM, i.e., in a QEMU process, there are two issues to resolve first. The first is communication between the host and OP-TEE. Compared to the previous “VMCALL” method, there’s no customized “VM exit” handling in generic QEMU/KVM hypervisor so VIRTIO is used instead. On the OP-TEE side, one special VIRTIO device driver is added to receive/send messages from/to host. On the host side, because the optee-driver runs in the Linux kernel, sending and receiving messages is bridged by the vhost-vsock module as shown in Figure 5.
The second issue is shared memory implementation. There’s no special Extended Page Tables (EPT) configuration in KVM to provide memory shared between host and OP-TEE. That means two blocks of memory with the same size are allocated in optee-driver and OP-TEE respectively. Memory copy based on VIRTIO messages syncs between these two blocks of “shared” memory.
Furthermore, to run OP-TEE as TD guest, some TDX feature support code should be added in OP-TEE bootloader and kernel as well. This includes TD launch code based on TDVF, basic TDCALL and TDVMCALL implementation, TDX private and shared memory configuration and so on.
You can try out the proof-of-concept with the OP-TEE binaries on GitHub.
For future scalability, OP-TEE can serve other normal VMs on top of host as well if communication and shared memory issues can be managed.
Conclusion
To date, OP-TEE can run on Intel x86 platforms with the above-mentioned solutions. The official test suite for OP-TEE, Xtest, shows a 100% pass rate for all Intel solutions. The OP-TEE on IA inherits all the features from OP-TEE on Arm, but also benefits from improvements enhanced by Intel technologies like VT and TDX. With these options, OP-TEE can easily be enabled for many TEE solutions on Intel platforms.
Acknowledgements
This post reflects the work of many people in Intel, including Jingdong Lu, Gang Chen, Shirley Zhao, Kai Wang, Bing Zhu, Zhenlong Ji, Chen Chen.
About the Author
Jingdong Lu is a security software engineer at Intel, with deep experience in Trusted Execution Environment (TEE) technology. He’s the architect and a major contributor to the OP-TEE IA open source project.