Introduction
Intel® Optane™ persistent memory (PMem) is a disruptive technology that creates a new tier between memory and storage. Intel Optane™ PMem modules support two modes: Memory Mode for volatile use cases and App Direct Mode that provides byte-addressable persistent storage.
In this portion of the Quickstart Guide to Provisioning Pmem, we describe some methods of provisioning from within Microsoft Windows Server OS, by providing a tutorial for configuring and managing Intel® Optane™ persistent memory (PMem) modules in Memory Mode, App Direct Mode or mixed-mode, using Windows Server PowerShell commands.
This quick start guide (QSG) has three sections:
- Quick Start Guide Part 1: Persistent Memory Provisioning Introduction
- Quick Start Guide Part 2: Provisioning for Intel® Optane™ Persistent Memory on Linux
- Quick Start Guide Part 3: Provisioning for Intel® Optane™ Persistent Memory on Microsoft Windows
Windows Server Support for Intel® Optane™ PMem
This section provides an overview of the software tools for provisioning PMem on Microsoft Server 2019.
Device Manager View
Device Manager provides a list of PMem devices available in the system.
Software: PowerShell cmdlets
Powershell has the following cmdlets for managing PMem.
- Get-PmemDisk
- Get-PmemPhysicalDevice
- New-PmemDisk
- Remove-PmemDisk
- Get-PmemUnusedRegion
Getting Help
For more information on managing PMem on Windows, see Understanding and Deploy PMem
Tutorial: Provisioning Intel Optane Persistent Memory
The tutorial provides instructions for configuring and managing Intel® Optane™ persistent memory (PMem) modules in App Direct Mode using ipmctl and use the PowerShell utility to finish the configuration.
All the commands described in this section are demonstrated on a two-socket system with six terabytes (TB) of configured PMem, and 384 gigabytes (GB) of DRAM installed.
This tutorial considers that an operating mode is configured already by the Original Device Manufacturer (ODM), which is common in new systems received by customers. Typically, we see systems configured in App Direct Mode or Memory Mode.
Note Ipmctl snippets and screenshots below are for ipmctl 2.0
Prerequisites
- Windows Server 2019
- Intel Optane PMem Modules are installed
- The ipmctl utility is used to configure and manage PMem modules.
- To determine if ipmctl is already installed, type ipmctl into the PowerShell window
- If ipmctl is not installed, please download and refer to the instructions in the ipmctl README Tutorial
Step 1: PMem Discovery
This tutorial considers that an operating mode is pre-configured by the system provider. New systems are commonly pre-configured in App Direct Mode or Memory Mode based on the customer requirement.
- ipmctl show -dimm command displays the PMem modules discovered in the system and verifies that software can communicate with them. Among other information, this command outputs each DIMM ID, capacity, health state, and firmware version. The PMem module firmware and BIOS releases are available from the system provider
PS C:\Windows\system32> ipmctl show -dimm DimmID | Capacity | LockState | HealthState | FWVersion =============================================================== 0x0001 | 126.422 GiB | Disabled | Healthy | 01.02.00.5435 0x0011 | 126.422 GiB | Disabled | Healthy | 01.02.00.5435 0x0021 | 126.422 GiB | Disabled | Healthy | 01.02.00.5435 …. 0x1121 | 126.422 GiB | Disabled | Healthy | 01.02.00.5435 - ipmctl show -topology command identifies which memory slots the DDR and PMem modules are installed in
PS C:\Windows\system32> ipmctl show -topology DimmID | MemoryType | Capacity | PhysicalID| DeviceLocator ================================================================================ 0x0001 | Logical Non-Volatile Device | 126.422 GiB | 0x0026 | CPU1_DIMM_A2 0x0011 | Logical Non-Volatile Device | 126.422 GiB | 0x0028 | CPU1_DIMM_B2 0x0021 | Logical Non-Volatile Device | 126.422 GiB | 0x002a | CPU1_DIMM_C2 0x0101 | Logical Non-Volatile Device | 126.422 GiB | 0x002c | CPU1_DIMM_D2 0x0111 | Logical Non-Volatile Device | 126.422 GiB | 0x002e | CPU1_DIMM_E2 0x0121 | Logical Non-Volatile Device | 126.422 GiB | 0x0030 | CPU1_DIMM_F2 0x1001 | Logical Non-Volatile Device | 126.422 GiB | 0x0032 | CPU2_DIMM_A2 0x1011 | Logical Non-Volatile Device | 126.422 GiB | 0x0034 | CPU2_DIMM_B2 0x1021 | Logical Non-Volatile Device | 126.422 GiB | 0x0036 | CPU2_DIMM_C2 0x1101 | Logical Non-Volatile Device | 126.422 GiB | 0x0038 | CPU2_DIMM_D2 0x1111 | Logical Non-Volatile Device | 126.422 GiB | 0x003a | CPU2_DIMM_E2 0x1121 | Logical Non-Volatile Device | 126.422 GiB | 0x003c | CPU2_DIMM_F2 N/A | DDR4 | 32.000 GiB | 0x0025 | CPU1_DIMM_A1 N/A | DDR4 | 32.000 GiB | 0x0027 | CPU1_DIMM_B1 N/A | DDR4 | 32.000 GiB | 0x0029 | CPU1_DIMM_C1 N/A | DDR4 | 32.000 GiB | 0x002b | CPU1_DIMM_D1 N/A | DDR4 | 32.000 GiB | 0x002d | CPU1_DIMM_E1 N/A | DDR4 | 32.000 GiB | 0x002f | CPU1_DIMM_F1 N/A | DDR4 | 32.000 GiB | 0x0031 | CPU2_DIMM_A1 N/A | DDR4 | 32.000 GiB | 0x0033 | CPU2_DIMM_B1 N/A | DDR4 | 32.000 GiB | 0x0035 | CPU2_DIMM_C1 N/A | DDR4 | 32.000 GiB | 0x0037 | CPU2_DIMM_D1 N/A | DDR4 | 32.000 GiB | 0x0039 | CPU2_DIMM_E1 N/A | DDR4 | 32.000 GiB | 0x003b | CPU2_DIMM_F1 - ipmctl show -memoryresources will report information about the current configuration if it has been configured. If no configuration has been created, ipmctl will return a message similar to "One or more PMem modules have invalid PCD data. A platform reboot is recommended to restore valid PCD data, then try again." In the output below, the MemoryCapacity and AppDirectCapacity values are used to determine if the system was configured in Memory mode, App Direct Mode, or mixed mode, here are a few examples:
- If the system is in Memory Mode, you will see something like this (see Volatile row):
PS C:\Windows\system32> ipmctl show -memoryresources MemoryType | DDR | PMemModule | Total ========================================================== Volatile | 189.500 GiB | 1512.000 GiB | 1512.000 GiB AppDirect | - | - | - Cache | 0.000 GiB | - | 0.000 GiB Inaccessible | 2.500 GiB | 5.066 GiB | 7.566 GiB Physical | 192.000 GiB | 1517.066 GiB | 1709.066 GiB - If the system is in AppDirect, you will see something like this (see App Direct Row):
MemoryType | DDR | PMemModule | Total ========================================================== Volatile | 189.500 GiB | 0.000 GiB | 189.500 GiB AppDirect | - | 1512.000 GiB | 1512.000 GiB Cache | 0.000 GiB | - | 0.000 GiB Inaccessible | 2.500 GiB | 5.066 GiB | 7.566 GiB Physical | 192.000 GiB | 1517.066 GiB | 1709.066 GiB - If the system is in Mixed Mode (50/50), you will see something like this (see volatile and AppDirect rows):
MemoryType | DDR | PMemModule | Total ========================================================== Volatile | 189.500 GiB | 758.000 GiB | 758.000 GiB AppDirect | - | 758.000 GiB | 758.000 GiB Cache | 0.000 GiB | - | 0.000 GiB Inaccessible | 2.500 GiB | 5.066 GiB | 7.566 GiB Physical | 192.000 GiB | 1517.066 GiB | 1709.066 GiB
- If the system is in Memory Mode, you will see something like this (see Volatile row):
Step 2: Determine a Configuration Goal
At this point, if your system is configured as you need it, consider the following step as just information. However, if you need to make changes to the configuration, this section explains the basics. If you need to change the platform configuration, such as Memory mode to App Direct Mode, see appendix A on resetting the system.
When setting a goal, we set the Intel Optane PMem modules to be used in one of the three standard modes: Memory Mode, App Direct, and mixed. Once the mode is set, the system needs to be rebooted for the changes to go into effect.
The first thing you need to do is decide which of the three modes you require and run the appropriate configuration command
- Memory Mode - 100% of PMem capacity across sockets can be provisioned in Memory mode, as described below. In this example, 100% of the available PMem capacity is to be provisioned in Memory mode. You can always use the -f option to overwrite any existing goal. The system must comply with recommended DRAM to PMem capacity ratios. At a minimum, a system must have at least a 1:4 ratio and no more than 1:16 per CPU socket. For example, a system with 192GB of PMem should have at least 768GB of PMem. A warning may be reported by ipmctl and the BIOS at boot time if the configuration does not meet the requirements.
- App Direct Mode - PMem can be provisioned in App Direct Mode with interleaving enabled or disabled. As described in the QSG Guide Part 1, interleaving increases the throughput of reads and writes to PMem.
- Mixed Modes - A percentage of PMem capacity can be provisioned in Memory mode, as described below. Any remaining PMem capacity can either be reserved or used to create an AppDirect region.
Step 3: Configure your Platform
At this point, we have three options, choose the sub-tutorial below to configure your system.
Memory Mode Example
In this example, 100% of the available PMem capacity is to be provisioned in Memory Mode.
- Enter the following on the command line ipmctl create -goal memorymode=100
PS C:\Windows\system32> ipmctl create -goal memorymode=100 The following configuration will be applied: SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size ================================================================== 0x0000 | 0x0001 | 502.0 GiB | 0.0 GiB | 0.0 GiB 0x0000 | 0x0011 | 502.0 GiB | 0.0 GiB | 0.0 GiB 0x0000 | 0x0021 | 502.0 GiB | 0.0 GiB | 0.0 GiB ... 0x0001 | 0x1111 | 502.0 GiB | 0.0 GiB | 0.0 GiB 0x0001 | 0x1121 | 502.0 GiB | 0.0 GiB | 0.0 GiB - A reboot is required to process new memory allocation goals.
App Direct Example
In this example, we configure the system into App Direct Mode.
- Enter the following on the command line ipmctl create -goal persistentmemorytype=appdirect
PS C:\Windows\system32> ipmctl create -goal persistentmemorytype=appdirect The following configuration will be applied: SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size ================================================================== 0x0000 | 0x0011 | 0.0 GiB | 502.0 GiB | 0.0 GiB 0x0000 | 0x0021 | 0.0 GiB | 502.0 GiB | 0.0 GiB 0x0000 | 0x0001 | 0.0 GiB | 502.0 GiB | 0.0 GiB ... 0x0001 | 0x1121 | 0.0 GiB | 502.0 GiB | 0.0 GiB 0x0001 | 0x1101 | 0.0 GiB | 502.0 GiB | 0.0 GiB Do you want to continue? [y/n] y Created following region configuration goal SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size ================================================================== 0x0000 | 0x0011 | 0.0 GiB | 502.0 GiB | 0.0 GiB 0x0000 | 0x0021 | 0.0 GiB | 502.0 GiB | 0.0 GiB 0x0000 | 0x0001 | 0.0 GiB | 502.0 GiB | 0.0 GiB ... 0x0001 | 0x1121 | 0.0 GiB | 502.0 GiB | 0.0 GiB 0x0001 | 0x1101 | 0.0 GiB | 502.0 GiB | 0.0 GiB - A reboot is required to process new memory allocation goals.
Mixed Mode Example
In this example, 60% of the available PMem capacity is to be provisioned in Memory Mode.
- Enter the following on the command line ipmctl create -goal memorymode=60
PS C:\Windows\system32> ipmctl create -goal Memorymode=60 SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size ================================================================== 0x0000 | 0x0011 | 310.0 GiB | 192.0 GiB | 0.0 GiB 0x0000 | 0x0021 | 310.0 GiB | 192.0 GiB | 0.0 GiB 0x0000 | 0x0001 | 310.0 GiB | 192.0 GiB | 0.0 GiB ... 0x0001 | 0x1121 | 310.0 GiB | 192.0 GiB | 0.0 GiB 0x0001 | 0x1101 | 310.0 GiB | 192.0 GiB | 0.0 GiB Do you want to continue? [y/n] y Created following region configuration goal SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size ================================================================== 0x0000 | 0x0011 | 310.0 GiB | 192.0 GiB | 0.0 GiB 0x0000 | 0x0021 | 310.0 GiB | 192.0 GiB | 0.0 GiB 0x0000 | 0x0001 | 310.0 GiB | 192.0 GiB | 0.0 GiB ... 0x0001 | 0x1121 | 310.0 GiB | 192.0 GiB | 0.0 GiB 0x0001 | 0x1101 | 310.0 GiB | 192.0 GiB | 0.0 GiB - A reboot is required to process new memory allocation goals.
Step 4: Provisioning of Intel Optane Persistent Memory Namespaces
This step of the tutorial is for continuing the provisioning process of creating PMem disks (equivalent to namespaces in Linux) for App Direct Mode and mixed-mode.
Now we need to create PMem disks to be used by an application(s). The disk size can be any % of the overall GiB capacity of the installed PMem modules. Please run the following commands to configure the system.
- Open Powershell as Administrator and enter the following commands
- Get-PmemUnusedRegion
- This command Is used to list the regions for PowerShell. If the output of this command is empty, no regions exist on the PMem modules. Recall that ipmctl is used to create regions.
- After creating regions using ipmctl and then rebooting, the newly created interleave sets are represented as PMem unused regions. These are the regions that are not assigned to a logical persistent memory device on the system
PS C:\Windows\system32> Get-PmemUnusedRegion RegionId TotalSizeInBytes DeviceId ---------- ----------------- ------------ 1 2156073582592 {1, 101, 11, 111}
- New-PmemDisk
- This command allows Windows PowerShell to create a PMem Disk to be created on a region, and the capacity to be visible to the Windows operating system and used by applications. Just as an SSD can be carved into partitions, PMem disks represent the unit of storage that appears as a device that can be used for I/O
- FSDax is the mode used by default
- Creating a new persistent memory disk. This operation may take a few moments
- Initialize each newly created PMem disk by going to the Computer Management > Storage > Disk Management console to view each new disk and initialize it.
- Initialize the disk using MBR or GPT partitioning before the logical disk manager can access it.
-
- Format the disk using NTFS (right-click on disk > Format…)
- Get-pmemdisk
- Verifies that PMemDisks was created and initialized
PS C:\Windows\system32> Get-pmemdisk DiskNumber Size HealthStatus AtomicityType CanBeRemoved PhysicalDeviceIds UnsafeShutdownCount ---------- ---- ------------ ------------- ------------ ----------------- ------------------- 1 2008 GB Healthy None True {1, 101, 11, 111} 2 PS C:\Windows\system32> Get-physicaldisk | where Mediatype -Eq SCM Number FriendlyName SerialNumber MediaType CanPool OperationalStatus HealthStatus ------ ------------ ------------ --------- ------- ----------------- ------------ 1 Persistent memory disk 03018089022aedcb17c3fe499155db38a408de05 SCM True OK Healthy
- Verifies that PMemDisks was created and initialized
- Get-PmemUnusedRegion
We have come to the end of the tutorial for provisioning PMEM within a Windows Server 2019 environment. See below for a link to other articles in this series.
- Quick Start Guide Part 1: Persistent Memory Provisioning Introduction
- Quick Start Guide Part 2: Provisioning for Intel® Optane™ Persistent Memory on Linux
Resources:
- Google PMem Group
- ipmctl user guide
- NDCTL User Guide
- pmem.io
- PMem Getting Started Guide
- #pmem Slack Channel – Join Here
Appendix
Appendix A: Misc commands
PS C:\Windows\system32> get-pmemdisk | Remove-PmemDisk
- This command will remove the persistent memory disk(s) from the system and results in data loss.