Intel® MPI Library Developer Guide for Windows* OS

Download PDF
ID 768730
Date 3/31/2023
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
Intel® MPI Library Developer Guide for Windows* OS
Intel® MPI Library Developer Guide for Windows* OS x
Introduction Installation and Prerequisites Compiling and Linking Running Applications Debugging Applications Analysis and Tuning Troubleshooting Notices and Disclaimers
Introduction x
Introducing Intel® MPI Library Conventions and Symbols Related Information
Installation and Prerequisites x
Installation Prerequisite Steps User Authorization
Compiling and Linking x
Compiling an MPI Program Configuring a Microsoft Visual Studio* Project
Running Applications x
Running Intel® MPI Library in Containers Selecting a Library Configuration Running an MPI Program Running an MPI/OpenMP* Program MPMD Launch Mode Fabrics Control Job Schedulers Support Controlling Process Placement Java* MPI Applications Support
Running Intel® MPI Library in Containers x
Build a Singularity* Container for an MPI Application Run the Application with a Container
Fabrics Control x
Selecting Fabrics Libfabric* Support OFI* Providers Support
Debugging Applications x
Debugging Using -gtool for Debugging
Analysis and Tuning x
Displaying MPI Debug Information Tracing Applications MPI Tuning
Troubleshooting x
Error Message: Bad Termination Case 1 Case 2 Error Message: No such file or Directory Error Message: Permission Denied Error Message: Fatal Error Error Message: Bad File Descriptor Error Message: Too Many Open Files Problem: MPI Application Hangs Problem: Password Required Problem: Cannot Execute Binary File Problem: MPI limitation for Docker*
Intel® MPI Library Developer Guide for Windows* OS

Error Message: Bad Termination

NOTE: The values in the tables below may not reflect the exact node or MPI process where a failure can occur.

Case 1

Error Message

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 1 PID 27494 RUNNING AT node1
=   KILLED BY SIGNAL: 11 (Segmentation fault)
===================================================================================

or: 

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 1 PID 27494 RUNNING AT node1
=   KILLED BY SIGNAL: 8 (Floating point exception)
===================================================================================

Cause

One of MPI processes is terminated by a signal (for example, Segmentation fault or Floating point exception) on the node01.

Solution

Find the reason of the MPI process termination. It can be the out-of-memory issue in case of Segmentation fault or division by zero in case of Floating point exception.

Case 2

Error Message

================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 1 PID 20066 RUNNING AT node01
= KILLED BY SIGNAL: 9 (Killed)
================================================================================

Cause

One of MPI processes is terminated by a signal (for example, SIGTERM or SIGKILL) on the node01 due to:

  • the host reboot;
  • an unexpected signal received;
  • out-of-memory manager (OOM) errors;
  • killing by the process manager (if another process was terminated before the current process);
  • job termination by the Job Scheduler (PBS Pro*,  SLURM*) in case of resources limitation (for example, walltime or cputime limitation).

Solution

  1. Check the system log files.
  2. Try to find the reason of the MPI process termination and fix the issue.
Parent topic: Troubleshooting
Level Two Title