To avoid deadlock situations, you can use the following approaches:

• Reorder MPI communication calls between processes.

• Implement non-blocking calls.

• Use MPI_Sendrecv or MPI_Sendrecv_replace.

• Use the buffered mode.

The following code section leads to a deadlock in your original application:

...
52      /* waiting for a message that has not been sent */
53      MPI_Recv( &recv, 1, MPI_CHAR, peer, 100, MPI_COMM_WORLD, &status );
54  
55      /*
56       * Too late, this code is not going to be reached.
57       * Beware, simply moving this code up would rely on
58       * buffering in the MPI. The correct solution is to
59       * use MPI_Isend() before the receive and MPI_Wait()
60       * afterwards.
61       */
62      send = 0;
63      MPI_Send( &send, 1, MPI_CHAR, peer, 100, MPI_COMM_WORLD );
...

To resolve the deadlock for the given example, you need to replace the MPI_Recv and MPI_Send calls with MPI_Sendrecv. Do the following:

1. Remove all lines from 52 to 61.

2. Replace the MPI_Recv function call (line 63) with the following MPI_Sendrecv call:

   MPI_Sendrecv( &send, 1, MPI_CHAR, peer, 100, 
   	 &recv, 1, MPI_CHAR, peer, 100, MPI_COMM_WORLD, &status );

3. Save this information into the file: MPI_Sendrecv.c.

4. Compile and run the modified application with the same parameters. The resulting output should look as follows:

     ...
   	[0] INFO: Writing tracefile MPI_Sendrecv.stf in /checking/global/deadlock/hard
   	[0] INFO: Error checking completed without finding any problems.

5. View the newly generated trace file with Intel® Trace Analyzer to make sure that the deadlock issue has been resolved:

   

As shown in the figure above, the deadlock problem no longer occurs, and both ranks successfully exchanged the messages.