Lab: MPI

Hello World
Sending and Receiving Messages
- Exercise 1
Broadcast communication
- Exercise 3

This lab is for you to learn a little about MPI, the Message Passing Interface. We'll be using Open MPI, an implementation of MPI that has been made available for a number of languages, including Java. The documentation is strangely not online, but is available locally on any computer that has Open MPI installed. On the lab machines, it is linked here.

Hello World

Here's the code for a simple Hello World program:

import mpi.*;
import java.net.*;

public class HelloWorld {

    public static void main(String[] args) throws MPIException, UnknownHostException {

        MPI.Init(args);

        int size = MPI.COMM_WORLD.getSize();
        int rank = MPI.COMM_WORLD.getRank();
        String name = MPI.COMM_WORLD.getName();
        String hostname = InetAddress.getLocalHost().getHostName();

        System.out.println("Hello, World! I am process " + rank + " of " +
                           size + " on " + name + " on host " + hostname);

        MPI.Finalize();
    }
}

Copy this into a file HelloWorld.java. Compile and run it with the following commands:

javac -cp .:/usr/local/lib/mpi.jar HelloWorld.java
mpirun -n 5 java HelloWorld

You should get hellos from five distinctly numbered processes.

One of the fabulous capabilities of MPI is that in can run parallel jobs on multiple computers easily. Split up HelloWorld so that it runs on your computer as well as the computers next to you.

javac -cp .:/usr/local/lib/mpi.jar HelloWorld.java
mpirun -n 8 -H cmc306-04,cmc306-05,cmc306-06 java HelloWorld

Sending and Receiving Messages

You can send and receive messages using the the send and recv methods of MPI.COMM_WORLD. These methods both take Buffer (of which IntBuffer is a subclass), which is much like an array but is optimized for transfering data. For example, here is a program in which the root process (process 0) sends a message to process 1.

import mpi.*
import java.nio.*;

public class SendRecv {

    public static void main(String[] args) throws MPIException{

        MPI.Init(args);

        int size = MPI.COMM_WORLD.getSize();
        int rank = MPI.COMM_WORLD.getRank();
        String name = MPI.COMM_WORLD.getName();

        if (size < 2) {
            System.out.println("Too few processes!");
            System.exit(1);
        }

        IntBuffer buffer = MPI.newIntBuffer(1);

        int numItemsToTransfer = 1;
        int sourceRank = 0;
        int destinationRank = 1;
        int messageTag = 0;  // supplemental tag, often just not used
        int bufferPosition = 0;

        if (rank == 0) {
            int valueToTransmit = 5;
            buffer.put(bufferPosition, valueToTransmit);
            MPI.COMM_WORLD.send(buffer, numItemsToTransfer, MPI.INT,
                                destinationRank, messageTag);
        } else if (rank == 1) {
            Status status = MPI.COMM_WORLD.recv(buffer, numItemsToTransfer,
                                                MPI.INT, sourceRank, messageTag);
            System.out.println("Process 1 received value " +
                               buffer.get(bufferPosition) + " from process 0");
        }

        MPI.Finalize();
    }
}

Exercise 1

Write a program that sends values in a circle, with each process sending a value to the next highest ranked process except for the highest-ranked process, which sends to process 0). Make sure to avoid deadlock. Whenever each process receives a value, it should print a line of text indicating which process it is, which process it received the value from, and what the value was.

One challenge you may or may not run into in the above exercise is that the results of the output may be interleaved, sometimes out of order. System.out.println is synchronized and thread-safe, but here we have multiple single-threaded processes all writing into the same output buffer, and the timing on how the results get flushed may not happen in the order that you expect. This problem seemed to be much more prevalent on my home Linux machine than on the department Macs, but I've seen it at least occasionally on the lab Macs as well. If you think that the terminal window is showing you the output in an incorrect order, here are some tricks you can do to verify that you are seeing the output in precisely the order it came through:

In your System.out.println statements, insert a call to System.nanoTime() to get a timestamp. Make sure that the timestamp is the first thing to get printed on each line.
Instruct mpirun to output the result of each process to a separate file, so that the results of each process don't get interleaved:
```
mpirun -n 5 -output-filename result java HelloWorld
```
When done, use the command-line program sort to concatenate together the various files in timestamp order:
```
sort result*
```

Broadcast communication

MPI's collective functions allow communications among all processes in a group. Broadcast sends a message to every process. Reduce collects values from all processes and aggregates them.

Here is some sample code that uses broadcast and reduce functionality. Copy the code into a file, and run it. See if you can figure out line-by-line what it's doing. Ask questions if you have any.

import mpi.*;
import java.nio.*;

public class BcastReduce {

    public static void main(String[] args) throws MPIException {

        MPI.Init(args);

        int size = MPI.COMM_WORLD.getSize();
        int rank = MPI.COMM_WORLD.getRank();
        String name = MPI.COMM_WORLD.getName();

        IntBuffer buffer = MPI.newIntBuffer(1);

        int numItemsToTransfer = 1;
        int sourceRank = 0;
        int bufferPosition = 0;
        int numberToPlayWith = 5;

        buffer.put(bufferPosition, numberToPlayWith * 3);

        MPI.COMM_WORLD.bcast(buffer, numItemsToTransfer, MPI.INT, sourceRank);

        int valueReceived = buffer.get(0);

        int valueModified = valueReceived + 4;

        buffer.put(bufferPosition, valueModified);

        IntBuffer finalAnswer = MPI.newIntBuffer(1);
        int initialValue = 0;
        MPI.COMM_WORLD.reduce(buffer, finalAnswer, numItemsToTransfer,
                              MPI.INT, MPI.SUM, initialValue);
        if (rank == 0) {
            System.out.println("Final answer = " + finalAnswer.get(0));
        }

        MPI.Finalize();

    }
}

Exercise 3

A classic example is computing the vaue of pi using numerical integration. Basically, we trying to calculate the area under the curve \(f(x) = 4 / (1 + x^2)\) between \(x = 0\) and \(1\), which happens to be pi. The more intervals we divide it into, the more accurate the answer. The root process broadcasts the number of intervals (n) to all processes, which compute part of the sum, then the reduce function

Here is pseudocode for the algorithm to compute pi; see if you can translate it to MPI! In case you are wondering if your solution is right, the answer I get for \(n = 1000\) is \(3.141593\).

root process only:
    ask the user for a value n
broadcast the value n to all processes
width <- 1 / n
sum = 0
forall 1 <= i <= n (in parallel - divide the i values between processes somehow):
    sum <- sum + 4 / (1 + (width * (i - 0.5)) ** 2)
mypi <- width * sum
sum all processes' mypi using reduce
print sum