MPJ Express Release 0.42

Dear MPJ Express Community,

We are pleased to announce the next beta release (0.42) of the MPJ Express software. This release mainly introduces updated runtime system for MPJ Express to support parallel job execution by multiple users. This version of MPJ Express supports running of parallel Java application on non-shared file systems. Parallel launching of daemons have also been introduced in this release with new scripts for daemons.

CHANGELOG
1. This release mainly introduces updated runtime system for MPJ Express to support parallel job execution by multiple users. Following are major changes to support parallel job execution in MPJ Express.
     a. Addition of multi-threaded Job launcher in MPJ Daemon. Whenever a process request posted to MPJ Daemon it launches a new thread to execute it.
     b. Java NIO Sockets are replaced with Java I/O Sockets in both MPJ Daemon and MPJRun code.
     c. Separate channels are introduced for I/O messages delivery to each job executer.
     d. Optimized mechanism for port allocation to processes for inter-process communication while distributing job on cluster. Read/Write ports are now assigned dynamically.
     e. Dynamically allocated ports are now written in mpjdev.conf file and NIODevice reads both ports from mpjdev.conf
     f. XML based process ticket has been introduced to send processes information between MPJRun and MPJDaemon.

2. This release also introduces support for using MPJ Express on non-shared file system.

3. In this release dependency of "Java Service Wrapper Project" has been removed and a new mechanism to launch daemons has been introduced. To support this new mechanism, some useful scripts have also been added in MPJ Express such as mpjstatus, mpjclean and mpjinfo.

You may download the new version of MPJ Express from

http://mpjexpress.org/


Feel free to contact us in case of any queries.

Best Regards,

-MPJ Express Team

MPJ Express support added in TAU

Dear MPJ Express Community,

MPJ Express support has been added in Tuning and Analysis Utility (TAU). MPJ Express users can now perform Profiling and Tracing of MPJ Express applications using TAU. MPJ Express Profiler has been released with latest version (0.41) of MPJ Express. TAU version 2.23.1b or higher is required by MPJ Express for Profiling and Tracing.

You may download MPJ Express from http://www.mpjexpress.org 
and TAU from http://www.cs.uoregon.edu/research/tau/tau_releases

Best Regards,

-Team MPJ Express

MPJ Express Release 0.41

Dear MPJ Express Community,

We are pleased to announce the next beta release (0.41) of the MPJ Express software. This release introduces a new communication device called native device.The goal of the native communication device is to interface the MPJ Express software with native MPI libraries (MPICH, Open MPI or MS-MPI). In this setting the bulk of messaging logic is offloaded to the underlying MPI library. This is attractive because MPJ Express can exploit latest features, like support for new interconnects and e cient collective communication algorithms of the native MPI library.
This release also introduces support for MPJ Express Profiler. The Profiler generates profiles and traces of MPJ Express applications.

CHANGELOG
1. This release mainly introduces support for using a Native MPI Implementation from MPJ Express. We call this native device.
2. This release also introduces support for MPJ Express Profiler. The Profiler generates profiles and traces of MPJ Express applications. Currently the Profiler does not support native device.
3. Major changes: $MPJ_HOME/src/mpjdev is now abstract and provides two implementations i) javampjdev and ii) natmpjdev. Intracomm is now a wrapper class that either uses PureIntracomm (for pure Java devices) or NativeIntracomm (for native decive).
4. libnatmpjdev.so is a C JNI wrapper library to call native MPI library. The source code is placed in $MPJ_HOME/src/mpjdev/natmpjdev/lib
5. Some bug fixes in runtime.

- MPJ Express Team

MPJ Express Release 0.40

Dear MPJ Express Community,

We are pleased to announce the next beta release (0.40) of the MPJ Express software. This release adds support for Eclipse based MPJ Express Debugger plugin.

CHANGELOG

1. This release mainly introduces debugging support for MPJ Express Applications using MPJ Express Debugger. MPJ Express Debugger is an Eclipse based plugin. Current version (1.0.0) of MPJ Express Debugger supports Multicore Mode, Cluster Mode and Hybrid Mode. It is developed by Rizwan Hanif, Amjad Aziz and Aleem Akhtar.


You may download the software and plugin from:

http://mpjexpress.org/

Feel free to contact us in case of any queries.


Best Regards,

Aamir Shafi
Mohsan Jameel
Ansar Javed
Bibrak Qamar
Aleem Akhtar

MPJ Express Release 0.39

Dear MPJ Express Community,

We are pleased to announce the next beta release (0.39) of the MPJ Express software. This release adds support for hybrid parallelism on cluster of multicore machines.

CHANGELOG

1. This release mainly introduces a new MPJ Express device 'hybdev' that supports hybrid parallelism on cluster of multicore machines. The device, hybdev, transparently achieves hybrid parallelism. It does not require code rewriting effort for new and existing MPJ Express applications. It is developed by Ansar Javed and Mohsan Jameel.

You may download the software from:

http://mpjexpress.org/

Feel free to contact us in case of any queries.

Best Regards,

Aamir Shafi
Mohsan Jameel
Ansar Javed
Bibrak Qamar
Aleem Akhtar

Installing, configuring and using MPJ Express on Linux

This tutorial is aimed at installing MPJ Express software and configuring it on Linux machines. It covers steps and processes required to make MPJ Express up and running. We assume that java 1.6 or higher is installed on your system or cluster, you want to run MPJ Express. Additionally, if you want to compile source code of MPJ Express Apache ant 1.6.2 or higher should be installed along with Perl. (Note that these are optional and may not require if you don’t want to recompile MPJ Express).
As a first step download latest version of MPJ Express (current latest is 0.38). Once downloaded extract it in the directory (assume you have downloaded and extracted it under your home directory i.e export/home/mpj/mpj-v_038). Now you need to set the environment variables from MPJ Express. You can add these environment variables in .bashrc (note: If you are using Ubuntu OS then you need to define these variables at the start of the file,  else it will not be set properly and will cause problems)

Once the environment variables are set you can proceed further with writing a Hello World program in HelloWorld.java

Then you need to compile it. If you look at the source code the first statement is “import mpi.*;”, this statement is responsible for including all the MPI related functionality implemented by MPJ Express. With the import statement we can now initialize MPI environment using  “MPI.Init(args);” and can terminate using “MPI.Finalize();”. We can also use other methods like “MPI.COMM_WORLD.Rank();” which return the current rank of process and “MPI.COMM_WORLD.Size();” which return the total size of world. Now we need to compile this MPI code written using MPJ Express.

$javac -cp .:$MPJ_HOME/lib/mpj.jar HelloWorld.java

The –cp (the java classpath options) is used to specify the MPJ Express jar file which contain all the implemented libraries that will help in compiling the java source file. This should create a HellowWorld.class file if the compilation process succeeds.

So far we were able to set environment and compile java code with MPJ Express. Now we have to run the compile code with MPJ Express. In the current version of MPJ Express, it supports two types of configuration i.e. multicore mode and cluster mode.

Running MPJ Express in multicore mode

In multicore mode MPJ Express executes or run processes on same machine, having multiple cores or SMPs. In this mode no network communication is involved.  The communication is performed through shared memory between processes. Running MPJ Express in multicore mode is much simpler than other configurations. To run MPJ Express job in multicore mode execute the following command.

$ mpjrun.sh -np 4 HelloWorld

“mpjrun.sh” is the command used to run MPJ Express jobs. It accepts different parameters. –np is used to specify the number of MPI (MPJ Express) processes to start. –np 4 means that to run HelloWorld program on 4 processes. In the above command we did not specified device (or configuration mode), the default is multicore mode. Following is the output of the above command.

Running MPJ Express in cluster mode

The second type of configuration or mode is cluster mode. The cluster mode uses network communication to do message passing. Currently there are two types of networks supported by MPJ Express i.e. niodev that uses java nio package to communicate and mxdev that runs on myrinet network. For this tutorial we will only use niodev. As we are running MPJ Express in cluster mode, which involve multiple node connected through network thus we need to start MPJ Daemon on the target machines. First create a file name "machines" and list down name of all the nodes (either IP address or Hostname). A sample machines file looks like

compute-0-1

compute-0-2

Now with help of mpjboot command we will start MPJ Daemon on target machines. For mpjboot to work properly, first enable ssh passwordless authentication, as mpjboot uses ssh to start MPJ Daemon on remote machines listed in machines file (Note we assume that cluster is using shared filesystem). 

The message above states that MPJ Daemon processes are started. But if you want to verify if daemon are running on the target machines you can do ssh onto the target node and with help of “ps aux” command you can verify if daemon has been started. If daemon are not running you should enable debugging and logging, which are saved in wrapper.log file. This can be done by following user guide Section 3. (you can contact MPJ Express mailing https://lists.sourceforge.net/lists/listinfo/mpjexpress-users list for your queries).    If MPJ Daemon is running properly than we can proceed forward and run MPJ Express job. For that we will use

$ mpjrun.sh -np 2 -dev niodev HelloWorld

mpjrun.sh –np option tells how many processes to launch, –dev switch specifies device to use, here we will use niodev, another device is mxdev (for myrinet network). For cluster mode mpjrun.sh also expect machines file to be placed in same directory where we are running this command. The output for the HelloWorld program in cluster mode is as follow.

MPJ Express (0.38) is started in the cluster configuration

Starting process <0> on <compute-0-1>

Starting process <1> on <compute-0-2>

Hi from <0>

Hi from <1>

Stopping process <0> on <compute-0-1>

Stopping process <1> on <compute-0-2>

If we want to shop MPJ Daemon we use

$ mpjhalt machines

This will stop the MPJ Daemon running on host listed in "machines" file.

   

MPJ Express Mailing List on Source Forge

We have moved MPJ Express mailing list to sourceforge now - the new mailing list is mpjexpress-users@lists.sourceforge.net.

Join the mailing list by visiting https://lists.sourceforge.net/lists/listinfo/mpjexpress-users.