The source code for applications (apps) in the test dir are examples of how to use fftMPI from C++, C, Fortran, and Python. The apps that start with the word "simple" are about 150 lines each and are a complete illustration of how to setup a distributed grid and perform an FFT. The apps that start with "test3d" or "test2d" are more complex; they take many optional command-line arguments that exercise all the options and methods that fftMPI provides. An explantion of all the command-line options is on the runtest doc page.
Details for all 4 languages are given here of what any application needs to include to perform 3d FFTs or 3d Remaps. Just change "3" to "2" to perform 2d FFTs or 2d Remaps. You can perform both 2d and 3d FFTs or Remaps from the same app, by including both the 2d and 3d header files and linking to both the 2d and 3d fftMPI library files.
Any file that makes a call to fftMPI needs to include a header file that defines the fftMPI API. These code lines also show how to allocate grids for single or double precision FFTs or Remaps.
IMPORTANT NOTE: As explained on the compile doc page, it is a compile-time choice to build fftMPI to perform either single or double precision FFTs. Your application must allocate its FFT grid data to match the library precision setting. You can look at the test apps to see how they do this in a flexible way so that the app can choose to perform its FFTs in either single or double precision.
In these code examples, fftsize is the number of grid points owned by the processor for an FFT. The "2" is for a pair of (real,imaginary) values stored for each grid point. If a remap is being performed "2" could be a different value, as explained on the remap API doc page. The value of fftsize to use in an app is returned by the setup() method and may include additional memory space for later stages of the FFT.
C++:
#include "fft3d.h" // if performing FFTs #include "remap3d.h" // if performing Remaps using namespace FFTMPI_NS;
work = (float *) malloc(2*fftsize*sizeof(float)); // single precision work = (double *) malloc(2*fftsize*sizeof(double)); // double precision
The header files fft3d.h and remap3d.h define a typedef for FFT_SCALAR which is set to "float" or "double" depending on the precision you build fftMPI with. So you can define work vectors to be of type FFT_SCALAR if you wish.
C:
#include "fft3d_wrap.h" // if performing FFTs #include "remap3d_wrap.h" // if performing Remaps
work = (float *) malloc(2*fftsize*sizeof(float)); // single precision work = (double *) malloc(2*fftsize*sizeof(double)); // double precision
As with C++, you can define work vectors to be of type FFT_SCALAR.
Fortran:
use iso_c_binding ! use these lines in any subroutine/function that calls fftMPI use fft3d_wrap ! if performing FFTs use remap3d_wrap ! if performing Remaps
real(4), allocatable, target :: work(:) ! single precision real(8), allocatable, target :: work(:) ! double precision allocate(work(2*fftsize))
Python:
import numpy as np from mpi4py import MPI from fftmpi import FFT3dMPI # if performing FFTs from fftmpi import Remap3dMPI # if performing Remaps
work = np.zeros(2*fftsize,np.float32) # single precision work = np.zeros(2*fftsize,np.float) # double precision
To use fftMPI from Python, you must have Numpy and mpi4py installed in your Python. This is discussed further below.
These header files listed above for each language are all in the fftMPI src directory. When you compile your app, it must be able to find the appropriate header file. The library files libfft3dmpi.so and libfft2dmpi.so (or *.a equilvalents) are also in the fftMPI src directory, after building the library:
For C++ and C, the compile and link commands can be something like this:
mpicxx -I/home/me/fftmpi/src -c test3d.cpp mpicxx -L/home/me/fftmpi/src test3d.o -lfft3dmpi -o test3d
where the -I and -L switches give the path to the fftMPI src dir.
For Fortran, the fft3d_wrap.f90 and/or remap3d_wrap.f90 files need to be in the directory with your app files. So you can copy it there; the fft*wrap.f90 files are already present in the test dir. The compile and link commands can then be something like this:
mpif90 -I/home/me/fftmpi/src -c fft3d_wrap.f90 mpif90 -I/home/me/fftmpi/src -c test3d_f90.f90 mpif90 -L/home/me/fftmpi/src test3d_f90.o fft3d_wrap.o -lfft3dmpi -lstdc++ -o test3d_f90
where the -I and -L switches give the path to the fftMPI src dir.
For Python, there is no build step. However your Python script needs to be able to find the src/fftmpi.py and library files at run time; see the next section.
Note that if you built fftMPI as a static library, using an external 1d FFT library (FFTW or Intel MKL) then the link lines above also need to include the 1d FFT library. If you built fftMPI as a shared library this is not needed; the dependencies on the 1d FFT libraries were satistified when the fftMPI library was built. See the buildtest doc page for details on how to include these 1d FFT libraries in the link. To use the provided KISS FFT library (just a header file), no additional link arguments are needed.
To use fftMPI from Python, there are 3 things to consider. If any of them are not satisfied you will get a run-time error when you launch your Python script. Each topic is discussed in more detail below:
(1) Numpy and mpi4py must be installed in your Python
The allocation of the FFT grid in your app must be via Numpy vectors or arrays, which are then passed to fftMPI. Numpy is a numeric package already installed in most Pythons.
The mpi4py package is a wrapper on MPI and allows you to run Python scripts in parallel.
You can test for both of these packages as follows:
% python >>> import numpy as np >>> from mpi4py import MPI
If an error results with either import command you will have to install the corresponding package.
For Numpy, go to http://www.numpy.org and follow the directions to download/install Numpy.
For mpi4py, go to http://www.mpi4py.org, unpack it, and install it via pip:
pip install mpi4py
Or if you are using anaconda for your Python package management, you can type the following to download and install either package
conda install numpy conda install mpi4py
After mpi4py imports sucessfully, try running the following test.py script on a few processors, like this mpirun -np 4 python test.py
# test.py script from mpi4py import MPI world = MPI.COMM_WORLD me = world.rank nprocs = world.size print "Me %d Nprocs %d" % (me,nprocs)
You should get 4 lines of output "Me N Nprocs 4", where N = 0,1,2,3.
IMPORTANT NOTE: When mpi4py is installed in your Python, it compiles an MPI library (e.g. inside conda) or uses a pre-existing MPI library it finds on your system. This MUST be the same MPI library that fftMPI and your app are built with. If they do not match, you will typically get run-time MPI errors when your app runs.
You can inspect the path to the MPI library that mpi4py uses like this:
% python >>> import mpi4py >>> mpi4py.get_config()
(2) Your app must be able to find the src/fftmpi.py file
The src/fftmpi.py file is a Python wrapper on the fftMPI C interface. Python loads it when a statement like this is executed:
from fftmpi import FFT3dMPI
If this fails because it cannot find fftmpi.py, you can do one of two things:
(a) Set the PYTHONPATH environment variable to include the fftMPI src dir, like this, either from the command line or in your shell start-up script:
setenv PYTHONPATH $PYTHONPATH:/home/sjplimp/fftmpi/src # csh or tcsh export PYTHONPATH=$PYTHONPATH:/home/sjplimp/fftmpi/src # bash
(b) From your Python app, you can augment the search path directly:
path_fftmpi = "/home/me/fftmpi/src" sys.path.append(path_fftmpi) from fftmpi import FFT3dMPI
(3) Your app must be able to find the fftMPI library file
Python loads the fftMPI library when a statement like this is executed. It looks for the fft3dmpi.so file. Note that you must build fftMPI as a shared library to use it from Python.
fft = FFT3dMPI(world,precision)
If this fails because if cannot find the library file (similar for 2d FFTs, or 3d/2d Remaps), you can do one of two things:
(a) You can add the fftMPI src dir to your LD_LIBRARY_PATH environment variable, e.g.
setenv LD_LIBRARY_PATH $LD_LIBRARY_PATH:/home/me/fftmpi/src # csh or tcsh export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/me/fftmpi/src # bash
(b) You can "install" the fftMPI library file in a location your where your system can find it, such as /usr/local/lib. See the compile doc page for details on installing fftMPI after you build it. This typically requires super-user or sudo priveleges.