Greetings Babelers!

Today for a variety of technical resasons, I've got to install a SVN snapshot of Babel on a bunch of different machines. Our existing toolstack on many of these machines is out of date, so I'm essentially going to start from scratch. For those of you who've tried to build Babel and wonder how we developers do it, I figured I'd write up this wiki as a kind of lab notes for the experience.

NOTE: You'll see that I'm installing a significant amount of build tools. This is because I'm toying with SVN checkouts. Most users download releases and never modify the makefiles or configure scripts in Babel, and can blissfully ignore some of the steps I'm doing. I'll use this glyph to signify things that are only build tool related and may not be a concern for the general audience.

Software

Here's a list of all the software I'm going to use today. (actually the next few days, since I've got mucho meetings interrupting the technical work.)

• UNIX Shell : (assumed)
• GNU Make : (assumed)
• C/C++/Fortran77/Fortran90 Compilers : (assumed) xlC on AIX, Intel 9.1 on Linux
• Java : (assumed) Sun's JDK is a binary distro on most *NIX systems and generally compiled with GCC. We have alternate distros available on our machines that are compiled with Intel and IBM compilers.
• XML Parser : (assumed, sort of). Traditionally, Babel has relied on libxml2 being present. There are a few technical problems that have arizen from this and at the time of writing we are transitioning to another, smaller XML parser that will be distributed with future Babel releases. For my purposes, I know libxml2 is present and will suffice.
• Chasm : Chasm -1.4rc2 Needed for Fortran 90 support.
• Python : Python-2.5 I'm installing my own python because I want Babel to support Python on both client and server side. Server-side python support requires a libpython.so which is often not present in default installations. A second reason to build your own Python may be that the default Python was compiled with an uncompatible compiler!
• NumPy : NumPy-1.0.1 Required for Python support. This is essentially an HPC array library that Babel arrays map to.
• Perl : (assumed) This is needed by automake. I've never had problems with system defaults here.
• M4 : m4-1.4.8. This is the traditional UNIX macro processor. In the past, we've overflowed their buffers and had to rely on patched versions. So I have the habit of avoiding the default one that's almost always in /usr/bin.
• Autoconf : autoconf-2.61 This begins the long chain of what folks collectively know of as autotools. Autoconf is the oldest of the three and depends on M4.
• Automake : automake-1.10
• Libtool : libtool-1.5.22

Systems

Since I've got a filesystem that's crossmounted on these machines, I'm going to do a lot of VPATH builds, but you can set up your system however you like. The three main systems I'm going to handle is

1. Power5 w/ Federation
2. Linux-x86 w/ elan3
3. Linux-Opteron w/ InfiniBand

Notice that I'm including the parallel fabric when describing these machines. I'm installing this stuff for a customer that also uses babel-config and babel-libtool to simplify a lot of its build issues. Even though Babel doesn't need MPI itself, it turns out that our use of libtool interferes with a lot of MPI-isms... so we can reverse engineer what the MPI compilation scripts do and integrate that in our build easier than trying to trick libtool to use the MPI scritps. I still have a lot to add in the BUG about build/configure strategies. There's a lot of arcana that I need to write down for posterity.

Setup

So here's my high-level directory structure:


The rest of the instructions will be in terms of these environment variables, so if you want to install along side with me, you'll want to set these to something appropriate for your system. Now I'm going to download and unpack all this code in my ${BABEL_SRCS} directory

Picking Your Compiler

Now we have to commit to a suite of compilers and compile-time flags. For my $X86 machine, I'll use the Intel 9.1 compilers for everything except Fortran 77, which I'll use g77... just for fun!

Staging the build

My instincts say to do VPATH builds for all the remaining codes. Unfortunately, they do not all follow GNU coding guidelines. So I'll simply build them in the code tree, install, and wipe the directory and create a fresh one from the tarball when I go to the next platform.

Chasm

Installation is relatively straightforward. The need to explicitly state the F90 compiler vendor is necessary. Also Chasm uses the more obvious (but deprecated by autotools) F90 variable instead of FC.


I did find one bug with Chasm while installing it. But its a cosmetic one that I logged in Sourceforge.

Python

This is fairly straightforward. The --enable-shared flag is often needed ot generate a libpython.so which is critical for server-side Python.


Python actually has three configure scripts that, combined, weighs in at 30K lines of Bourne Shell. Babel has only two configure scripts, but they are 47K and 46K lines of Bourne Shell each.

NumPy

This uses Python's build system. So if you just installed Python in the last step, here's a little self-test to make sure things are going well.


Now you're ready to install NumPy.



If you are using a pre-existing Python that is installed somewhere you cannot write to, then you'll have to specify an alternate location to install NumPy and set up your environment so that Python will look at the alternate location for resources.

If you'd like to verify all is well, try this.

If you don't get an ImportError, you're in business.



Note the glyph in the title? It means that most users can skip this entire section. Folks who need this information already know who they are.

The tricky part about the build tools is installing them in proper order as each depends on the other. The order is always M4, Autoconf, Automake, Libtool. The fastest way to do this is put the install directory in your search path and rehash after every part is intalled.

I generally install build tools in a special directory away from other things. Most people who use Babel don't need to use our particular version of autotools, only the few folks who are developing in the Babel tree do. Futhermore, we can have multiple Babel installs per machine (different compiler suites, etc.) and these tools clearly are immune from these variations.



In my setup, I'll have several versions of Babel built against all the resources we've
already installed. So I'll make a new subdirectory under $PREFIX for Babel. Plus,
I'll do a VPATH build to not muddy up the SVN checkout


There are a couple of one-time environment settings that need to be made. I need to specify custom JAVA settings, for example. This machine has a commercial JVM that was compiled with Intel compilers, so your JNI_INCLUDES and JNI_LDFLAGS will probably be different. One-time settings such as these can be done simply as arguments to the configure script. The -C flag accellerates the configuration process somewhat by pushing the two configure scripts to share a cache file. Since this is a two processor node, I'll use -j4 for make. This creates four-way parallelism in the build (rule of thumb is two threads per processor is the sweet-spot).