Search
"You take a million, billion tonnes of flaming inferno and turn it into 'twinkle, twinkle little star' ..."

Sun, 21 Jun 2009

Debian Meteorology : Status, Summer Solstice 2009.

Unfortunately I won't be able to make it to DebConf9, so as an aid to those who are going, here's a summary of current work:

I've just uploaded terralib 3.3.1 to Debian, and its sitting in the NEW queue, as the older version was removed from the archive due to lack of maintenance (it had an RC bug in both including its own copy of libtiff and failing to link against it - now it links against an external copy of libgeotiff and libtiff).

In the NEW queue it joins g2clib, hdf-eos4, hdf-eos5 and udunits. These are there as dependencies of other Meteorology-related packages I'm working on: magics++ needs terralib and gshhs; zygrib needs gshhs (it has a copy built-in). NCL (NCAR Command Language) has a rake of dependencies including udunits, hdf-eos, g2clib and vis5d+ (ITP'd) . I'm also packaging VISIT for visualization.

Then there is the GSHHS issue: I think I'll end up packaging 'gmt-coastline-high', but the format of the coastline maps needs to be decided (netCDF or its own binary format) and updating the sundry packages to read the latest version needs tackling.

I'm packaging these as they are used at ICHEC and i've experience building them. One of the main aims I had in setting up Debian Meteorology (beyond adding the software to Debian) was to help integrate all the Free and Open Source code in the Earth-sciences field, and sort out dependency and build issues. I hadn't expected to encounter quite so many so quickly, though. I don't expect to get more done before vacation-time, but I'll be happy if I get these done this summer.

Thu, 18 Jun 2009

Maps and Coastlines in Debian

As mentioned before, I've started working on Debian Meteorology, adding "standard" meteorology-related packages to Debian. Part of the aim of this is to jump-start an effort of integrating the FLOSS in the field: all the usual libraries that people working in the field use and expect to be on the supercomputers and workstations they use.

So, two packages I've been working on are Magics++ and zyGrib, which are plotting and visualisaton tools. respectively. So they both contain coastline maps of the world. Digging deeper shows they use the same files : a binary database called 'GSHHS', or Global Self-consistent Hierarchical High-resolution Shorelines. Some scope for integration here.

So, I start investigating GSHHS in order to create a 'coastline data' package to be shared. It turns out that building GSHHS depends on GMT, the Generic Mapping Tools, already present in Debian, and this coastline issue has been explored before, and a package gmt-coast-low created.

"gmt-coast-low" is 5.5 MB in size, and as its name suggests, there was once a "gmt-coast-high", but this has since been dropped for taking up too much space in the Debian archive (in its place, a script which will download this data for you has been created. But the files in gmt-coastline-low are in netCDF rather than GSHHS's own binary format; what to do? Posting a mail for help and it turns out that another package is being considered, Basemap, an add-on for Mathplotlib, that also includes the GSHHS data.

I've summarized the files, sizes and versions here in the Debian Wiki. Offhand it appears that there is scope for re-adding a gmt-coastline-high package (with perhaps additional small datafiles on states boundaries, etc. seen in Basemap), though some questions remain:

  • Is 170 MB of arch-independent data too much these days in the Debian archive, especially since it appears at least 4 packages can use it ?
  • It seems that some packages would need to be patched to bring them up to date with the latest format version for the database. What format should the data be in, this special binary format (quite simple) or netCDF ?

Tue, 07 Apr 2009

Debian Meteorology

I've added a Debian Meteorology section to the DebianScience page on Debian Wiki. The aim is to add the free and open-source meteorology packages I currently maintain and work on at ICHEC to Debian.

So far I've packaged CDO (Climate Data Operators), and am working on EMOSLIB, an interpolation library. Enrico Zini is packaging GRIB API. Other interesting packages include the OASIS coupler and the VISIT visualization software, and adding support for meteorology data formats to /etc/magic, with desktop icons and mime filetypes, etc.

Tasks to investigate include (1) What other software are people interested in, and (2) getting added to Debian pure blends in Alioth.

Thu, 28 Jun 2007

Fun Filenames

Ok, some interesting output from a build script of a nameless code I'm working on: 

...
scanning lib[10].a ...
scanning lib[11].a ...
scanning lib[12].a ...
scanning lib[13].a ...
scanning lib[14].a ...
scanning lib[15].a ...

pgf90 -Wl,--start-group ./master.o -L. -l[1] -l[2] -l[3] -l[4] -l[5] -l[6] -l[7] -l[8] -l[9] -l[10] -l[11] -l[12] -l[13] -l[14] -l[15] -Wl,--end-group -llapack -lblas
./lib[2].a(cprep1.o)(.text+0x181d): In function `cprep1':
...

Next time, think of someone trying to find these interesting files. People using regexps, ok ? What do these 'libraries' contain, anyway ?

Thu, 24 May 2007

Comment-free code

Often looking through old code you find yourself wishing for more comments, or even more correct comments. Rarely, less comments.

But looking at some code I'm debugging at the moment at work I found the following bizarre idiom (details changed to protect the afflicted) ... 

#ifdef DOC
C  Some fortran comments.
C  Yes, ordinary comments, nothing fancy like preprocessor directives, but if you want
C  your code without a paragraph explaining what it does, you can just preprocess it
C  and remove it.
C 
C Perhaps they'd expected that their fellow developers would write essays in the
C source code that would make it hard to follow, but then, you'd still need to edit
C the full source code to debug and develop it, right?
C
#endif
Preprocessor abuse : chapter 2 of lots.

Those who program C are used to seeing function / subroutine definitions in header files. Not so in Fortran, there people use the C pre-processor to put Header files in subroutine definitions.

Don't try this at home, folks.

Again, names changed to protect the afflicted ... 

subroutine  funmame( param1,
	      param2, param3,
#ifdef FOO32
	     foo, bar,
#endif
	      param7, param8, 
#include "basarg.h"
	     )

Remarkably, basarg.h included: 

	C Comment.
	param5, param6,
#ifdef SOMETHING
        param10, param20,
#endif

There were in fact dozens of parameters in basarg.h.. And even a higher.h that included basarg.h but had more parameters ...

Fri, 23 Mar 2007

Preparing for Daylight Savings Time

While computers mostly handle the Daylight Savings Time changeover automatically, sometimes work is required. In our case at ICHEC, arranging for some jobs to run at set times, means changing the scheduler at the changeover.

We use PBS and Maui to schedule what jobs run when on our cluster. Mostly jobs get run according to priorities: the scheduler (software) decides which what the priority jobs are and when; it collects free nodes until it has sufficient free CPUs to run the jobs at hand. It can do backfilling: when it's e.g. trying to run a large 128 node job but has only 10 nodes free, it will reserve nodes as they become free until it has all the resources needed to run the job. If by looking at the remaining time left on running jobs it realises that it won't get nodes free for the large job for say 2 hours, it can run small jobs with runtimes of less than two hours on the nodes it had reserved. Clever.

The fun issue comes when you need to set reservations: say you need to run a job at 2am every day on 32 nodes, as we do: the scheduler needs to plan accordingly. The scheduler needs to really have 32 nodes free at that time.

Thats 2am local time, of course: but next Sunday, at 2am, the clock changes...

So, working out the consequences, and preparing for the change, I sympathise with our American friends who've moved three weeks early to save energy. Thus causing lots of midnight oil to be burned around the world by sysadmins and programmers updating software. Hmmm.

Note to Self:

When Posting a blog entry asking for help on getting blog comments to work with openid, try leaving the comments setup on the blog working, so people can leave replies. Fixed now.