"You take a million, billion tonnes of flaming inferno and turn it into 'twinkle, twinkle little star' ..."

Wed, 22 Apr 2009

pkg-config, -as-needed and multiple compilers

At work I've been promoting the use of pkg-config and modules to solve the problems of avoiding hard-coded paths and an environment where we've multiple compilers. In summary, "module load intel-cc" to load the intel compiler, "module load netcdf-intel" to, which among other things appends /ichec/packages/netcdf/4.0-intel/pkgconfig to $PKG_CONFIG_PATH, and then:

NCDF_INCS := `pkg-config netcdf --cflags`
NCDF_LIBS := `pkg-config netcdf --libs`
in the application code. Replace "intel-cc" and "netcdf-intel" with "gcc" and "netcdf-gcc", and it builds with gcc.

This would work better if upstream supplied .pc files, which means the next stage in world domination is to send patches to do just that. But it's not that easy, apparently.

netcdf (for example) supplies two libraries, libnetcdf.so and libnetcff.so, with the second including code only needed for Fortran. So, for gcc I have the following netcdf file:

prefix=/ichec/packages/netcdf/4.0-gcc
exec_prefix=${prefix}
libdir=${prefix}/lib
includedir=${prefix}/include

Name: netcdf
Description: netCDF libraries, include files and development tools (gcc version)
Version: 4.0
Libs: -L${libdir} -Wl,--as-needed -lnetcdf   -lnetcdff 
Cflags: -I${includedir}
While for Intel I have:
prefix=/ichec/packages/netcdf/4.0-intel
# Add fortran libs 
forlibs=/ichec/packages/intel/fce/11.0.081/lib/intel64
exec_prefix=${prefix}
libdir=${prefix}/lib
includedir=${prefix}/include

Name: netcdf
Description: netCDF libraries, include files and development tools
Version: 4.0
Libs: -L${libdir} -lnetcdf -L${forlibs}  -lnetcdff -lifcore
Cflags: -I${includedir}

The main problem is that --as-needed is not understood by non GNU-ld linkers, and must be conditionally removed somehow. Any ideas ? (There is a second wrinkle of needing to add additional libraries for Intel Fortran here, but I'm sure I can remove that with the addition of more .pc files.).

A second issue is that --as-needed can break otherwise working pkg-config usages. Thanks to galtgendo at PhP Bugs for this example:

alastair@ailm:~$ cc -o testx  -Wl,--as-needed `pkg-config glib-2.0 --cflags` test.c `pkg-config glib-2.0 --libs`
works, but:
alastair@ailm:~$ cc -o testx  -Wl,--as-needed `pkg-config glib-2.0 --cflags`  `pkg-config glib-2.0 --libs` test.c
/tmp/ccsUEwtk.o: In function `main':
test.c:(.text+0x20): undefined reference to `g_print'
collect2: ld returned 1 exit status
The problem being that --as-needed removes libraries as unnecessary before the linker sees the test.o code.

To address the first issue, I have two suggestions: - Stop using the proprietary Intel compiler and/or going out of your way to support it when it lacks useful features, or... - Fix the Intel compiler to have --as-needed, or get Intel to do so.
Actually, the order of providing the flags for linking has always been positional, and for some non-GNU linkers it always worked in the "--as-needed way". Putting the libraries before the source or object files is not a valid usage, even though it works under generic conditions.
Denying proprietary compilers is simply not an option: the performance benefit is significant when you run a large supercomputer and are paying for the energy (on the large climate projects, for example, it translates to days of runtime and tens of thousands of euro in energy).

While I may work on Debian to enable free code, and thing free code is a good thing and promote more of it, denying people the right to use proprietary alternatives is not on.

Asking Intel to add --as-needed is a good suggestion but doesn't solve the basic problem of conditional replies in pkg-config: Is --as-needed available on BSD, for example ? What about non-gnu compilers and linkers in Linux ?
pyblosxom bug: Including "--" in the title of an article confuses the comments page, adding &!-- into the page, and losing comments.

So as a workaround the title has changed to "-as-needed" for the moment while I debug.
@Alastair: Have you actually tested GCC versus ICC for the programs you care about?  Lately?  Or do you just assume ICC will win?  It often doesn't; GCC does better and better with each release.
@Anonymous: Daily. We test and install the latest Intel and gcc; using modules allows us to test multiple versions of each compiler. Yes gcc is getting better, but for our applications, Intel normally wins.

Mostly it wins on OpenMP and the fact we use the Intel Math Kernel Libraries on quad-core Harpertowns. But the whole point of wanting pkg-config to be  compiler-agnostic is to allow us to test and use multiple compilers!
AAAAAAAAAAAAAAAAAAAARG!

You do know that --as-needed massively changes the syntax of linking?
There is absolutely no excuse for putting that into a .pc file. Even if you are sure it does not break your library, it might break other libraries an application using this use.

So please, please, please, remove it always and educate anyone putting
it there with a LART tool, preferably the size of the moon.
I'm not quite sure, I understood your problem.. anyway:

Instead of putting those flags into *.pc files, a lib or program that wants them should test for its availability, e.g. at configure time.

For doing this using autotools, I created a m4 file[1] that provides the makro LINK_AS_NEEDED which you can use in your configure.ac/.in file. After having added this makro, it will perform a check and add the --as-needed flag if applicable. In this case, no .pc file includes that flag.

I've got no idea of ICC, but at least it's also working on BSD. Maybe, this is useful to you?

1: http://gnomecc.svn.sourceforge.net/viewvc/gnomecc/trunk/m4/link_as_needed.m4?revision=41
You need -Wl,-( and -Wl,-) there.

Greetings from the BSD world, which largely gets things done
without pkg-config, and fixes it otherwise.

PS: OpenID isn’t working.

Post a Comment

Name: 
Your email address: 
Your website: 
 
Comment: