Interaction of LD_PRELOAD and dlopen #

Source code: bit-bcast/linking

Let’s consider add as a simple substitution for a BLAS function; and naturally BLAS is just a substitute any shared library. It’s implemented in the following convoluted manner:

// file: add_impl.cpp
extern "C" {
double lmmr_add_impl(double a, double b) {
  std::cout << LMMR_TAG << ": lmmr_add_impl \n";
  return a + b;
}
}

// file: add.cpp
extern "C" {
double lmmr_add(double a, double b) {
  std::cout << LMMR_TAG << ": lmmr_add \n";
  return lmmr_add_impl(a, b);
}
}

We will call this most useful library lmmr, i.e liblmmr.so.

This will allow us to demonstrate the following behaviour:

• Instrumenting lmmr_add via preloading.
• Instrumenting lmmr_add_impl via preloading.
• Using LD_PRELOAD to use a different implementation of liblmmr.so.
• Dynamically load lmmr_add observe the effect of LD_LIBRARY_PATH and LD_PRELOAD.

We’ll need two versions of liblmmr.so, one in v1/ and another in v2/. Furthermore, we’ll need to partial implementations of LMMR, which we’ll use for preloading only one of the functions.

We’ll also need a program:

int main(int argc, char* argv[]) {
  std::cout << "Regular function call: \n";
  lmmr_add()

  std::cout << "Dynamically loaded function call: \n";
  void* liblmmr = dlopen(argv[1], RTLD_NOW);

  auto dyn_lmmr_add = (double (*)(double, double))(dlsym(liblmmr, "lmmr_add"));

  dyn_lmmr_add(a, b);

  dlclose(liblmmr);

  return 0;
}

Case I: without preloading. #

Let’s demonstrate dynamically loading from v2:

$ LD_LIBRARY_PATH=${PWD}/v1 ./main ${PWD}/v2/liblmmr.so
Regular function call:
v1: lmmr_add(double, double)
v1: lmmr_add_impl(int, int)

Dynamic function call:
v2: lmmr_add(double, double)
v1: lmmr_add_impl(int, int)

Case II: preload lmmr_add #

Now we try pre-loading a function that dynamically loaded:

$ LD_PRELOAD=${PWD}/instrumented/libadd_instrumented.so LD_LIBRARY_PATH=${PWD}/v1 ./main ${PWD}/v2/liblmmr.so
Regular function call:
instrumented: lmmr_add(double, double)
v1: lmmr_add_impl(int, int)

Dynamic function call:
v2: lmmr_add(double, double)
v1: lmmr_add_impl(int, int)

Note, that pre-loading does not interfere with the dlsym in the sense that we’re loading the version inside the library we dlopened. Not the version that was preloaded.

Case III: preload lmmr_add_impl. #

Let’s preload a function that called from a dynamically loaded function:

$ LD_PRELOAD=${PWD}/instrumented/libadd_impl_instrumented.so LD_LIBRARY_PATH=${PWD}/v1 ./main ${PWD}/v2/liblmmr.so
Regular function call:
v1: lmmr_add(double, double)
instrumented: lmmr_add_impl(int, int)

Dynamic function call:
v2: lmmr_add(double, double)
instrumented: lmmr_add_impl(int, int)

Note: The use of LD_PRELOAD has replaced all uses of lmmr_add_impl with the instrumented version. Even if the function calling lmmr_add_impl was dynamically loaded.