Exploiting Tierney's new shared library mechanism

B. Narasimhan
Department of Statistics
Stanford University
Stanford, CA 94305

Draft of 1998/07/01

Table of Contents

Abstract

A very flexible shared library mechanism has been introduced by Tierney in the new release of Lisp-Stat. This is a document in progress that records some of my experiments in getting the best of both the Lisp-Stat and C worlds.

Introduction

[*]

There are times when one wishes to avoid the overhead associated with lisp functions and lisp data types. In some programs I have written, for example, there is a dire need for speed in dynamic graphic computations. Recent releases of Lisp-Stat have introduced many features, among them are the new shared library mechanism and support for C-LONG and C-DOUBLE arrays. In this document, I record some of my experiments in exploiting these features. I must confess there might be pitfalls that I am not aware of. Let me know if you spot any.

Avoiding some overhead with calling C functions

[*]

In [cite tier:1990] and [cite tier:1998a] Tierney describes how call-cfun actually works. To glue the C program and Lisp-Stat together the arguments passed to the C routine are actually coerced into either C long or C double sequences. Then copies of the sequences are passed to the actual routine. In some cases, it might be desirable to get rid of this coercing and copying overhead provided the programmer takes care to ensure that all requirements are met. Such a routine could then be used to great effect when these arrays are stored in data slots in an object. Of course, it is assumed then that such an object will not be garbage-collected.

In my applications, I have need for manipulating large arrays both in C and Lisp-Stat. In order to have both C and Lisp-Stat share the same data, I previously used C programs based on Xlisp internals to manage this. There were some drawbacks to this approach.

  1. The header file xlisp.h had to be included and the poor user had to know where to find it.
  2. There could conceivably have been some flags used in compiling Lisp-Stat that would have to be replicated in compiling the C programs to be safe.
  3. If any of the Lisp-Stat internals changed, the programs might bomb.
  4. Platform specific issues might intrude in a big way.

With the modern mechanism, it appears that all these problems can be elegantly avoided. I present an illustrative example below.

An example

[*]

Consider the following C function which basically prints the contents of an array.

<C Routine FOO>=
#include <stdio.h>

void foo(n, x)
     int *n;
     double *x;
{
  int i;

  for (i = 0; i < *n; i++) {
    printf("x[%d] is %f\n", i, x[i]);
  }
}

* Now suppose I have an array in Lisp-Stat of type C-DOUBLE that I wish to pass to this function without the usual copying overhead associated with call-cfun, how could I do it? In other words, how could I ``share'' the same data in C and Lisp-Stat?

The following modified function of Tierney helps.

<Call by reference oldcfun.lsp>= (U->)
(defun call-by-reference-oldcfun (name lib &rest args)
  "Applies function NAME from shared library handle LIB with arguments.
All compound data are passed by reference but simple ones are not."
   (let* ((fun-addr (shlib::shlib-symaddr lib name))
          (argvecs (mapcar #'lisp-to-arg-if-not-compound-data args))
          (arg-addrs (mapcar #'array-data-address argvecs)))
      (apply #'shlib::call-by-address fun-addr arg-addrs)))
Defines call-by-reference-oldcfun (links are to index).

The helper routine lisp-to-arg-if-not-compound-data is needed.

<Helper routine>= (U->)
(defun lisp-to-arg-if-not-compound-data (x)
   (if (compound-data-p x)
        x
      (if (integerp x)
          (coerce (list x) '(vector c-long))
        (coerce (list x) '(vector c-double)))))
Defines list-to-arg-if-not-compound-data (links are to index).

For extraction here is a package.

<Call by Reference Package>=
(defpackage "CALL-BY-REFERENCE" (:use "XLISP"))
(in-package "CALL-BY-REFERENCE")
<Helper routine>
<Call by reference oldcfun.lsp>
(export 
        '(call-by-reference-oldcfun))

Here then is an example session after creating the shared library libfoo.so.

<Example>=
(require "call-by-reference")
(use-package "CALL-BY-REFERENCE")
(def lib (shlib::shlib-open "./libfoo.so"))
(def n 24)
(def x (make-array '(2 3 4) :initial-contents (iseq 23 0)
                   :element-type 'c-double))
(call-by-reference-oldcfun "foo" lib n x)

List of code chunks

This list is generated automatically. The numeral is that of the first definition of the chunk.

Index

Here is a list of the identifiers used, and where they appear. Underlined entries indicate the place of definition. This index is generated automatically.

References

[1] Luke Tierney. LISP-STAT: An Object-oriented Environment for Statistical Computing and Dynamic Graphics. John Wiley &Sons (New York, Chichester), 1990.

[2] Luke J. Tierney. Shared libraries for xlisp-stat. URL http://www.stat.umn.edu/~luke/xls/projects/, 1998.

*