objitsu.com

Stranger In A Strange Land, Welcome to FFI and LISP!

I have managed to blunder my way around the ASDF system sufficiently well enough to do all the things that I have wanted to do… so far. I have been using LISP more and more, on and off, for about eighteen months now and the more I suffer the more I like it. It *absolutely* does make you think differently and *absolutely* does make you realise Philip Greenspun was bang on the mark. This could be a long blog post but interesting to me anyway…. LOL.

Having used the cl-opengl bindings for hacking in the past, I decided I wanted to have a go at the cl-glfw bindings instead as I do not like the way the event loop forces you to structure your application and also, having downloaded and built the glfw library and run the examples it seemed like it would be a good exercise to convert them into LISP as the examples worked well on my 10.5.8 intel Mac, in both ‘-x11′ and Cocoa builds of the code. There is a problem with the keyboard handling in Cocoa mode, which oddly enough is listed as a bug on the cl-glfw site with the status ‘closed’ and yet it still broke on my Mac. The X11 build was just fine.

So, firing up EMACS + SLIME I duly attempted to install the cl-glfw binding only to be met by this error in the debugger:

Unable to load any of the alternatives:
   ((:FRAMEWORK "GLFW") (:DEFAULT "glfw") (:DEFAULT "libglfw"))
   [Condition of type CFFI:LOAD-FOREIGN-LIBRARY-ERROR]
 
Restarts:
 0: [RETRY] Try loading the foreign library again.
 1: [USE-VALUE] Use another library instead.
 2: [TRY-RECOMPILING] Recompile glfw and try loading it again
 3: [RETRY] Retry performing #<ASDF:LOAD-OP NIL {1003C8A391}> on #<ASDF:CL-SOURCE-FILE "glfw" {1003607D91}>.
 4: [ACCEPT] Continue, treating #<ASDF:LOAD-OP NIL {1003C8A391}> on #<ASDF:CL-SOURCE-FILE "glfw" {1003607D91}> as hav$
 5: [RETRY] Retry SLIME REPL evaluation request.
 --more--

WTF? Cut to the present, skipping a few hours of very frustrating head banging trying to get the damned thing to load into my SBCL image made me realise that sometimes you have to go back to basics. In this case, that will be the FFI system. Up until now most of the packages I have used (cl-sql, cl-opengl and many others) have all used the UFFI system, but this particular binding is built atop the more modern and allegedly funkier CFFI.

Over the next hour or three I tried in vain to tweak DYLD_LIBRARY_PATH, I read the manual pages for dlopen(3), I copied the library to ‘/usr/lib’, I tried to manually use CFFI:DEFINE-FOREIGN-LIBRARY and CFFI:LOAD-FOREIGN-LIBRARY at the REPL to reduce the problem to its smallest incarnation and all with the same message as before. I was by now beginning to think that the actual library itself might be faulty in some way, so I did this:

otool -tv /usr/local/lib/libglfw.dylib | grep __* | less

and got a huge list of exported symbols. This kind of made me feel a little more confident that the library that had been built was in fact usable: if otool managed to load it and print out the symbols then at least it was a good ‘un!

So either CFFI was having issues in actually locating it or it was not able to actually load the thing into memory to apply the bindings. Now we are getting to the limits of my knowledge on this stuff so from here on in, I have been learning / hacking and writing as I go so bear with me. At this juncture I do not even know if I will win or end up going back to cl-opengl.

The first thing I thought to do was go read the source code where the error condition was actually raised: this is in the libraries.lisp file, here is the code…

246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271

(defun load-foreign-library (library)
  "Loads a foreign LIBRARY which can be a symbol denoting a library defined
through DEFINE-FOREIGN-LIBRARY; a pathname or string in which case we try to
load it directly first then search for it in *FOREIGN-LIBRARY-DIRECTORIES*;
or finally list: either (:or lib1 lib2) or (:framework <framework-name>)."
  (restart-case
      (typecase library
        (symbol
         (let* ((lib (get-foreign-library library))
                (spec (foreign-library-spec lib)))
           (when spec
             (setf (foreign-library-handle lib)
                   (load-foreign-library-helper library spec))
             lib)))
        (t
         (make-instance 'foreign-library :spec (list (list library))
                        :handle (load-foreign-library-helper nil library))))
    ;; Offer these restarts that will retry the call to
    ;; LOAD-FOREIGN-LIBRARY.
    (retry ()
      :report "Try loading the foreign library again."
      (load-foreign-library library))
    (use-value (new-library)
      :report "Use another library instead."
      :interactive read-new-value
      (load-foreign-library new-library))))

So was it failing to load because it didn’t find it or because it couldn’t eat it?

The next thing that I tried was loading CFFI manually first, then adding the location of the library to the foreign library path list like so:

(pushnew "/usr/local/lib/" cffi:*foreign-library-directories*)

and tried again. Same error. Just to make sure, I evaluated that variable from inside the debugger by hitting ‘e’ and entering the name and getting…

cffi:*foreign-library-directories* [Return]
("/usr/local/lib/")  => result

So it looks like the library is refusing to load or I have done something really really dumb up to this point. If you are in a position to put me out of my misery then please leave a comment, only (legal) suggestions relevant to the problem at hand will be considered.

Finally I read the CFFI source code and ultimately all I got was a bad head-ache. Sometimes reading the source is *not* a good idea as it only serves to remove you even further from a solution unless you absolutely know what you are doing, and thus far in time I haven’t a clue about how CFFI works.

As it is a portable wrapper across various LISP implementations I guess that it would be using the built-in SBCL FFI mechanisms. I found a file in the CFFI src folder called ‘cff-sbcl.lisp’ which I am now going to wander off and study over a nice cold pint of cider. Apple mac and apple juice. A great combination.

So. right now I am no nearer getting myself a working solution BUT there are some points to bear in mind for on-going progress..

1. I have not given up yet
2. I never give up, esp. with dumb computers.
3. What one man can do, another man can do.

I know it *can* be done because the author of the cl-glfw binding used it to write CityScape and if he managed to do that then I sure as hell will get over my problems.

I am now going to write some code to use the SBCL FFI calls to try and load the library and if that fails I am gong to write some tutorials on writing a dynamically loadable library using gcc on a Mac and then load it using SBCL/FFI, just so as I know what the hell is going on.

Then I will write my own FFI binding for my own little library and hopefully then things will start to make a little more sense.

Stay tuned. …