Play With Lua!

Lua is designed for embedding into larger host programs. Last time I talked about how to make C functions available to Lua, now I want to talk about how to bundle Lua libraries in with your host program.

First, we have to understand what it means to load a library. Most Lua libraries are packaged into modules (which I’ll get into later), one module per file. You call require and pass it a module name, and it returns the module (usually as a table containing some functions).

Notably, you’re not passing a file name to require, you’re passing a module name. The fact that these modules map to files is just sort of coincidence. In actuality, require looks in several different places (locations on package.path, C libraries on package.cpath, and so on) for a function that will evaluate to the module.

A function that evaluates to a module is called a loader. Searchers try to find loaders, so there’s a searcher that looks in files and returns functions that call dofile on them, for example. Each searcher that Lua uses is stored in the confusingly-named package.loaders, so one way to find this would be to add a searcher that will look for a particularly-named function (exposed from C) and call it, like if I try to load “foo” that means try to call “load_foo_from_c()“.

But, even before it starts calling searchers, it checks if you’ve provided a loader for that module by hand. You can do that by putting it in a table, package.preload. If you only want one module this is much easier to do.

Let’s say you have a module, Penlight, in pl.lua. You want to embed this in your app. So, you store it in your binary as a string, and provide a C function that returns the string (actually you’d use some kind of data file, but I’m keeping this simple). The first place it’ll look for the loader is package.preload['pl'], so put this there:

package.preload['pl'] = function()
    local str = load_pl_from_c()
    local fn = loadstring(str)
    return fn()
end

Now, when you call require 'pl' it will execute this function, read the text of the pl library from wherever the host stored it, and run it. And, if you’re testing your script outside the host program, it will still work; just don’t add anything to preload and it will just read the file from the path.

It’s easy to see how you could build a whole package system with this: byte-compile the files first and run them through zlib, put them all in a data file, since package.preload is a table that means you can put a metatable on it, etc. One idea I had was to tie this to curl and have it load packages from a repository over http.