{"id":44,"date":"2011-08-08T10:05:43","date_gmt":"2011-08-08T15:05:43","guid":{"rendered":"http:\/\/www.playwithlua.com\/?p=44"},"modified":"2020-05-22T13:39:19","modified_gmt":"2020-05-22T18:39:19","slug":"lua-distribution","status":"publish","type":"post","link":"http:\/\/www.playwithlua.com\/?p=44","title":{"rendered":"Lua distribution"},"content":{"rendered":"<p>So you have made a program in Lua. You want to give it to people to run. You run into two problems right away: one, you don&#8217;t want to require people to install Lua to run your program, and two, you may not want them to have your source code (if it&#8217;s a commercial program). So, what to do?<\/p>\n<p>If we were in a language like C, we could precompile our program into a binary and distribute that. So, since Lua can very closely integrate with C, can we do that with Lua? The answer turns out to be &#8220;yes&#8221;.<br \/>\n<!--more--><br \/>\nThere are three steps to this process: first, we need to turn our Lua files into objects that a linker can put into a C program. Then, we need to tell the C program how to load the embedded objects. Finally, we need a way for Lua to know how to get them out of C, so that <code>require<\/code> will work.<\/p>\n<h2>Turning Lua into object files<\/h2>\n<p>When you compile a C program, there are two general steps: the compiler (gcc, say) turns your source code into an <em>object file<\/em>. This is a binary file that defines a bunch of objects, like variable declarations and functions. Then, a linker comes along and turns the object files into a program, by pulling out objects from each file (and shared files you reference, like shared libraries) and embedding them into a binary. So, if we can turn a Lua script into an object file, we can use the linker to embed it into the program.<\/p>\n<p>One way to do this is to just make a C source file with a big string that has all your code in it. This would work but it&#8217;s a little stupid; you&#8217;d have to care about escaping quotes and things, it gets real hard real fast. It turns out that there&#8217;s an easier way:<\/p>\n<p>Let&#8217;s say we have the following Lua code, in <code>hello.lua<\/code>:<\/p>\n\n<div class=\"my_syntax_box\"><div class=\"my_syntax\"><div class=\"code\"><pre class=\"lua\" style=\"font-family:monospace;\"><span style=\"color: #0000aa;\">print<\/span> <span style=\"color: #ff6666;\">&quot;Hello, world!&quot;<\/span><\/pre><\/div><\/div><\/div>\n\n<p>We can do this to make an object file out of it:<\/p>\n<pre>\r\nld -r -b binary -o hello.o hello.lua\r\n<\/pre>\n<p>(Unfortunately, this is only an option with GNU ld, not Apple&#8217;s, so if you have a Mac, you&#8217;ll need something different. It&#8217;s not too hard to write a program is Lua to turn an arbitrary file to a C source file)<\/p>\n<p>So, now you have an object file that you can link in with the rest of your program. This object file will define three file-scoped variables we can refer to:<\/p>\n\n<div class=\"my_syntax_box\"><div class=\"my_syntax\"><div class=\"code\"><pre class=\"c\" style=\"font-family:monospace;\"><span style=\"color: #000000; font-weight: bold;\">extern<\/span> <span style=\"color: #993333;\">char<\/span> binary_hello_lua_start<span style=\"color: #009900;\">&#91;<\/span><span style=\"color: #009900;\">&#93;<\/span><span style=\"color: #339933;\">;<\/span>\n<span style=\"color: #000000; font-weight: bold;\">extern<\/span> <span style=\"color: #993333;\">char<\/span> binary_hello_lua_end<span style=\"color: #009900;\">&#91;<\/span><span style=\"color: #009900;\">&#93;<\/span><span style=\"color: #339933;\">;<\/span>\n<span style=\"color: #000000; font-weight: bold;\">extern<\/span> <span style=\"color: #993333;\">char<\/span> binary_hello_lua_size<span style=\"color: #009900;\">&#91;<\/span><span style=\"color: #009900;\">&#93;<\/span><span style=\"color: #339933;\">;<\/span><\/pre><\/div><\/div><\/div>\n\n<p>We only care about the start and size variables, end isn&#8217;t really useful.<\/p>\n<h2>Loading embedded resources in C<\/h2>\n<p>In our main program file, we need to declare the variables as extern references, just like above. Then, we can use <code>luaL_loadbuffer<\/code> to compile and load the code:<\/p>\n\n<div class=\"my_syntax_box\"><div class=\"my_syntax\"><div class=\"code\"><pre class=\"lua\" style=\"font-family:monospace;\">lua_State <span style=\"color: #66cc66;\">*<\/span>lua <span style=\"color: #66cc66;\">=<\/span> lua_open<span style=\"color: #66cc66;\">&#40;<\/span><span style=\"color: #66cc66;\">&#41;<\/span><span style=\"color: #66cc66;\">;<\/span>\nluaL_openlibs<span style=\"color: #66cc66;\">&#40;<\/span>lua<span style=\"color: #66cc66;\">&#41;<\/span><span style=\"color: #66cc66;\">;<\/span>\nluaL_loadbuffer<span style=\"color: #66cc66;\">&#40;<\/span>lua<span style=\"color: #66cc66;\">,<\/span> binary_hello_lua_start<span style=\"color: #66cc66;\">,<\/span> <span style=\"color: #66cc66;\">&#40;<\/span>int<span style=\"color: #66cc66;\">&#41;<\/span>&amp;amp<span style=\"color: #66cc66;\">;<\/span>binary_hello_lua_size<span style=\"color: #66cc66;\">,<\/span> <span style=\"color: #ff6666;\">&quot;hello.lua&quot;<\/span><span style=\"color: #66cc66;\">&#41;<\/span><span style=\"color: #66cc66;\">;<\/span><\/pre><\/div><\/div><\/div>\n\n<p>This compiles our code (loaded from the embedded resource) and pushes it, as a function, on to the stack. We can call it with <code>lua_pcall<\/code>:<\/p>\n\n<div class=\"my_syntax_box\"><div class=\"my_syntax\"><div class=\"code\"><pre class=\"c\" style=\"font-family:monospace;\">lua_pcall<span style=\"color: #009900;\">&#40;<\/span>lua<span style=\"color: #339933;\">,<\/span> <span style=\"color: #0000dd;\">0<\/span><span style=\"color: #339933;\">,<\/span> LUA_MULTRET<span style=\"color: #339933;\">,<\/span> <span style=\"color: #0000dd;\">0<\/span><span style=\"color: #009900;\">&#41;<\/span><span style=\"color: #339933;\">;<\/span><\/pre><\/div><\/div><\/div>\n\n<p>So now if we run this, it ought to say &#8220;Hello, World!&#8221;. But this is sort of pointless if we can only embed one file; most of the time we&#8217;ll have multiple modules or libraries or whatever. It would be nice if we could load things the same way in embedded mode as normal mode, too.<\/p>\n<h2>Letting Lua decide what to load<\/h2>\n<p>When you require a module in Lua, it uses the very flexible package loading system to find your module. There&#8217;s an array of searcher functions that take a module name and either return the module compiled as a function, or nil. Conveniently, they return exactly what <code>luaL_loadbuffer<\/code> does above. So, if we had a (C) function that would take a module name and either call <code>luaL_loadbuffer<\/code> or push nil, then we could register that function in Lua, add it to <code>package.loaders<\/code>, and then use <code>require<\/code> normally.<\/p>\n<p>The fundamental problem here is that we need to do reflection in C: we need a way to translate between the string &#8220;hello&#8221; and the variable <code>binary_hello_lua_start<\/code>. Unfortunately there&#8217;s no easy way to do that; we have to build a lookup table, so I&#8217;ll leave that as an exercise (since the focus here is on the actual embedding). If we just have our function assume that you&#8217;re always going to load &#8220;hello&#8221;, we can write it like this:<\/p>\n\n<div class=\"my_syntax_box\"><div class=\"my_syntax\"><div class=\"code\"><pre class=\"c\" style=\"font-family:monospace;\"><span style=\"color: #993333;\">int<\/span> get_module<span style=\"color: #009900;\">&#40;<\/span>lua_State <span style=\"color: #339933;\">*<\/span>lua<span style=\"color: #009900;\">&#41;<\/span><span style=\"color: #009900;\">&#123;<\/span>\n    <span style=\"color: #993333;\">const<\/span> <span style=\"color: #993333;\">char<\/span> <span style=\"color: #339933;\">*<\/span>modname <span style=\"color: #339933;\">=<\/span> luaL_checkstring<span style=\"color: #009900;\">&#40;<\/span>lua<span style=\"color: #339933;\">,<\/span> <span style=\"color: #0000dd;\">1<\/span><span style=\"color: #009900;\">&#41;<\/span><span style=\"color: #339933;\">;<\/span>\n&nbsp;\n    <span style=\"color: #808080; font-style: italic;\">\/* This is just an example, in a real app we'd\n       turn modname into a real resource pointer\n       instead of just using hello.lua *\/<\/span>\n    <span style=\"color: #b1b100;\">if<\/span><span style=\"color: #009900;\">&#40;<\/span><span style=\"color: #0000dd;\">1<\/span> <span style=\"color: #339933;\">==<\/span> <span style=\"color: #0000dd;\">1<\/span><span style=\"color: #009900;\">&#41;<\/span><span style=\"color: #009900;\">&#123;<\/span>\n        luaL_loadbuffer<span style=\"color: #009900;\">&#40;<\/span>lua<span style=\"color: #339933;\">,<\/span> binary_hello_lua_start<span style=\"color: #339933;\">,<\/span> <span style=\"color: #009900;\">&#40;<\/span><span style=\"color: #993333;\">int<\/span><span style=\"color: #009900;\">&#41;<\/span><span style=\"color: #339933;\">&amp;<\/span>amp<span style=\"color: #339933;\">;<\/span>binary_hello_lua_size<span style=\"color: #339933;\">,<\/span> modname<span style=\"color: #009900;\">&#41;<\/span><span style=\"color: #339933;\">;<\/span>\n    <span style=\"color: #009900;\">&#125;<\/span> <span style=\"color: #b1b100;\">else<\/span> <span style=\"color: #009900;\">&#123;<\/span>\n        lua_pushnil<span style=\"color: #009900;\">&#40;<\/span>lua<span style=\"color: #009900;\">&#41;<\/span><span style=\"color: #339933;\">;<\/span>\n    <span style=\"color: #009900;\">&#125;<\/span>\n&nbsp;\n    <span style=\"color: #b1b100;\">return<\/span> <span style=\"color: #0000dd;\">1<\/span><span style=\"color: #339933;\">;<\/span>\n<span style=\"color: #009900;\">&#125;<\/span><\/pre><\/div><\/div><\/div>\n\n<p>Now we just need to register it and put it in <code>package.loaders<\/code>:<\/p>\n\n<div class=\"my_syntax_box\"><div class=\"my_syntax\"><div class=\"code\"><pre class=\"c\" style=\"font-family:monospace;\">lua_register<span style=\"color: #009900;\">&#40;<\/span>lua<span style=\"color: #339933;\">,<\/span> <span style=\"color: #ff0000;\">&quot;get_module&quot;<\/span><span style=\"color: #339933;\">,<\/span> get_module<span style=\"color: #009900;\">&#41;<\/span><span style=\"color: #339933;\">;<\/span>\nluaL_dostring<span style=\"color: #009900;\">&#40;<\/span>lua<span style=\"color: #339933;\">,<\/span> <span style=\"color: #ff0000;\">&quot;table.insert(package.loaders, get_module)&quot;<\/span><span style=\"color: #009900;\">&#41;<\/span><span style=\"color: #339933;\">;<\/span><\/pre><\/div><\/div><\/div>\n\n<p>Finally we just load the module, and <code>require<\/code> will take care of the rest:<\/p>\n\n<div class=\"my_syntax_box\"><div class=\"my_syntax\"><div class=\"code\"><pre class=\"c\" style=\"font-family:monospace;\">luaL_dostring<span style=\"color: #009900;\">&#40;<\/span>lua<span style=\"color: #339933;\">,<\/span> <span style=\"color: #ff0000;\">&quot;require 'hello'&quot;<\/span><span style=\"color: #009900;\">&#41;<\/span><span style=\"color: #339933;\">;<\/span><\/pre><\/div><\/div><\/div>\n\n<h2>Wrapping up<\/h2>\n<p>So, now we can take any arbitrary Lua code, link it into a C binary, and distribute a Lua-powered application without giving up our source code. The C library can statically link Lua, and can export other functions (to tie in other libraries), so that we have an application that&#8217;s the best of both worlds: high-level garbage-collected language for all our logic, low-level language for talking to libraries, and a simple binary distribution to make it easy for users to run it.<\/p>\n<p>The code for this entry is <a onclick=\"javascript:pageTracker._trackPageview('\/outgoing\/gist.github.com\/1131236');\"  href=\"https:\/\/gist.github.com\/1131236\">here.<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>So you have made a program in Lua. You want to give it to people to run. You run into two problems right away: one, you don&#8217;t want to require people to install Lua to run your program, and two, you may not want them to have your source code (if it&#8217;s a commercial program). [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"http:\/\/www.playwithlua.com\/index.php?rest_route=\/wp\/v2\/posts\/44"}],"collection":[{"href":"http:\/\/www.playwithlua.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.playwithlua.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.playwithlua.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.playwithlua.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=44"}],"version-history":[{"count":0,"href":"http:\/\/www.playwithlua.com\/index.php?rest_route=\/wp\/v2\/posts\/44\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.playwithlua.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=44"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.playwithlua.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=44"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.playwithlua.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=44"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}