TarantoolScript: Static Typing for Tarantool Lua Scripts

Introduction

If you have ever written scripts for Tarantool, you probably understand my pain. Tarantool is a remarkable tool: it not only stores relatively large volumes of data and runs CRUD operations on that data surprisingly fast, but it also gives you very wide options for processing the data right inside the Tarantool runtime. And by data processing I don't just mean validation and applying some math to it, but nearly the whole range of what the Lua language provides, plus a whole pile of useful modules that come with Tarantool or are installed from third-party sources.

To write, say, a full-featured HTTP server on Tarantool (please don't beat me up for that phrasing), you need to know very little: the basics of Lua syntax and the API of Tarantool's own core modules. Lua itself is simple — I doubt anyone finds learning it in one evening a hard task — but Tarantool's modules are a bit trickier. You can study the official documentation cover to cover and still run into one unpleasant problem as soon as you start writing the script: building anything reasonably large for Tarantool is painfully awkward.

Problem

Here is where my pain comes from. I write everything in VS Code, and I end up writing Tarantool scripts quite often. That means no familiar editor autocompletion and no static type analysis while I work. Sure, the Lua language server gives you tooling for working with plain Lua source code, but as soon as Tarantool's built-in modules come into play, almost all of the related code has to be written blind. I can't be sure which module holds which function, what their signatures are, what they return, or how to call them. Memorizing the documentation is not an option ~~(I already forced myself to read it — what more do you want from me?)~~, and switching to the browser window every second to look something up is honestly not great.

Searching for a solution

One day, sitting at my laptop and typing away on yet another Lua script, an interesting thought occurred to me. What if there is a tool that lets you describe the types Tarantool provides, and then use that description while developing new scripts? Experience with JavaScript and TypeScript suggests that solving the problem this way is not a bad idea. The idea is really simple, and it is strange that it only came to me a year after I first picked up Tarantool. A quick search turned up three promising directions for working on the problem.

LuaLS — a VS Code extension with a language server for Lua. It lets you write your own add-on that describes the type schema you need. I already use this extension anyway, and the idea of adding the pieces I need to it looks quite interesting.
Teal — a Lua dialect with static typing. Something like TypeScript, but for Lua. It also lets you describe the types you use and then compile the program to a plain Lua script. Looks interesting as well.
TypeScriptToLua (TSTL) — and this one is real TypeScript that compiles to Lua.

There are probably other tools that solve my problem, but I decided not to look past these three. The first two look fairly simple to implement, so I figured it would be nice to pick something ~~more useful~~ harder. I went with the third tool. Might as well brush up on TypeScript along the way.

Solution

I won't ~~drag it out~~ go on at tedious length about how I wrote my TSTL package, so as not to send the reader to sleep — here is the result of my ~~suffering~~ work straight away: TarantoolScript, a package that provides type declarations that can be useful when writing Tarantool scripts. Watch closely: I like Lua a lot, so I did everything I could not to write Lua :)

To get this thing running, you of course need the TypeScript and TSTL packages installed. And you need to know TypeScript, obviously. And you should skim the TSTL documentation a bit. So on top of the one Tarantool documentation, you now have two more docs to study. Great return on the effort, if you ask me.

But seriously, yes: instead of writing Lua mostly blind, the package lets you use almost all of TypeScript's features easily and without much pain. Compiling all of that down to Lua is TSTL's job.

Examples

A database migration, if you can call it that, might look something like this:

{
        name: '_create_partners',
        up: function (this: Migration, userName: string): void {
            box.once(this.name, () => {
                const space = box.schema.space.create('partners', {
                    if_not_exists: true,
                    engine: 'memtx',
                    user: userName,
                });

                space.format([
                    { name: 'id', type: 'uuid' },
                    { name: 'name', type: 'string' },
                ]);

                space.create_index('primary', {
                    unique: true,
                    if_not_exists: true,
                    parts: ['id'],
                });

                box.space._schema.delete(`once${this.name}_down`);
            });
        },
        down: function (this: Migration): void {
            box.once(`${this.name}_down`, () => {
                box.space.get('partners').drop();
                box.space._schema.delete(`once${this.name}`);
            });
        },
    },

Here is how it compiles:

{
        name = "_create_partners",
        up = function(self, userName)
            box.once(
                self.name,
                function()
                    local space = box.schema.space.create("partners", {if_not_exists = true, engine = "memtx", user = userName})
                    space:format({{name = "id", type = "uuid"}, {name = "name", type = "string"}})
                    space:create_index("primary", {unique = true, if_not_exists = true, parts = {"id"}})
                    box.space._schema:delete(("once" .. self.name) .. "_down")
                end
            )
        end,
        down = function(self)
            box.once(
                self.name .. "_down",
                function()
                    box.space.partners:drop()
                    box.space._schema:delete("once" .. self.name)
                end
            )
        end
    }

And here is how you can set up an HTTP server:

import * as http_server from 'http.server';

const httpd = http_server.new_(host, port, {
    log_requests: true,
});

httpd.route({ path: '/sync/:partner_id/:partner_user_id', method: 'GET' }, handlerSync);
httpd.route({ path: '/match', method: 'GET' }, handlerMatch);
httpd.route({ path: '/partners', method: 'GET' }, handlerPartners);
httpd.route({ path: '/partners', method: 'POST' }, handlerPartners);
httpd.route({ path: '/partners/:partner_id', method: 'GET' }, handlerPartners);
httpd.route({ path: '/partners/:partner_id', method: 'DELETE' }, handlerPartners);
httpd.start();

TSTL turns it into this:

local http_server = require("http.server")
local httpd = http_server.new(host, port, {log_requests = true})
httpd:route({path = "/sync/:partner_id/:partner_user_id", method = "GET"}, handlerSync)
httpd:route({path = "/match", method = "GET"}, handlerMatch)
httpd:route({path = "/partners", method = "GET"}, handlerPartners)
httpd:route({path = "/partners", method = "POST"}, handlerPartners)
httpd:route({path = "/partners/:partner_id", method = "GET"}, handlerPartners)
httpd:route({path = "/partners/:partner_id", method = "DELETE"}, handlerPartners)
httpd:start()

And all of it with editor autocompletion and static type checking. A fairy tale, right!

Not a fairy tale

There are problems, of course. TSTL itself, for instance, has been around for quite a while but has a number of issues. Just look at the number of open issues in the project's repository (119 at the time of writing). Source maps don't work quite right, module import controls are quite limited, there is constant confusion over whether functions need an explicit this or not, and so on. The project is far from abandoned, but it is not exactly updated every day.

Things to watch out for

If you have read this far, the article has probably caught your interest. I'll assume you might also want to try my library out with your own hands. So here is an approximate list of what to keep in mind when working with TarantoolScript in particular and TSTL in general:

Before you can access the box module, you have to declare it explicitly. In other words, somewhere at the top of your source file there must be this snippet: import { Box } from 'tarantoolscript'; declare const box: Box;
If you want to use some module other than box, you need to make a small change to tsconfig.json. For example, if you need to pull in the log module, the config must have these settings: { "compilerOptions": { "paths": { "log": ["tarantoolscript/src/log.d.ts"] } }, "tstl": { "noResolvePaths": [ "log" ] } } For luatest.helpers, for example, you need to add: { "compilerOptions": { "paths": { "luatest.helpers": ["tarantoolscript/src/luatest.helpers.d.ts"] } }, "tstl": { "noResolvePaths": [ "luatest.helpers" ] } } After that the modules can be imported: import * as log from 'log'; or import * as luatest_helpers from 'luatest.helpers'; For other modules, if they already have type descriptions, it is the same thing — only the names change.
To correctly receive the values of functions that return several values at once, use destructuring: const [ok, body] = pcall(() => req.json());

In general, you can pick up a lot of useful syntax from this and this project. The first is just a set of samples that show how TS code compiles into Lua (just don't forget npm run build). The second is an example of a project you can write using the tools I'm describing.

Conclusion

Near the end of the article I suddenly remembered that it would have been good to describe the goals of this work at the very beginning. But since ~~I'm not entirely sane~~ rewriting is a chore, I'll risk stating the goals here.

The first goal was, of course, to have a convenient tool that makes my life easier. The second and probably the main goal was to gain experience with Lua, TypeScript and Tarantool. The second goal has definitely been achieved, and how well the first goal worked out may only become clear after some time passes.

I'll guess that the most obvious question a skeptical reader can ask looks something like this: "Why do we need all this?" My equally obvious answer: because it's interesting! Otherwise why would we all be gathered here?

Links

Thanks for reading!