Introducing Witch, a Text Editor and Command Line Shell for Lua Programmers

For the past few weeks, I’ve been hard at work on Witch, a program that combines a text editor and a programming REPL, a desire I’ve written about before.

The goal is to make a GUI program that replaces a text editor, my terminal emulator, the Lua command-line interpreter, and a testing environment for Lua code I’m writing. It seems like a lofty goal, but that’s only really true if you don’t see how all these things are related. On that subject, let’s talk about Lua.

Sandboxing with Lua

I love Lua. It’s a dead-simple programming language that comes with just enough features that I can feel productive without installing a billion dependencies, but it’s small enough that I feel like it’s possible to master the language and its environment instead of being stuck as a “perpetual beginner.”

One of the features I enjoy most about Lua is that it provides code that allows the user to compile arbitrary text as if it was Lua code.

-- Compile and execute Lua code contained in the given string.
function interpret(code)
	local func = load(code)
	return func()
end

Of course, whatever code is passed in would have full access to all global variables that exist in Lua — quite the security risk! Thankfully, the load() function allows the caller to specify a custom environment in which the code will execute. The environment is a key/value table where each key is the name of a variable in that environment, and the value is the value of a variable. Let’s use this to secure the code by only giving it access to a few helper libraries.

-- Only give access to the builtin table and string libraries, plus the print function.
local interpret_env = {
	table = table,
	string = string,
	print = print,
}

local function interpret(code)
	local f = load(code, nil, nil, interpret_env)
	f()
end

Let’s test this with Witch.

Note that we call into interpret() several times, each using a pair of square brackets to indicate a long string of text. The first three calls to interpret() run fine — it’s only the last call, which calls into a function in Lua’s debug library, that we generate an error. Given how we’ve implemented interpret(), this is to be expected as the environment the custom code is loaded into does not include Lua’s debug library!

Inventing a Programming Language in Real Time

One application of Lua is to write domain specific languages (DSLs) to be applied in specific problem domains. The most common application is for configuration, where Lua’s somewhat flexible semantics allow for relatively unskilled users to write simple configuration scripts to control a larger C program. Usually, a sandbox environment is created that provides a safe interface to the aforementioned program, safely limiting the user’s access through Lua.

We’re going to do something a little different.

In cases where Lua code tries to do something that is going to fail in some capacity, Lua consults something called the Metatable. Failure cases include trying to convert a value into a string, or querying a non-existent entry in a table. In the former case, Lua generates a boilerplate string that describes the type of the object and where it lives in memory. In the latter case, it returns a nil (nothing) value — unless Lua sees a metatable entry for indexing on that table.

local example_mt = {
	__index = function(self, key)
		return key
	end,
}

In this trivial example, when trying to query for a nonexistent key on a table, the key is returned verbatim. Not terribly useful.

Remember in the example above, how an environment table had a function in it? Well, what if when we tried to call a nonexistent function… we created one on the fly?

local interpret_mt = {
	__index = function(self, key)
		return function(...)
			local tbl = { key, ... }
			return table.concat(tbl, "\t")
		end
	end,
}

…and when we test it…

Let’s break down that call to interpret().

1. We call into print(), which is a function that exists.
2. We call into test(), which is a function that doesn’t exist.
3. Our metatable generates a function for test which returns a string with the function name and its arguments joined together with tab characters delimitng them.
4. The dynamic test() function is called, returning the string, which is passed into print(), a real function.

So, now we have an infinitely large library for generating strings based on the names of functions. The strings are, by and large, useless, but it wouldn’t take much effort to make this library much more useful.

local function render_tag(name, child)
	if type(child) == "table" then child = table.concat(child, "") end
	return ("<%s>%s</%s>"):format(name, child, name)
end

local env_mt = {
	__index = function(self, key)
		return function(child)
			return render_tag(key, child)
		end
	end,
}
local env = {}
setmetatable(env, env_mt)

local function html(code)
	return assert(load(code, nil, nil, env))()
end

The first function, render_tag(), renders an HTML tag with any arbitrary number of children, and returns the result as a string of text. The second takes in arbitrary Lua code, and creates an environment with no functions where any function call instead returns an HTML tag as returned by render_tag().

This means, we can write Lua code to generate syntactically equivalent HTML. Let’s test it!

We now have a generator for a language like HTML, all without needing to define any functions whose names are shared with HTML elements. Notice that the table named “env” contains no elements — this new DSL doesn’t define any built-in functions. By formalizing the logic of how this DSL is used, we’ve completely circumvented the need to actually define it.

On top of that, it’s immediately useful for generating HTML — the sidebar’s output shows as much!

We Can Do Better

…but the output of this DSL isn’t very useful on its own. To get anything meaningful, one would need to copy it from the sidebar, paste it into an HTML document, and open the document in the browser.

We can do better than that, thanks to the Magic that Witch provides: conjuring.

local lgi = require "lgi"
local WebKit = lgi.require "WebKit"
local web_view = WebKit.WebView()

local function preview(html)
	web_view:load_html(html)
end

conjure(web_view)

This snippet uses a library called Lua GObject-Introspection (LGI) to dynamically bind to WebKitGtk, which provides a full web browser engine as Gtk widgets. It then instantiates a web view, and defines a function that loads arbitrary HTML into the web view. Lastly, the call to conjure() invokes a function provided by Witch which displays an arbitrary Gtk widget in a new tab within Witch itself.

I think you know the drill a this point — let’s test this live. First, our exact test case.

Next, this is what Witch looks like after that snippet is executed.

Seven meager lines of code, and the humble text editor becomes a full web preview environment.

To be clear, Witch does not provide a web browser at all. It doesn’t need to. By exposing a little function for plugging in Gtk widgets, Witch allows itself to become whichever tool the programmer needs.

And now, the rug pull.

Witch isn’t actually a text editor.

I thought what I was developing was a text editor, but when it came time to add support for multiple open files at once, where it would be necessary for each editor instance to provide its own functions to the command line, Witch made its true nature clear to me: It is a shell to quickly iterate on code using an extensible programming environment.

The text editor that is shown for these demos is a separate module which will conjure() itself into Witch when editing a text file.

Incidentally, this sandbox environment provided by Witch when conjuring components is how print() statements show their output in Witch’s sidebar. This application of environments allows Witch to be an excellent testing ground for arbitrary Lua modules, even without writing custom code specific to Witch.

Stitching it All Together

With conjuring in mind, let’s stitch together everything we’ve done so far.

Notice that we’re about to test it in a different way than we usually do; Instead of calling buffer_exec(), we’re calling selection_exec() so that we only execute the highlighted part of the buffer. The highlighted part calls into buffer_exec() so we don’t have to. The entire highlighted section lives inside a multi-line paragraph, so the buffer_exec() in our selection shouldn’t cause any recursion.

Immediately, we are presented with an underwhelmingly empty web view tab. Witch’s sidebar notes that a new function has been defined. Rather than define preview() as a local function like in previous tests, the helper code commented into our baby HTML framework has given us a global function. This function takes a string, executes it as Lua-flavoured HTML, then previews the result.

Let’s put it to work!

Notice that this code doesn’t involve anything but the DSL — nothing Witch-specific whatsoever. Instead, we’ll test this code by retrieving it using buffer_text(), a function provided by the text editor, and then passing the text to the preview() function we defined earlier.

The result?

We now have a way to directly render the output of our HTML generator, without needing to copy-paste anything around. No need to leave the editor to preview documents, but also no need to use a bespoke document editor either. It all exists in one place, stiched together by an extensible software environment that seamlessly blends the craft of text editing with the rapid iteration of the read-eval-print loop.

Next Steps

There’s a lot more to Witch than the thin vertical slice presented in this article, such as a way to specify custom syntax for commands provided by conjured widgets, as well as the ability to escape to the shell. There are also a few very rough edges that make Witch currently unsuitable for release — I hope to address that soon, so that Witch can bring its magic to your computers as well.

Stay tuned!