My Big Ninkasi Presentation

So quite a while ago I started building a scripting language and, over the course of about a year, actually finished it (for some definitions of "finished"). The scripting language was named "Ninkasi", after the Sumerian beer goddess.

Years later, as part of a job interview, the company I was applying to wanted me to do a presentation to the programming team. The presentation could be whatever I wanted it to be, so I picked the one thing I'm more familiar with than anyone else, more complex than most stuff I've worked on, has well-defined goals, and is a project that I actually finished, and that was my scripting language.

I've gone ahead and transcribed the slides I made from that presentation here, for anyone interested in the inner workings or development of it.

It's a project I'm still damn proud of, even if there are a few things I'd do differently if I did it again, and I cover those things near the end of the presentation.

Original slides are available here, created on Aug 8th, 2020: Google Slides deck. (Going forward, any updates are going to happen on this article, so the slides may be outdated.)

The scripting language source itself is available here:

The preprocessor is available here:

1. Ninkasi

I made a scripting language and now you have to hear me talk about it.

2. Why on Earth would you do something like that?

I spent years on this so I better make this answer good.

3. Features I wanted

4. More features I wanted

5. Even more features I wanted

6. Features I wanted, last one

7. Okay I lied. This is the most important one

8. Other cool stuff

Wasn’t part of the original reasoning but I like it anyway!

9. Cool stuff

10. Cool stuff (cont.)

11. Cool stuff (cont.)

12. Example code

What the heck does this thing even look like?

13. C API Example

This is an extremely simple program that creates a VM, hooks up an output function, compiles a one-line hard-coded script, executes it, and cleans up. Error reporting is minimal.

#include "nkx.h"

#include <stdio.h>
#include <assert.h>

// "print" function callback.
void printFunc(struct NKVMFunctionCallbackData *data)
{
    nkuint32_t i;
    for(i = 0; i < data->argumentCount; i++) {
        printf("%s", nkxValueToString(data->vm, &data->arguments[i]));
    }
}

int main(int argc, char *argv[])
{
    // Create the VM.
    struct NKVM *vm = nkxVmCreate();

    // Create a compiler so we can compile new code. (Loading binary
    // state snapshots doesn't need this.)
    struct NKCompilerState *compiler = nkxCompilerCreate(vm);

    // The most basic function we will need is "print". Otherwise it's
    // not possible to get anything out. (Well, it is. You just have
    // to have the hosting application explicitly pull data out of the
    // finished VM.)
    nkxVmSetupExternalFunction(
        vm, compiler,
        "print",   // Used as as internal name for matching up during
                   // deserialization AND as a variable name so the
                   // script can call it.
        printFunc, // The function itself.
        nktrue,    // True to add this as a global variable (otherwise
                   // there is no way to call it right away).
        NK_INVALID_VALUE // Everything else is optional argument type
                         // and argument count checking.
        );

    // Compile the script. (You would do this multiple times before
    // finalizing for multi-file scripts, or use a preprocessor to
    // make it all go in as a single string.)
    const char *scriptText = "print(\"foobar\\n\");\n";
    nkxCompilerCompileScript(
        compiler,
        scriptText,
        "internal" // Source file name (for error reporting).
        );

    // Done adding source files to compile. Write the end and finish
    // setting up exported data.
    nkxCompilerFinalize(compiler);

    // TODO: Error check the compiler output for real (and display
    // error messages if needed).
    assert(!nkxVmHasErrors(vm));

    // Run the program. (There are other ways to trigger execution,
    // but this is the simplest.)
    nkxVmExecuteProgram(vm);

    // TODO: Error check execution.
    assert(!nkxVmHasErrors(vm));

    // Clean up and return success.
    nkxVmDelete(vm);
    return 0;
}

14. Ninkasi Code Example

Here's a simple example of Ninkasi script code. It generates a Mandelbrot pattern (using numbers instead of colors), and prints it to the console.

For this script to work, it would need a "print" function created, just like above in the C API example. ("print" is not a build-in function.)

// Mandelbrot generator demo for Ninkasi

for(var y = -1.0; y < 1.0; y = y + 0.05) {

    for(var x = -2.0; x < 1.0; x = x + 0.03) {

        var u = 0.0;
        var v = 0.0;
        var u2 = u * u;
        var v2 = v * v;
        var k;

        for(k = 1; k < 100 && u2 + v2 < 4.0; ++k) {
            v = 2.0 * u * v + y;
            u = u2 - v2 + x;
            u2 = u * u;
            v2 = v * v;
        }

        if(k < 40) {
            print(k % 10);
        } else {
            print(".");
        }
    }

    print("\n");
}

15. The language itself

16. Types

17. Syntax

18. Syntaxes

19. Syntax 3

20. Syntax: Resurrection

Coroutines:

// This is the function that the coroutine will execute.
function functionToCall()
{
    print("Hello\n");
    // Yield control back to the parent context.
    yield();
    print("there!\n");
}

// Create the coroutine object.
var coroutineObject = coroutine(functionToCall);

// Run it.
print("1... ");
resume(coroutineObject);
print("2... ");
resume(coroutineObject);

Output:

1... Hello
2... there!

21. Error handling

What to do when everything explodes so you don’t HCF for real.

22. “Normal” runtime/compile errors

23. malloc() failure handling

24. malloc() failure is considered a "catastrophic error"

25. Multi-module compilation

Or how I learned to stop worrying and wrote a C89 preprocessor from scratch.

26. I wrote a C89 preprocessor from scratch.

27. … in C89.

28. Bytecode

There’s no assembler so if you want to go lower-level you get to do it the hard fun way.

29. 42 instructions

30. Memory

31. Serialized binary format

32. Difficult Fun bugs found during development

33. Real-mode DOS specific stuff

34. PowerPC (Linux) specific

35. Everywhere

36. Everywhere (cont.)

37. Lessons learned

38. Future Work

39. Faster hash tables

40. PUSHLITERAL_INT is like 1/4 to 1/2 of my generated instructions

41. Documentation’s not great

42. Add postfix increment/decrement

43. Remove the previously-mentioned anti-feature

44. “Closures” handled in a mediocre way

45. Variadic functions

46. Inline function compilation

47. Type queries

48. An actual standard library

That’s all!

Gruedorf

Also, I'm counting this as my official first Gruedorf entry.

It's a bit coincidental, because I didn't realize it had been going until right before I decided to publish it (and also, hopefully, have it be the start of me blogging regularly again).

Posted: 2021-08-02

Tags: gruedorf, ninkasi