When Go meets Lua

While playing at GitHub, I stumbled upon Gopher lua, a lua virtual machine implemented purely in Golang. We know that Golang is a static language, while Lua is a dynamic language. Golang’s performance and efficiency are very good in various languages, but in dynamic ability, it is certainly not comparable to Lua. So if we can combine the two, we can combine the best of both.

In the project Wiki, we can see that gopher-Lua is only worse in terms of execution efficiency and performance than THE C implementation bindings. So from a performance standpoint, this should be a very good virtual machine solution.

Hello World

Here is a simple Hello World program. We created a new virtual machine and then DoString it (…). Explains executing lua code and finally shutting down the virtual machine. Executing the program, we will see the string “Hello World” on the command line.

package main

import (
	"github.com/yuin/gopher-lua"
)

func main(a) {
	l := lua.NewState()
	defer l.Close()
	if err := l.DoString(`print("Hello World")`); err ! =nil {
		panic(err)
	}
}

// Hello World
Copy the code

To compile

After viewing the above DoString(…) Method after the call chain, we find that every time we execute DoString(…) Or DoFile (…). Parse and compile are executed once each.

func (ls *LState) DoString(source string) error {
	iffn, err := ls.LoadString(source); err ! =nil {
		return err
	} else {
		ls.Push(fn)
		return ls.PCall(0, MultRet, nil)}}func (ls *LState) LoadString(source string) (*LFunction, error) {
	return ls.Load(strings.NewReader(source), "<string>")}func (ls *LState) Load(reader io.Reader, name string) (*LFunction, error) {
	chunk, err := parse.Parse(reader, name)
	// ...
	proto, err := Compile(chunk, name)
	// ...
}
Copy the code

From this point of view, in a scenario where the same Lua code will be executed multiple times (as in HTTP Server, the same Lua code will be executed on each request), if we can pre-compile the code, You should be able to reduce the overhead of parse and compile if this is hotPath code. According to Benchmark, early compilation does reduce unnecessary overhead.

package glua_test

import (
	"bufio"
	"os"
	"strings"

	lua "github.com/yuin/gopher-lua"
	"github.com/yuin/gopher-lua/parse"
)

// Compile the Lua code field
func CompileString(source string) (*lua.FunctionProto, error) {
	reader := strings.NewReader(source)
	chunk, err := parse.Parse(reader, source)
	iferr ! =nil {
		return nil, err
	}
	proto, err := lua.Compile(chunk, source)
	iferr ! =nil {
		return nil, err
	}
	return proto, nil
}

// Compile lua code files
func CompileFile(filePath string) (*lua.FunctionProto, error) {
	file, err := os.Open(filePath)
	defer file.Close()
	iferr ! =nil {
		return nil, err
	}
	reader := bufio.NewReader(file)
	chunk, err := parse.Parse(reader, filePath)
	iferr ! =nil {
		return nil, err
	}
	proto, err := lua.Compile(chunk, filePath)
	iferr ! =nil {
		return nil, err
	}
	return proto, nil
}

func BenchmarkRunWithoutPreCompiling(b *testing.B) {
	l := lua.NewState()
	for i := 0; i < b.N; i++ {
		_ = l.DoString(`a = 1 + 1`)
	}
	l.Close()
}

func BenchmarkRunWithPreCompiling(b *testing.B) {
	l := lua.NewState()
	proto, _ := CompileString(`a = 1 + 1`)
	lfunc := l.NewFunctionFromProto(proto)
	for i := 0; i < b.N; i++ {
		l.Push(lfunc)
		_ = l.PCall(0, lua.MultRet, nil)
	}
	l.Close()
}

// goos: darwin
// goarch: amd64
// pkg: glua
// BenchmarkRunWithoutPreCompiling-8 100000 19392 ns/op 85626 B/op 67 allocs/op
// BenchmarkRunWithPreCompiling-8 1000000 1162 ns/op 2752 B/op 8 allocs/op
// PASS
/ / ok glua 3.328 s

Copy the code

Vm instance pool

In the scenario where the same Lua code is executed, in addition to optimizing performance with precompilation, we can also introduce virtual machine instance pools.

Because creating a New Lua virtual machine involves a lot of memory allocation, creating and destroying a Lua virtual machine on every run can consume a lot of resources. Introducing a VM instance pool can reuse VMS and reduce unnecessary overhead.

func BenchmarkRunWithoutPool(b *testing.B) {
	for i := 0; i < b.N; i++ {
		l := lua.NewState()
		_ = l.DoString(`a = 1 + 1`)
		l.Close()
	}
}

func BenchmarkRunWithPool(b *testing.B) {
	pool := newVMPool(nil.100)
	for i := 0; i < b.N; i++ {
		l := pool.get()
		_ = l.DoString(`a = 1 + 1`)
		pool.put(l)
	}
}

// goos: darwin
// goarch: amd64
// pkg: glua
// BenchmarkRunWithoutPool-8 10000 129557 ns/op 262599 B/op 826 allocs/op
// BenchmarkRunWithPool-8 100000 19320 ns/op 85626 B/op 67 allocs/op
// PASS
/ / ok glua 3.467 s
Copy the code

Benchmark results show that virtual machine instance pools can indeed reduce memory allocation.

The instance pool implementation provided by the README is shown below, but it is noted that the implementation did not create enough virtual machine instances in its initial state (the initial number of instances was 0), and the dynamic scaling of Slice is problematic, both of which are areas for improvement.

type lStatePool struct {
    m     sync.Mutex
    saved []*lua.LState
}

func (pl *lStatePool) Get(a) *lua.LState {
    pl.m.Lock()
    defer pl.m.Unlock()
    n := len(pl.saved)
    if n == 0 {
        return pl.New()
    }
    x := pl.saved[n- 1]
    pl.saved = pl.saved[0 : n- 1]
    return x
}

func (pl *lStatePool) New(a) *lua.LState {
    L := lua.NewState()
    // setting the L up here.
    // load scripts, set global variables, share channels, etc...
    return L
}

func (pl *lStatePool) Put(L *lua.LState) {
    pl.m.Lock()
    defer pl.m.Unlock()
    pl.saved = append(pl.saved, L)
}

func (pl *lStatePool) Shutdown(a) {
    for _, L := range pl.saved {
        L.Close()
    }
}

// Global LState pool
var luaPool = &lStatePool{
    saved: make([]*lua.LState, 0.4),}Copy the code

Module calls

Gopher – Lua supports lua to call the Go module. Personally, this is a very exciting function point, because in Golang program development, we may design many commonly used modules, this cross-language call mechanism, so that we can reuse the code, tools.

Of course, there are also Go calls to the Lua module, but I don’t think the latter is necessary, so I won’t cover it here.

package main

import (
	"fmt"

	lua "github.com/yuin/gopher-lua"
)

const source = `
local m = require("gomodule")
m.goFunc()
print(m.name)
`

func main(a) {
	L := lua.NewState()
	defer L.Close()
	L.PreloadModule("gomodule", load)
	iferr := L.DoString(source); err ! =nil {
		panic(err)
	}
}

func load(L *lua.LState) int {
	mod := L.SetFuncs(L.NewTable(), exports)
	L.SetField(mod, "name", lua.LString("gomodule"))
	L.Push(mod)
	return 1
}

var exports = map[string]lua.LGFunction{
	"goFunc": goFunc,
}

func goFunc(L *lua.LState) int {
	fmt.Println("golang")
	return 0
}

// golang
// gomodule
Copy the code

Variable pollution

Another tricky problem is introduced when using instance pools to reduce overhead: global variables in the same virtual machine may be changed because the same Lua code may be executed multiple times. If the code logic depends on global variables, unpredictable results can occur (which smacks of “non-repeatable reads” in database isolation).

The global variable

If we need to limit Lua code to local variables, we need to limit global variables from this point of view. So how do you do that?

As we know, Lua is compiled into bytecode and then interpreted. We can then restrict the use of global variables during the bytecode compilation phase. After checking Lua virtual machine instructions, we found that there are two instructions involving global variables: GETGLOBAL (Opcode 5) and SETGLOBAL (Opcode 7).

At this point, the general idea is that we can limit the use of global variables in our code by determining whether the bytecode contains GETGLOBAL and SETGLOBAL. Bytecode can be fetched by calling CompileString(…) And CompileFile (…). , the FunctionProto of Lua Code is obtained, where the Code attribute is byte Code slice and the type is []uint32.

In the virtual machine implementation code, we can find a utility function that outputs the corresponding OpCode according to the bytecode.

// Get the OpCode of the corresponding instruction
func opGetOpCode(inst uint32) int {
	return int(inst >> 26)}Copy the code

With this utility function, we can check global variables.

package main

// ...

func CheckGlobal(proto *lua.FunctionProto) error {
	for _, code := range proto.Code {
		switch opGetOpCode(code) {
		case lua.OP_GETGLOBAL:
			return errors.New("not allow to access global")
		case lua.OP_SETGLOBAL:
			return errors.New("not allow to set global")}}// Check global variables for nested functions
	for _, nestedProto := range proto.FunctionPrototypes {
		iferr := CheckGlobal(nestedProto); err ! =nil {
			return err
		}
	}
	return nil
}

func TestCheckGetGlobal(t *testing.T) {
	l := lua.NewState()
	proto, _ := CompileString(`print(_G)`)
	if err := CheckGlobal(proto); err == nil {
		t.Fail()
	}
	l.Close()
}

func TestCheckSetGlobal(t *testing.T) {
	l := lua.NewState()
	proto, _ := CompileString(`_G = {}`)
	if err := CheckGlobal(proto); err == nil {
		t.Fail()
	}
	l.Close()
}
Copy the code

The module

In addition to variables that can be contaminated, imported Go modules can also be tampered with during runtime. Therefore, we need a mechanism to ensure that modules imported into the virtual machine are not tampered with, that is, imported objects are read-only.

After checking the relevant blogs, we can modify the __newindex method of Table to set the module to read-only mode.

package main

import (
	"fmt"
	"github.com/yuin/gopher-lua"
)

// Set the table to read-only
func SetReadOnly(l *lua.LState, table *lua.LTable) *lua.LUserData {
	ud := l.NewUserData()
	mt := l.NewTable()
	// Set the pointer of the field in the table to table
	l.SetField(mt, "__index", table)
	// Restrict update operations on tables
	l.SetField(mt, "__newindex", l.NewFunction(func(state *lua.LState) int {
		state.RaiseError("not allow to modify table")
		return 0
	}))
	ud.Metatable = mt
	return ud
}

func load(l *lua.LState) int {
	mod := l.SetFuncs(l.NewTable(), exports)
	l.SetField(mod, "name", lua.LString("gomodule"))
	// Set it to read-only
	l.Push(SetReadOnly(l, mod))
	return 1
}

var exports = map[string]lua.LGFunction{
	"goFunc": goFunc,
}

func goFunc(l *lua.LState) int {
	fmt.Println("golang")
	return 0
}

func main(a) {
	l := lua.NewState()
	l.PreloadModule("gomodule", load)
    // Try to modify the imported module
	if err := l.DoString(`local m = require("gomodule"); m.name = "hello world"`); err ! =nil {
		fmt.Println(err)
	}
	l.Close()
}

// <string>:1: not allow to modify table
Copy the code

Write in the last

Golang and Lua integration, broaden my horizons: the original static language and dynamic language can be so integrated, static language efficient operation, with dynamic language development efficiency, think of all excited (escape.

After searching the Internet for a long time, I found no technical sharing about Go-Lua, only found a slightly related article (continuous architecture optimization on JINGdong level 3 listing page — Golang + Lua (OpenResty) Best Practices), and in this article, Lua runs on C. Due to the lack of information and my (student party) lack of development experience, I can not evaluate the feasibility of the scheme in actual production. Therefore, this article can only be regarded as “idle”, haha.

The resources

Lua virtual machine
A No-Frills Introduction to Lua 5.1 VM Instructions
cocos2d-lua disable unexpected global variable
Set read-only tables in Lua
MetableEvents
Github.com/zhu327/glua…

Hello World

To compile

Vm instance pool

Module calls

Variable pollution

The global variable

The module

Write in the last

The resources

Related Posts

How does Java simulate true concurrent requests?

Redis accelerator –SDS

Either! Java time and space complexity algorithm, 3 minutes you can learn which?