Testing Plugins

Did you install the xtp CLI?

The xtp CLI is required to run tests & host simulations, so please see these instructions to be sure you have the latest CLI installed.

curl https://static.dylibso.com/cli/install.sh -s | bash

Once you have xtp, you can now run unit tests that call your plugins and assert various things about them, such as outputs for given inputs, plugin state, or timing the performance of plugin function calls. To do this, we provide test harness libraries to create these unit tests for you to run with xtp.

To begin testing your plugins, use any of the following frameworks to write tests:

Within each of these repositories, you will find detailed instructions on how to write, compile, and run tests. It's important to note, that while these libraries are available in JS/TS, Rust, Go, and Zig, you can use them to test plugins which were written in any of the Extism PDK languages.

Want another language supported? Reach out to us on the #xtp channel in the Extism Discord to let us know!

Testing a Plugin with Imports

When your plugins need to make calls to Imports ("host functions"), the obvious question introduced is "who is implementing the imports?". Since the host application isn't running the plugin, these import functions must be mocked.

To solve for this, we provide an optional --mock-host argument to the xtp CLI when you execute tests. With this argument, you pass a path on disk or URL to separate module that supplies these import functions to your plugin being tested.

Here's an end-to-end example, split into 3 different Wasm projects:

a KV datastore (mock), exporting KvRead and KvWriteAppend functions (these act in place of real host functions and are imported by the plugin)
the plugin (plugin), which interacts with the KV datastore via host function imports
an XTP test plugin (test), which verifies the behavior of the plugin

all of the following code can be found in full here: testing-xtp-plugins

Mock Host Functions
Plugin using Host Functions
XTP Test Plugin

The mock project is a simple key-value store that exports KvRead and KvWriteAppend functions. These functions are used by the plugin project to read and write key-value pairs. When this project is compiled, it will produce a Wasm file that can be used as a host for the plugin project, passed as the --mock-host argument to the xtp CLI.

mock/host.go
package main

import (
	"encoding/json"
  // to simulate Host Functions, use the PDK to manage host/guest memory.
	pdk "github.com/extism/go-pdk"
)

// this is our in-memory KV store (e.g. a mock database)
var kv map[string][]string = make(map[string][]string)

// This export will be made available to the plugin as an import function.
// The offset param is the location in memory to the 'key' string
// The return offset is the location in memory to the 'value' string
//
//go:export KvRead
func KvRead(offset uint64) uint64 {
	// find the memory block that contains the key, read the bytes, and look up the
	// corresponding value in the KV store
	keyMem := pdk.FindMemory(offset)
	k := string(keyMem.ReadBytes())
	// if the entry is not found, return non-zero code
	v, ok := kv[k]
	if !ok {
		v = make([]string, 0)
	}

	// allocate a new memory block for the value, storing the string in memory
	valMem, _ := pdk.AllocateJSON(v)

	// return the memory offset
	return valMem.Offset()
}

// A struct to deserialize the JSON input behind the memory address passed to `KvWriteAppend`
// NOTE: you probably have these types defined somewhere else and can import them here
type WriteParams struct {
	Key   string `json:"key"`
	Value string `json:"value"`
}

// A struct to serialize the return from `KvWriteAppend` to store in memory.
// Non-zero `Code` value indicates an error.
// NOTE: you probably have these types defined somewhere else and can import them here
type WriteReturns struct {
	Message string `json:"message"`
	Code    int8   `json:"code"`
}

// This export will be made available to the plugin as an import function
// The offset param is the location in memory to the JSON-serialized 'WriteParams'
// The return offset is the location in memory to the JSON-serialized 'WriteReturns'
//
//go:export KvWriteAppend
func KvWriteAppend(offset uint64) uint64 {
	// find the memory block that contains the WriteParams,
	// read its bytes and deserialize into WriteParams
	paramsMem := pdk.FindMemory(offset)
	var params WriteParams
	err := json.Unmarshal(paramsMem.ReadBytes(), &params)
	if err != nil {
		mem, _ := pdk.AllocateJSON(WriteReturns{
			Message: err.Error(),
			Code:    2,
		})
		return mem.Offset()
	}

	// store the key-value pair in the KV store
	kv[params.Key] = append(kv[params.Key], params.Value)

	// return the WriteReturns offset for the caller to read
	mem, _ := pdk.AllocateJSON(WriteReturns{
		Message: "",
		Code:    0,
	})
	return mem.Offset()
}

// for now, an empty main function is required by the compiler
func main() {}

The plugin project is a plugin that interacts with the KV datastore via host function imports. This plugin reads and writes key-value pairs and stores them in the plugin's state. We will verify this behavior using the test project in the next tab.

const std = @import("std");
const plugin_allocator = std.heap.wasm_allocator;

const schema = @import("schema.zig");
const Host = schema.Host;

/// takes a LogRequest input to do some inspection and
/// aggregate some stats to be returned
/// It takes LogRequest as input (the data provided by a log event)
/// And returns LogStats (an object indicating the log handling status)
pub fn handleLogEvent(input: schema.LogRequest) ![]const u8 {
    // write the log into the KV store
    const write = schema.WriteParams{ .key = input.source, .value = try input.timestamp.toRfc3339(plugin_allocator) };
    const res = try Host.KvWriteAppend(write);
    if (res.code != 0) {
        return error.WriteAppendFailed;
    }

    // read the data from the KV for each of the source options
    // aggregate the data into the log stats, where keys are sources and values
    // are the count of each log entries
    // e.g. { 'cli': 2, 'api': 100, ... }
    var stats_map = std.StringHashMap(u32).init(plugin_allocator);
    defer stats_map.deinit();
    const sources = [4]schema.SourceSystem{ .webapp, .postgres, .api, .cli };
    for (sources) |source| {
        const key = @tagName(source);
        const logs = try Host.KvRead(key);
        try stats_map.put(key, @intCast(logs.len));
    }

    return try stringifyHashMap(plugin_allocator, stats_map);
}

fn stringifyHashMap(allocator: std.mem.Allocator, map: std.StringHashMap(u32)) ![]const u8 {
    const T = u32; // Define the type of values stored in the map
    const JsonArrayHashMap = std.json.ArrayHashMap(T);
    var json_map = JsonArrayHashMap{
        .map = try std.StringArrayHashMapUnmanaged(T).init(allocator, &[_][]const u8{}, &[_]T{}),
    };
    defer json_map.deinit(allocator);

    // Iterate through the StringHashMap and add entries to the ArrayHashMap
    var it = map.iterator();
    while (it.next()) |entry| {
        try json_map.map.put(allocator, entry.key_ptr.*, entry.value_ptr.*);
    }

    return try std.json.stringifyAlloc(allocator, json_map, .{});
}

The test project is an XTP test plugin that verifies the behavior of the plugin project. When we run this test, we will pass the mock project as the host, so that the plugin project can interact with the KV store we simulate in the mock project.

import { Test } from "@dylibso/xtp-test";

// expected output format from the plugin call:
// {"api":0,"cli":0,"webapp":1,"postgres":0}
interface SourceStats {
  api: number;
  cli: number;
  webapp: number;
  postgres: number;
}

export function test() {
  const mockInput = Test.mockInputString();
  Test.group("Verify KV store integration", () => {
    for (let i = 0; i < 4; i++) {
      const output = Test.callString("handleLogEvent", mockInput);
      const stats: SourceStats = JSON.parse(output);

      Test.assertEqual(
        `webapp source increments to ${i + 1} based on iteration`,
        i + 1,
        stats.webapp,
      );
    }
  });
}

After compiling each of these Go projects to Wasm, you can run the test using the xtp CLI, and pass the --mock-host argument to specify the host Wasm file, to stitch all the pieces together:

xtp plugin test plugin/zig-out/bin/plugin.wasm \
  --with test/dist/plugin.wasm \
  --mock-host mock/host.wasm \
  --mock-input-file mock-input.json

You should see output like this:

🔨 Building integrated host mock + plugin test
🧪 Testing zig-out/bin/plugin.wasm (integrated host mock + plugin test)

📦 Group: Verify KV store integration
PASS ...... webapp source increments to 1 based on iteration
PASS ...... webapp source increments to 2 based on iteration
PASS ...... webapp source increments to 3 based on iteration
PASS ...... webapp source increments to 4 based on iteration

4/4 tests passed (completed in 4.958ms)

all tests completed in 4.977ms

Mocking input data to plugin test calls

Configure your test with dynamic input provided by a xtp CLI parameter or xtp.toml file. Read runtime-provided input that mocks the actual input when a plugin is called:

Note: this is available in each of the JavaScript/TypeScript, Rust, Go, & Zig test harness libraries.

//go:export test
func test() int32 {
  // use the MockInputBytes() function to read the input data provided by the test runner
  // (there are variations of this function in other xtp-test libraries)
  notEmpty := xtptest.CallString("count_vowels", xtptest.MockInputBytes())
  xtptest.AssertNe("with mock, not empty", notEmpty, "")
  // ...
}

Providing mock input data

There are two ways to provide input data to the plugin test calls:

xtp CLI, using --mock-input-data or --mock-input-file
xtp.toml file

Using the `xtp` CLI

CLI supports args --mock-input-data and --mock-input-file to pass text or load a file.

For example:

xtp plugin test plugin.wasm --with test.wasm --mock-input-data "this is my mock input data"
# or a path to a file for --mock-input-file

Using `xtp.toml`

xtp.toml supports syntax such as:

# path or url locating the wasm plugin to test
bin = "https://raw.githubusercontent.com/extism/extism/main/wasm/code.wasm"

[[test]]
# label this test something recognizable to see in CLI output
name = "basic"
# build the test wasm module, is run before the test
build = "cd examples/countvowels && tinygo build -o test.wasm -target wasi test.go"
# the wasm module to use as the test
with = "examples/countvowels/test.wasm"
# provide mock input data to the plugin test call, returned to a 'MockInput' type of function call
mock_input = { data = "this is my mock input data" }

[[test]]
name = "basic - file input"
build = "cd examples/countvowels && tinygo build -o test.wasm -target wasi test.go"
with = "examples/countvowels/test.wasm"
# load mock input data from a file instead of inline
mock_input = { file = "examples/countvowels/test.go" }

(see examples used in examples/countvowels)

Overriding `xtp.toml` location

When running xtp plugin test, if xtp.toml is present in the current directory, it will be used to configure the test. The location of the file can be overridden using --path:

xtp plugin test --path tests/countvowels

Usage in tests

The various XTP libraries provide convenient functions to dynamically read input from the host, mocked out by the supported options above.

Testing a Plugin with Imports​

Mocking input data to plugin test calls​

Providing mock input data​

Using the xtp CLI​

Using xtp.toml​

Overriding xtp.toml location​

Usage in tests​