j
jaipkg.dev
packages / library / rjyv

rjyv

7010f40library

A high-performance, zero-copy serializer inspired by rkyv. Maximum compile-time optimization.

MIT · updated 9 months ago

RJYV: Zero-Copy Serializer for Jai

A high-performance, zero-copy serializer inspired by rkyv. Maximum compile-time optimization.

⚠️ NOTE: This is untested code. I think it's fine like this, but use at your own risk.

Features

  • ⚡ Zero-copy deserialization - Read data directly from buffer without parsing
  • 🚀 Compile-time optimized - Field layouts, offsets, and sizes computed at compile-time
  • 🎯 @Skip support - Mark fields with @Skip to exclude from serialization (zero bytes in output)
  • 📦 Memory-mappable - Save to disk and mmap for instant access
  • 🔒 Type-safe - Compile-time field validation

Quick Start:

#import,file "rjyv.jai";

Person :: struct {
    name     : string;
    age      : s32;
    password : string; @Skip  // Won't be serialized
    email    : string;
}

main :: () {
    person: Person;
    person.name = "Alice";
    person.age = 30;
    person.password = "secret123";
    person.email = "alice@example.com";

    // Serialize
    data := serialize(person);

    // Deserialize
    restored := deserialize(data, Person);
    print("%\n", restored.name);  // "Alice"
    print("%\n", restored.age);   // 30
    print("%\n", restored.password); // "" (was skipped)
}

API

Serialization

serialize :: (value: $T) -> Serialized_Data;

Converts a struct into binary format. Returns Serialized_Data containing the buffer and root offset.

Deserialization

deserialize :: (data: Serialized_Data, $T: Type) -> T;

Converts binary data back into a struct. Allocates memory for strings/arrays.

Zero-Copy Access

For maximum performance, read fields directly from the buffer without deserializing:

// POD types (int, float, bool)
age   := get_field(s32, data, Person, "age");
score := get_field(float32, data, Person, "score");

// Strings (zero-copy, points into buffer)
name  := get_field(string, data, Person, "name");

// Arrays (zero-copy, points into buffer)
items := get_field([]int, data, Person, "items");

Zero-copy strings/arrays are read-only views into the buffer.

Supported Types

  • Integers (int, s8, s16, s32, s64, u8, u16, u32, u64)
  • Floats (float, float32, float64)
  • Booleans
  • Strings
  • Arrays ([]T)
  • Structs (nested structs work)

@Skip Note

Mark any field with @Skip to exclude it from serialization:

Person :: struct {
    public_data : string;
    secret_key  : string; @Skip      // Not serialized
    internal_id : s64;    @Skip      // Not serialized
}

Skipped fields:

  • Take zero bytes in the serialized output
  • Get default values when deserialized (empty string, 0, etc.)
  • Are completely removed from the binary format at compile-time

Binary Format

The serializer uses relative offsets instead of pointers, making the data:

  • Relocatable - Can be moved in memory
  • Persistent - Can be saved to disk
  • Memory-mappable - Can be loaded with mmap for instant access

Internal Format

  • POD types: Raw bytes (native endianness)
  • Strings: {offset: s64, length: s64} + data
  • Arrays: {offset: s64, length: s64} + elements
  • Structs: Fields laid out sequentially (skipped fields omitted)

Performance

All of these are computed at compile-time:

  • Field offsets in serialized format
  • Struct size calculations
  • Field name validation
  • Skip detection
  • Code generation per struct type

Runtime overhead: Just memory copies and pointer arithmetic.

File I/O Example

// Serialize and save
data := serialize(person);
write_entire_file("person.bin", data.buffer.data, data.buffer.count);

// Load and deserialize
file_data := read_entire_file("person.bin");
loaded_data: Serialized_Data;
loaded_data.buffer.count = file_data.count;
loaded_data.buffer.data = file_data.data;

person := deserialize(loaded_data, Person);

Limitations

  • Only structs are fully supported (not bare integers/strings at top level)
  • No pointer serialization (use indices or flatten data)
  • No cyclic references
  • Native endianness only (no cross-platform binary compatibility yet)
  • Untested - This is theoretical code

Compile-Time Guarantees

// Compile error: Field doesn't exist
ptr := get_field(data, Person, "nonexistent");

// Compile error: Type mismatch
data := serialize(42);  // Not a struct

Future Ideas

  • Endianness control (big/little)
  • Versioning support
  • Compression
  • Custom allocators
  • Validation/checksums

Again: This is untested. I think the design is solid, but YMMV. Test thoroughly before using in production!