Skip to content

streamvbyte

CI coverage

Pure-Go Stream VByte integer (uint32) compression with a SIMD encoder and decoder on all six of Go's 64-bit SIMD targets — amd64, arm64, ppc64le, s390x, riscv64, loong64 — and a portable scalar fallback everywhere else. No cgo, no GOEXPERIMENT, plain go build. This is the first Go port covering all six arches (others ship SIMD for amd64 only). Repository →

Stream VByte is the byte-oriented integer codec of Lemire, Kurz & Rupp (arXiv:1709.08990). A []uint32 is stored as a control stream of 2-bit lengths (one byte per four integers) followed by a data stream of each integer's significant little-endian bytes (1–4). The wire format is byte-for-byte identical to the reference C library (lemire/streamvbyte, standard 1234 variant) — C-interop verified bidirectionally.

API

Function Description
EncodedMaxLen(n int) int upper bound on the encoded size of n integers
Encode(dst []byte, src []uint32) int encode; returns bytes written
Decode(dst []uint32, src []byte, n int) int decode n integers; returns bytes read 
import "github.com/go-simd/streamvbyte"

src := []uint32{1, 280, 70000, 0xFFFFFFFF, 42}

buf := make([]byte, streamvbyte.EncodedMaxLen(len(src)))
n := streamvbyte.Encode(buf, src)
buf = buf[:n] // the compressed bytes

out := make([]uint32, len(src))
streamvbyte.Decode(out, buf, len(src)) // out == src

Decode(Encode(x)) round-trips exactly. The count n is not stored in the stream (store it yourself), matching the reference format.

Algorithm

Both directions are SIMD. Per group of four integers the decode kernel loads 16 data bytes, looks up the control byte's 16-byte shuffle mask in a 256-entry shuffle LUT, performs one vector permute that drops each integer's bytes into a zero-extended uint32 lane, and stores 16 result bytes; encode is the inverse (a mirror-image LUT packs four uint32 lanes down to their significant bytes). The Go wrapper runs each kernel only over groups with a full 16-byte lookahead and finishes the < 4 remainder (and any short-input tail) with the shared scalar path, so the wide load/store never over-reads. Length classification stays scalar, so encode's SIMD gain is more modest than decode's.

Per-arch kernels

  • amd64 — PSHUFB (SSSE3; runtime-detected via golang.org/x/sys/cpu, scalar fallback otherwise).
  • arm64 — VTBL (NEON, baseline).
  • riscv64 — vrgather.vv (RVV, VLEN >= 128).
  • loong64 — vshuf.b (LSX) with a zero companion register.
  • ppc64le — VPERM on POWER8+ (LXVB16X / STXVB16X byte-order-stable loads keep index == memory-offset; zero companion register).
  • s390x — VPERM on z13+. Big-endian: a decoded uint32 is stored MSB-first while the data stream stays LSB-first, so the shuffle table reverses each lane's bytes (permTableBE); the byte order is pinned by a position-dependent test.

Each arch ships both a decode and an encode kernel using the same shuffle primitive; both are generated by go-asmgen.

Other Go ports

thempatel/streamvbyte-simdgo, bmkessler/streamvbyte, mhr3/streamvbyte and nelz9999/stream-vbyte-go ship SIMD only for amd64 (scalar elsewhere). This package is, to our knowledge, the first to provide a SIMD encoder and decoder on all six of Go's 64-bit SIMD architectures. It complements bitpack.

Performance — honest

go test -bench . on a 4096-element mixed-width slice (16 KiB of uint32):

Target Decode (SIMD) Decode (scalar) Speedup
arm64 (Apple M-series, native) ~18.6 GB/s ~1.9 GB/s ~10×
amd64 (emulated VM*) ~0.73 GB/s ~0.32 GB/s ~2.3×
ppc64le (POWER9, VSX, native) ~3695 MB/s ~311 MB/s ~11.6×
riscv64 (SpacemiT X60, RVV 1.0, native) ~829 MB/s ~184 MB/s ~4.5×
loong64 (Loongson 3A5000, LSX, native) ~11.8× scalar (real silicon) — ~11.8×
s390x (IBM z15, VXE2, native, 2026-07-03) ~20× scalar (real silicon) — ~20×

* The amd64 figure was measured inside an emulated x86-64 VM (no hardware virtualization on the dev host), so it understates native silicon by a large margin; treat it as a correctness-grade lower bound. ppc64le, riscv64 and loong64 are measured on real silicon (GCC Compile Farm, Go 1.26.4, 2026-06-26), each running full SIMD encode and decode; the X60 is a low-power in-order RVV core so its absolute MB/s are conservative. s390x is now measured on real IBM z15 (VXE2) (2026-07-03, -count=6): SIMD decode ~20× the scalar baseline (the largest decode win in this table), encode ~1.4× (only the byte compaction vectorises).

Coverage

Round-trip table tests plus FuzzRoundTrip (Decode(Encode(x)) == x) run with 100% statement coverage on every architecture: amd64 and arm64 natively, and ppc64le, s390x, riscv64, loong64 under QEMU. Format interop with the reference C library was verified bidirectionally and byte-for-byte. BSD-3-Clause.