Skip to content

int8dot

CI coverage

SIMD-accelerated integer dot products over byte slices — the multiply-accumulate at the heart of quantized machine-learning inference, where embeddings and weights are stored as 8-bit integers to cut memory bandwidth and exploit per-byte vector throughput. The signed int8, unsigned uint8, and mixed uint8×int8 contractions, each summed exactly into a 32-bit accumulator. It is the integer counterpart to floats. Repository →

Pure Go: CGO_ENABLED=0, stable Go, no GOEXPERIMENT. The hot MAC loops are real integer SIMD on five of the six 64-bit Go targets (see below), assembly generated by go-asmgen.

import "github.com/go-simd/int8dot"

d  := int8dot.Dot(a, b)        // []int8  · []int8   -> int32
du := int8dot.DotUint8(a, b)   // []uint8 · []uint8  -> uint32
ds := int8dot.DotU8S8(a, b)    // []uint8 · []int8   -> int32  (quantized GEMM layout)

API

function operands result
Dot(a, b []int8) int8 × int8 int32
DotUint8(a, b []uint8) uint8 × uint8 uint32
DotU8S8(a []uint8, b []int8) uint8 × int8 int32

All three require len(a) == len(b) and panic otherwise. INT8 quantization schemes differ in the sign of each operand: symmetric weights are int8, ReLU-style activations quantize to uint8, and the uint8 × int8 layout (DotU8S8) is what hardware INT8 GEMM paths (x86 VNNI, ARM dot-product) are built around — so it gets its own entry point.

Saturation safety

Each byte×byte product fits in 16 bits and the sum accumulates in a 32-bit integer, so there is no clamping: the result is exact. With worst-case magnitudes int32 does not overflow until ~131k elements — embedding vectors (256–4096 dims of small quantized values) stay comfortably inside. Every SIMD kernel is required to be bit-for-bit identical to the scalar reference on every architecture (differential tests + FuzzDot).

Per-arch kernels

arch int8 / uint8 kernel u8×s8 kernel notes
amd64 VPMOVSXBW/VPMOVZXBW + VPMADDWD (AVX2) AVX2 runtime-gated on AVX2; VNNI deferred
arm64 VSMULL/VUMULL widening (NEON, Go 1.27+) VUXTL+VSXTL+VSMULL (NEON, Go 1.27+) scalar on stable Go ≤ 1.26
ppc64le VMULESB/VMULOSB even/odd widening (VMX) scalar no mixed-sign byte multiply on Power
s390x VMEB/VMOB even/odd widening (vector facility) scalar big-endian; no mixed-sign byte multiply
riscv64 VWMULVV + VWREDSUMVS (RVV) VWMULSUVV (RVV) runtime-gated on RVV
loong64 VMULWEVHB/VMULWODHB even/odd widening (LSX) scalar no mixed-sign byte multiply on LoongArch

Integer multiply-accumulate is associative and exact, so — unlike the floating-point sibling — there is no lane-order or ULP subtlety; each kernel widens byte products to 16 bits, accumulates into several 32-bit lanes (for ILP), and horizontally sums at the end, with a scalar tail for the remainder.

Honest architecture notes

  • arm64 — NEON SIMD on Go 1.27+, scalar on stable Go ≤ 1.26. All three dot products have a NEON widening multiply-accumulate kernel, but the integer-multiply mnemonics (VSMULL/VUMULL) are only exposed by the Go arm64 assembler from Go 1.27 (1.26 has only VPMULL, polynomial multiply, useless for integer arithmetic), so the kernel is //go:build arm64 && go1.27 and stable Go ≤ 1.26 falls back to the scalar reference. DotU8S8 is full SIMD here too (unlike ppc64le/s390x/loong64). The ARMv8.2 dot-product instructions (SDOT/UDOT/USDOT) would be the one-instruction kernel, but as of the Go 1.27 dev tree the assembler exposes them only in their SVE form — and SVE is not implemented by Apple Silicon, where this package is natively tested.
  • amd64 — AVX2, not VNNI (yet). VPDPBUSD (AVX-512-VNNI) is the ideal uint8×int8 instruction, but qemu's TCG implements no AVX-512, so neither the differential tests nor the 100% gate can exercise it. Rather than ship an unvalidated kernel, amd64 uses the AVX2 path (validated bit-exact on a native x86-64 VM). A VPDPBUSD fast path is a clean follow-up once a VNNI-capable, test-covered runner exists.
  • ppc64le / s390x / loong64 — DotU8S8 is scalar. None of these ISAs has a mixed unsigned×signed byte multiply primitive; Dot and DotUint8 are full SIMD on all three.

Performance — honest

Measured ratio of the AVX2 kernel against the scalar reference (go test -bench, x86-64 under qemu TCG — absolute throughput is depressed by emulation, but the ratio is representative; native silicon is ~4×):

dim scalar AVX2 speedup
256 254 MB/s 836 MB/s ~3.3×
768 255 MB/s 965 MB/s ~3.8×
4096 257 MB/s 1034 MB/s ~4.0×

arm64 NEON (gotip / Go 1.27, native Apple Silicon): ~2.4× the scalar reference at large sizes (~1.8× at dim 64).

Measured on real silicon (GCC Compile Farm, Go 1.26.4, 2026-06-26)

  • riscv64 — real SpacemiT X60 (RVV 1.0): RVV INT8 MAC (VWMULVV + VWREDSUMVS) runs Dot at dim 4096 at ~1557 vs ~172 MB/s scalar — ~9.1×, the biggest RVV win in the whole go-simd suite — the ideal RVV shape (a long, arithmetic-bound widening multiply-accumulate reduction).
  • loong64 — real Loongson 3A5000 (LSX): the LSX MAC kernels run at ~8.9× the scalar reference overall, with DotUint8 the standout at ~13.8× (DotU8S8 is scalar on loong64 — no mixed-sign byte multiply).
  • ppc64le — real POWER9 (VSX): VSX INT8 MAC runs Dot at dim 4096 at ~3055 vs ~744 MB/s scalar — ~4.1× native.
  • s390x is now measured on real IBM z15 (VXE2) (2026-07-03, -count=6): the vector-facility INT8 MAC kernels (VMEB/VMOB) run Dot at ~9.1× and DotUint8 at ~10× the scalar reference. DotU8S8 is scalar on s390x (no mixed unsigned×signed byte multiply), so it tracks the scalar baseline (~1.0×, no vector win).

Coverage

Dot/DotUint8/DotU8S8 are checked against a straight-line scalar loop by table tests, extreme-value tests, and FuzzDot, with 100% statement coverage gated on every architecture: amd64 and arm64 natively, riscv64/loong64/ppc64le/ s390x under QEMU. Dispatch branches (AVX2 / RVV / scalar) are exercised directly. BSD-3-Clause.