utf8¶

A drop-in fast path for UTF-8 validation and rune counting, byte-identical to unicode/utf8. The bulk runs a SIMD kernel generated by go-asmgen (amd64 SSE/SSSE3 and AVX2); the short tail and the rune straddling the last SIMD block reuse the standard library, so results match exactly. Repository →

API¶

ok := utf8.Valid(p)              // same bool as unicode/utf8.Valid
ok := utf8.ValidString(s)        // same bool as unicode/utf8.ValidString
n  := utf8.RuneCount(p)          // same int  as unicode/utf8.RuneCount
n  := utf8.RuneCountInString(s)  // same int  as unicode/utf8.RuneCountInString

arch	kernel
amd64	SSE2/SSSE3 + SSE4.1 (16 B/block) and AVX2 (32 B/block), runtime-dispatched
ppc64le	VSX/AltiVec (16 B/block, POWER8 baseline) — qemu-validated; native perf pending
s390x	vector facility (16 B/block, z13 baseline, big-endian) — qemu-validated; native perf pending
arm64 / loong64 / riscv64	scalar (`unicode/utf8`) — NEON/LSX/RVV planned

The ppc64le and s390x kernels are 1:1 ports of the amd64 SSE path (no runtime dispatch, since VSX and the vector facility are baseline on POWER8+ and z13+): the PSHUFB nibble lookups become VPERM, the range compares VCMPGTSB/VCHB, and the rune-count popcount becomes VPOPCNTD (ppc64le) / VPOPCT+VSUMB (s390x). On ppc64le the source is loaded with LXVB16X; on big-endian s390x, VL already places the lowest memory address in lane 0, so the kernel is byte-identical to stdlib with no endianness fix-up.

Algorithm¶

`Valid`¶

Lemire & Keiser's Validating UTF-8 In Less Than One Instruction Per Byte: each byte is classified by its high nibble via PSHUFB nibble-lookups, then over the (prev, curr) byte pair (PALIGNR / VPERM2I128+VPALIGNR) a handful of checks run — continuation-length carry, the ED/F4 first-continuation maxima, the overlong-encoding minima, and byte > 0xF4 — each ORing into a running error accumulator. Blocks are valid iff the accumulator is all-zero (PTEST). Ported 1:1 from Lemire's reference intrinsics. The caller backs the SIMD/scalar split up to a rune boundary so a sequence straddling the last block is re-validated by unicode/utf8.

`RuneCount`¶

The number of runes in valid UTF-8 equals the number of non-continuation bytes — every rune has exactly one leading byte with (b & 0xC0) != 0x80. The kernel marks continuation bytes per block (PAND 0xC0 then PCMPEQB 0x80), extracts a bitmask (PMOVMSKB) and POPCNTs it; the count is blockBytes − continuations. That identity holds only for valid UTF-8, so the kernel is trusted only over a prefix the Valid validator confirms; if the prefix is invalid the whole input falls back to the scalar decoder, making the result byte-identical to unicode/utf8.RuneCount on all input. RuneCountInString defers to the stdlib to avoid an allocating []byte copy.

Performance¶

Valid throughput, ~1 MiB mixed-UTF-8 buffer, native amd64 (AMD EPYC 7763, AVX2, GOAMD64=v1, median of 6):

implementation	kind	MB/s	vs stdlib
`unicode/utf8.Valid` (stdlib)	scalar	312	1.00×
this package (`Valid`)	pure-Go SIMD (AVX2)	5940	19.07×
`stuartcarnie/go-simd`	pure-Go SIMD (SSE4/AVX2)	5738	18.42×

This package edges the competitor by ~3.5% and beats stdlib ~19× on this AVX2 runner. Both implement the same Lemire reference; the margin comes from this package's tighter go-asmgen-emitted loop and rune-boundary tail split. On a local AVX2 VM RuneCount measured ~4.6× stdlib; the native-runner CI fills the headline.

charlievieth/simdutf is a cgo wrapper around the C++ simdutf library — excluded from this pure-Go comparison.

Coverage¶

100% of the Go code (native amd64 drives the AVX2/SSE/POPCNT branches; native arm64 covers the generic fallback; ppc64le and s390x run under QEMU where VSX / the vector facility are baseline). The .s kernels are validated by differential tests against unicode/utf8 plus fuzzing — on a real AVX2 box for amd64 (~75 M executions, zero mismatches), and under qemu-user for the VSX and big-endian vector-facility kernels. BSD-3-Clause; the kernel derives from Lemire's reference (Apache-2.0).