Unicode and runes

Dusk gained Unicode in 0.5.2: a rune type for one Unicode scalar value, the \u{...} escape, strict UTF-8 checking of string literals, and std.unicode, a pure dusk decode and encode layer. The string type itself did not change. A string is still the NUL terminated byte view it always was, UTF-8 by convention, and s[i] still reads one byte. This guide walks the pieces in the order you reach for them. The full signatures live under std.unicode, and the rune and char primitives sit on the type system page.

The rune type

A rune is a 4 byte primitive holding one Unicode scalar value, the codepoint alone with no encoding attached. Where a char is one byte and stands for a single ASCII byte inside a string, a rune is wide enough to name any character in Unicode, 中, 😀, or a plain ASCII letter alike.

You write a rune literal with r'...':

a := r'a'              // an ASCII scalar
zh := r'中'             // a CJK scalar
grin := r'\u{1F600}'   // a scalar named by its hex codepoint

Every ordinary character escape works inside r'...', plus the \u{...} escape covered below. sizeof(rune) is 4, and a rune crosses the foreign function boundary as a C i32. No user defined type may be named rune; the name is reserved for the primitive.

A rune and an integer convert both ways under the same rule a char and an integer already follow. A rune widens to any integer width, and a wide integer narrows back to a rune with the same silent truncation char uses. A rune and a char do not mix in either direction, since a byte and a scalar are different things even though both ride an integer register. Assigning a char to a rune, or a rune to a char, is rejected: type annotation that does not match its value at an annotation, and argument N has the wrong type at a call.

A rune carries no arithmetic of its own. To compute on a codepoint, bind the rune to an int64, do the arithmetic there, and assign the result back to a rune:

runes.dusk

func main() -> int32 {
    x := r'中'
    println(x)             // 20013
    v: int64 = x           // a rune widens to an int, like a char
    y: rune = v + 1        // compute on the int, then narrow back
    println(y)             // 20014
    return 0
}

println on a rune prints the codepoint number, not a glyph, so println(r'中') prints 20013. That is a deliberate asymmetry with char. Since 1.1.0 a char, a char[N], and a char[] print as their text bytes, so a byte read out of a string prints its glyph, while a rune still prints its codepoint number. A char is one byte of a string’s text and a rune is a 4 byte scalar, and only encode_rune turns a scalar into displayed text. Printing the character itself is an encoding step, covered under Encoding and building below. Comparison is the one operator a rune takes directly: two runes compare, and a rune and an integer literal compare, the same as char. A match pattern does not bind a rune literal (nor a char or int literal), so compare a rune scrutinee with an if chain rather than a match arm.

The \u{...} escape

The \u{...} escape names a Unicode scalar by its hex codepoint, 1 to 6 hex digits between the braces:

tab := r'\u{9}'
zh := r'\u{4E2D}'                    // 中
grin := r'\u{1F600}'                 // 😀
line: string = "first\u{A}second"    // a newline inside a string

It is legal inside a string literal and a rune literal, where it may name any scalar up to the Unicode maximum 0x10FFFF, excluding the surrogate range 0xD800..0xDFFF. Inside a char literal it is legal only for a value that fits one byte, 0x7F and under.

Five ways to get it wrong are each their own build error:

r'\u{}'         // \u escape needs 1 to 6 hex digits
r'\u{1F600'     // unterminated \u escape; expected '}'
r'\u{D800}'     // \u escape is a surrogate code point, not a scalar value
r'\u{110000}'   // \u escape is above 0x10FFFF, the Unicode maximum
'\u{100}'       // a char is one byte; this escape does not fit, use a rune literal or a string

String literals are strict UTF-8

This one is a behavior change, so it is worth calling out. A string literal is now validated as UTF-8 at compile time and rejected if it is malformed: string literal is not valid UTF-8. Earlier releases lexed a bad byte sequence silently, replacing it with the U+FFFD replacement character and moving on. Since 0.5.2 the compiler stops instead.

If a program relied on the old silent replacement, fix the literal’s encoding. If the bytes are intentional, do not spell them in a literal at all. Build the string at runtime with encode_rune or sb_push_rune from std.unicode, shown below.

A string is still a byte view

A string’s representation never changed to add Unicode. It is the same NUL terminated byte buffer it always was, UTF-8 by convention rather than by any different layout. That has one consequence worth keeping straight: s[i] reads one byte, not one character and not one scalar. There is no new indexing form that returns a scalar.

To walk a string scalar by scalar, decode. std.unicode’s decode_rune(s, i) reads the bytes starting at i and returns the scalar there paired with its width in bytes, so you step forward by the width you get back:

scalars.dusk

@paradigm procedural
@import std.unicode

func main() -> int32 {
    s: string = "a中b"
    mut i: int64 = 0
    while s[i] != 0 {
        r, w := decode_rune(s, i)
        println(r)          // 97, then 20013, then 98
        i = i + w
    }
    return 0
}

decode_rune is total. The NUL terminator decodes to (0, 0), and any malformed byte resyncs to exactly (0xFFFD, 1), so the walk always moves forward one byte at a time and never stalls. Its one precondition is that i stays in [0, str_len(s)], which a walk that steps by the width it gets back never leaves.

Encoding and building

Going the other way, from a scalar to bytes, is encode_rune(r, buf). It writes a rune’s 1 to 4 UTF-8 bytes into a caller sized buffer and returns the count; rune_len(r) reports that same width without writing. The buffer is a *raw char, so size it to at least 4:

buf: *raw char = alloc_bytes(4)
n := encode_rune(r'中', buf)   // writes 3 bytes, returns 3

The friendlier path for building display text is sb_push_rune, which appends a scalar’s encoded bytes straight onto a StringBuilder:

@import std.string
@import std.unicode

g: *StringBuilder = alloc(sb_new())
sb_push_rune(g, r'中')
sb_push_rune(g, r'\u{1F600}')
println(sb_cstr(g))            // 中😀
sb_free(g)
free(g)

Two more helpers round out the module. rune_count(s) walks a whole string and counts scalars, and utf8_valid(s) reports whether a string is strict, well formed UTF-8. Both share decode_rune’s resync, so their answers never drift from what a decode walk sees. The std.unicode page has the full signatures.

What std.unicode does not do

std.unicode is a decode and encode layer, nothing more. Case folding, normalization, and grapheme clustering sit outside it and are not part of this release. A rune is one scalar value, so an emoji built from several scalars joined together counts as several runes, and rune_count counts scalars rather than the glyphs a terminal renders.

Where to go next

• std.unicode: the full signature and behavior of every function here.
• Type system: the rune and char primitives and the string type they sit beside.
• std.string: StringBuilder and the read only string helpers sb_push_rune builds on.