Text encoding

STRING remains valid UTF-8. These built-in functions explicitly convert between STRING and LIST OF BYTE. No import, native codec library, locale setting or runtime mapping file is required. Existing EncodeUtf8/DecodeUtf8 keep their contracts; file/socket/database APIs do not silently choose a legacy encoding.

bytes=EncodeText("café",TextEncoding.Windows1252)
PRINT DecodeText(bytes,TextEncoding.Windows1252)

Read encoded files with ReadBinaryFile, decode their bytes, then encode text before writing bytes. Conversion adds no NUL terminator.

Profiles

Both GB versions represent every Unicode scalar, including private-use, unassigned and noncharacter values. Their byte mappings differ. The 2022 profile preserves strict scalar round trips; ICU/web fallback aliases are separate behavior. See exact profiles , pinned 2022 derivation and Unicode’s mapping-change account . No detection, normalization, transliteration, newline conversion or best-fit fallback is implicit. CP932/CP950/HKSCS/CP949, ISO-2022 and UTF-7 are outside this API.

Functions

Encoding is required. Defaults: errors=TextErrorPolicy.Strict, decode bom=TextDecodeBom.Preserve, encode bom=TextEncodeBom.Omit, maxOutputBytes=67108864 (64 MiB per call/chunk). The limit accepts 0…1 GiB and counts actual output bytes, including emitted BOMs. It does not cap the stream.

EncodedText.Bytes and DecodedText.Text hold the output; .Info contains InputBytes, OutputBytes, Replacements, FirstReplacementOffset (-1 if none), and Finished. Streaming counters are cumulative and use checked INTEGER byte offsets.

result=EncodeTextWithInfo("a€😀",TextEncoding.ASCII,TextErrorPolicy.Replace)
PRINT DecodeUtf8(result.Bytes)
PRINT result.Info.Replacements

Replacement is opt-in: decode uses U+FFFD per malformed span; legacy encode uses ? once per unrepresentable scalar. Strict encoding rejects lossy aliases, including Shift-JIS yen/overline aliases. Unicode encoding of a valid STRING is lossless. Duplicate byte spellings may canonicalize when re-encoded.

Errors and BOMs

TextEncodingError fields: Encoding, Operation, Offset, Length, Reason, Message. TextOperation is Decode or Encode. Reasons: InvalidSequence, Unrepresentable, MissingBom, UnexpectedBom, InvalidOption, OutputLimit, AlreadyFinished. Offset/Length identify external source bytes on decode and input UTF-8 bytes on encode, not BASIL scalar positions. Offsets remain absolute across stream chunks. Constructors declare TextEncodingError. Conversion/chunk/finish functions also retain IndexError, NumericError and StringError contracts from checked helpers; they do not mislabel implementation failures as malformed data.

TRY
    text=DecodeText([BYTE(255)],TextEncoding.UTF8)
CATCH err AS TextEncodingError
    PRINT err.Offset
    PRINT err.Length
END TRY

Decode Preserve retains U+FEFF. Consume removes a matching initial BOM when present; Require demands it. Consume/Require reject opposite byte order even in replacement mode; they never change the chosen encoding. Under UTF16LE, FF FE 00 00 is BOM plus NUL; under UTF32LE it is just BOM. Encode Emit writes one initial BOM, including for empty text, and retains existing U+FEFF content. Consume/Require/Emit reject non-Unicode profiles.

Streaming and ownership

decoder=NewTextDecoder(TextEncoding.UTF16LE)
PRINT DecodeTextChunk(BYREF decoder,[BYTE(65)])
PRINT DecodeTextChunk(BYREF decoder,[BYTE(0)],final=TRUE)
PRINT DecoderInfo(decoder).Finished

A decoder retains at most seven pending bytes for split BOMs, surrogate pairs, regional sequences or Hangul composition. Encoders take complete STRING chunks and emit a BOM at most once. final=TRUE finishes; explicit Finish supplies empty input. Reusing a finished converter, including a second Finish, fails.

Converters are copyable opaque values: copies fork independently. Returned buffers and metadata snapshots are independent. BYREF chunk operations exclusively borrow the state. Every recoverable chunk failure leaves it unchanged and returns no partial chunk. Earlier chunks/external writes are not rolled back. Allocation failure follows the language’s fatal allocation policy. Ordinary cleanup and compiler scheduling/cancellation apply; no codec thread, hidden shared mutable state, GC or reference counting is introduced.