XestiTokens

A rules-based lexical tokenizer.

Overview

The XestiTokens framework provides a highly configurable rules-based lexical tokenizer written in Swift. You can use it to tokenize a wide variety of text input.

“If you can tokenize it with lex or flex, you can tokenize it with XestiTokens.”

Tokenizing revolves around a small pipeline:

Stage	Type	Input → Output
Tokenize	Tokenizer	String → [Tokenizer.Token]
Match	TokenMatcher	Sequence<Tokenizer.Token> → parsed result

A Tokenizer is configured with an array of Tokenizer.Rule values, each pairing a regular expression with the disposition — save as a token, or skip — to apply when it matches. Rules can also carry start conditions, familiar to anyone who has used lex or flex, letting the same input character trigger different rules depending on the tokenizer’s current mode; and an action closure, letting a rule inspect or update contextual user information and decide its disposition dynamically at match time.

A TokenMatcher then walks the resulting token sequence one token at a time, offering simple lookahead and consume operations — nextMatches, readIfMatches, readMustMatch, failOnNext — that a hand-written recursive descent parser can build on directly.

Requirements

• iOS 16.0+ / macOS 14.0+
• Swift 6 language mode

Installation

Swift Package Manager

XestiTokens is distributed exclusively through the Swift Package Manager.

To add XestiTokens to a Swift package, add it to the dependencies in your Package.swift:

dependencies: [
    .package(url: "https://github.com/eBardX/XestiTokens.git",
             .upToNextMajor(from: "1.1.2"))
]

Then add XestiTokens to the dependencies of any target that uses it:

.target(name: "MyTarget",
        dependencies: [.product(name: "XestiTokens",
                                package: "XestiTokens")])

To add XestiTokens to an Xcode project, choose File ▸ Add Package Dependencies… and enter the repository URL:

https://github.com/eBardX/XestiTokens.git

Quick Start

Define the kinds of token you want to recognize, describe them as rules, then tokenize an input string:

import XestiTokens

let integerKind: Tokenizer.Token.Kind = "integer"
let opKind: Tokenizer.Token.Kind = "op"

let rules: [Tokenizer.Rule] = [Tokenizer.Rule(/[0-9]+/, integerKind),
                               Tokenizer.Rule(/[-+*\/]/, opKind),
                               Tokenizer.Rule(regex: /\s+/,
                                              disposition: .skip(nil))]
let tokenizer = Tokenizer(rules: rules)

// 1. Tokenize the input string into an array of tokens.
let tokens = try tokenizer.tokenize(input: "1 + 2 * 3")

tokens.map(\.kind)   // [‹integer›, ‹op›, ‹integer›, ‹op›, ‹integer›]
tokens.map(\.value)  // ["1", "+", "2", "*", "3"]

// 2. Walk the tokens with a matcher to build a parsed result.
var matcher = TokenMatcher(tokens)

while matcher.hasMore {
    let token = try matcher.readMustMatch([integerKind, opKind])

    print(token)
}

Whenever more than one rule matches at the current position, the rule matching the longest substring wins; ties are broken by the order in which the rules were provided to the tokenizer. Input that matches no rule fails tokenization with a descriptive error rather than being silently skipped.

Documentation

Every public declaration carries a DocC comment; Tokenizer, Tokenizer.Rule, and TokenMatcher in particular describe their behavior in detail.

Reference Documentation

Full reference documentation is available courtesy of DocC.

Credits

John Gary Pusey (ebardx@gmail.com)

License

XestiTokens is available under the MIT license.