Class Lexer

The lexer is responsible for turning a string of characters into a stream of tokens. The tokens are then used by the parser to build an abstract syntax tree.

The lexer handles parsing of string literals, digraphs (e.g. <=), and numbers. It also handles detecting keywords and identifiers.

Constructors

constructor

Properties

_currLocCache charOffset codeFile colOffset currentExtraTokens errorCollector lineOffset start startWithWhitespace tokens

Methods

addToken advance consumeFileNameInChevrons consumeIdentifierOrKeyword consumeNumberLiteral consumeNumberOrIdentifierOrKeyword consumeStringLiteral getLoc isAtEnd match peek peekNext peekRegex scan scanToken

Constructors

constructor

Properties

Protected_currLocCache

_currLocCache: null | CodeLocation = null

ProtectedcharOffset

charOffset: number = 0

codeFile

codeFile: CodeFile

ProtectedcolOffset

colOffset: number = 0

ProtectedcurrentExtraTokens

currentExtraTokens: ExtraToken[] = []

errorCollector

errorCollector: ErrorCollector

ProtectedlineOffset

lineOffset: number = 0

Protectedstart

start: CodeLocation

ProtectedstartWithWhitespace

startWithWhitespace: CodeLocation

tokens

tokens: Token[] = []

Methods

ProtectedaddToken

  • addToken<TValue>(tokenType, value?): void

  • Adds a token to the token list. If a value is provieded a LiteralToken is pushed.

    Additionally it handles clearing and attaching the extra tokens.

    Type Parameters

    • TValue = any

    Parameters

    Returns void

Protectedadvance

ProtectedconsumeFileNameInChevrons

ProtectedconsumeIdentifierOrKeyword

ProtectedconsumeNumberLiteral

ProtectedconsumeNumberOrIdentifierOrKeyword

  • consumeNumberOrIdentifierOrKeyword(): void

  • Returns void

ProtectedconsumeStringLiteral

ProtectedgetLoc

ProtectedisAtEnd

Protectedmatch

  • match(expected): boolean

  • Parameters

    • expected: string

    Returns boolean

Protectedpeek

ProtectedpeekNext

ProtectedpeekRegex

  • peekRegex(regex): string

  • Parameters

    • regex: RegExp

    Returns string

scan

  • scan(): Token[]

  • Scans the whole CodeFile and splits it into tokens.

    Returns Token[]

    Throws

    LexingError

ProtectedscanToken

Settings

Member Visibility

On This Page

Constructors
Properties
Methods