From Source Text to Execution: Designing a Small Language Runtime

A programming language begins as text, but text cannot execute directly.

Before a statement can produce a result, a language implementation must recognize its symbols, determine its structure, validate its meaning, translate it into an executable representation, and manage the runtime state required to evaluate it.

Consider a small statement:

let result = 8 + 4 * 2;
print result;

The intended output is:

16

Producing that result requires several distinct engineering stages.

A well-designed language implementation keeps those stages separate:

1. Source loading
2. Lexical analysis
3. Parsing
4. Semantic analysis
5. Intermediate representation
6. Bytecode generation
7. Virtual-machine execution
8. Runtime diagnostics

Each stage should have a clear input, output, ownership model, and failure contract.

Source Text Is an Input Artifact

The first responsibility is to load source text safely.

A source object should preserve more than a character buffer. It should also retain the filename and enough position information to produce useful diagnostics.

#include <filesystem>
#include <string>

struct SourceFile {
    std::filesystem::path path;
    std::string text;
};

This object establishes a relationship between the bytes being processed and the file from which they originated.

Later compiler stages should not need to reopen the file or reconstruct its identity.

A source location can describe a bounded region inside that text:

#include <cstddef>

struct SourceSpan {
    std::size_t offset;
    std::size_t length;
    std::size_t line;
    std::size_t column;
};

Tokens, syntax nodes, and diagnostics can carry these spans forward. When a later stage discovers an error, it can report the original source location rather than only stating that something failed.

Lexing Converts Characters Into Tokens

The lexer reads characters and groups them into meaningful units.

For the statement below:

let result = 8 + 4 * 2;

A lexer might produce:

LET
IDENTIFIER("result")
EQUAL
INTEGER("8")
PLUS
INTEGER("4")
STAR
INTEGER("2")
SEMICOLON
END_OF_FILE

A bounded token representation might be:

#include <string_view>

enum class TokenKind {
    let_keyword,
    print_keyword,
    identifier,
    integer,
    equal,
    plus,
    minus,
    star,
    slash,
    left_parenthesis,
    right_parenthesis,
    semicolon,
    end_of_file,
    invalid
};

struct Token {
    TokenKind kind;
    std::string_view lexeme;
    SourceSpan span;
};

Using std::string_view allows tokens to reference the source buffer without allocating a new string for every lexeme.

That choice introduces a lifetime requirement: the source text must remain alive as long as tokens reference it.

This is an example of how compiler design and memory ownership are inseparable.

The Lexer Must Always Make Progress

A lexer typically examines the current character, determines which token begins there, and advances its cursor.

class Lexer {
public:
    explicit Lexer(std::string_view source)
        : source_(source)
    {
    }

    Token next_token();

private:
    bool at_end() const
    {
        return cursor_ >= source_.size();
    }

    char current() const
    {
        return at_end() ? '�' : source_[cursor_];
    }

    std::string_view source_;
    std::size_t cursor_ = 0;
    std::size_t line_ = 1;
    std::size_t column_ = 1;
};

One essential lexer invariant is:

Every call that has not reached the end of the source must either consume input or report a controlled failure.

Without this invariant, an unexpected character can cause the lexer to return the same invalid token repeatedly without advancing.

That creates an infinite loop in the compiler frontend.

Parsing Converts Tokens Into Structure

Tokens identify individual units, but they do not express relationships.

The expression:

8 + 4 * 2

must preserve multiplication precedence. Its structure is equivalent to:

    +
   / 
  8   *
     / 
    4   2

A recursive-descent parser can encode precedence through separate functions.

class Parser {
public:
    Expression parse_expression();

private:
    Expression parse_additive();
    Expression parse_multiplicative();
    Expression parse_primary();
};

The calling structure establishes precedence:

Expression Parser::parse_expression()
{
    return parse_additive();
}

Expression Parser::parse_additive()
{
    Expression expression =
        parse_multiplicative();

    while (
        check(TokenKind::plus) ||
        check(TokenKind::minus)
    ) {
        Token operation = advance();
        Expression right =
            parse_multiplicative();

        expression = make_binary(
            std::move(expression),
            operation,
            std::move(right)
        );
    }

    return expression;
}

Because parse_additive obtains its operands from parse_multiplicative, multiplication is grouped before addition.

The grammar becomes executable control flow.

Syntax Trees Represent Meaningful Structure

An abstract syntax tree records the structure needed by later stages without preserving every punctuation token.

#include <memory>
#include <variant>

struct IntegerExpression {
    std::int64_t value;
    SourceSpan span;
};

struct BinaryExpression;

using Expression = std::variant<
    IntegerExpression,
    std::unique_ptr<BinaryExpression>
>;

struct BinaryExpression {
    TokenKind operation;
    Expression left;
    Expression right;
    SourceSpan span;
};

This representation raises several design questions:

• Are nodes stored individually or in an arena?
• Are child relationships owning or non-owning?
• Can nodes move after creation?
• How are source spans preserved?
• How does a visitor distinguish node variants?
• When is the entire tree released?

For a small frontend, unique ownership can make lifetimes explicit. For a larger compiler, arena allocation may reduce allocation overhead and simplify collective lifetime management.

The correct choice depends on the required invariants rather than on syntax alone.

Semantic Analysis Validates Meaning

A parser can determine that this statement is structurally valid:

print missing_value;

It cannot determine whether missing_value has been declared unless it also performs semantic work.

A symbol table maps names to language-level entities.

#include <string>
#include <unordered_map>

using SymbolId = std::uint32_t;

struct Symbol {
    SymbolId id;
    std::string name;
    SourceSpan declaration;
};

class SymbolTable {
public:
    bool declare(Symbol symbol);
    const Symbol *find(
        std::string_view name
    ) const;

private:
    std::unordered_map<
        std::string,
        Symbol
    > symbols_;
};

Semantic analysis may verify:

• Identifiers are declared before use.
• A name is not declared twice in the same scope.
• Operators receive compatible operand types.
• Function calls use valid argument counts.
• Return statements occur in valid contexts.
• Control-flow requirements are satisfied.

These checks belong in a distinct stage because grammatical validity and semantic validity are different properties.

Diagnostics Are Part of the Compiler Interface

A compiler should not fail with only:

syntax error

A useful diagnostic identifies what happened and where.

example.poise:3:9: error: expected expression after '+'
    let total = value + ;
                        ^

A diagnostic representation can preserve structured information:

enum class DiagnosticSeverity {
    note,
    warning,
    error
};

struct Diagnostic {
    DiagnosticSeverity severity;
    std::string message;
    SourceSpan primary_span;
};

The compiler driver can collect diagnostics and decide whether the current stage may continue.

This separates detecting an error from presenting it.

It also makes diagnostics testable. A test can verify the severity, message, and source span without comparing an entire terminal session.

Intermediate Representation Creates a Stable Boundary

Directly executing an abstract syntax tree is possible, but an intermediate representation creates a stronger boundary between the frontend and runtime.

For a small stack-based runtime, bytecode can serve as that representation.

enum class OpCode : std::uint8_t {
    constant,
    add,
    subtract,
    multiply,
    divide,
    define_global,
    load_global,
    print,
    halt
};

The source statement:

let result = 8 + 4 * 2;
print result;

might compile to:

CONSTANT 8
CONSTANT 4
CONSTANT 2
MULTIPLY
ADD
DEFINE_GLOBAL result
LOAD_GLOBAL result
PRINT
HALT

This representation makes evaluation order explicit.

It is also easier to serialize, inspect, test, and execute than a collection of frontend-specific syntax nodes.

Instructions Need a Defined Encoding

An instruction format must specify how operation codes and operands are represented.

#include <cstdint>
#include <vector>

struct Chunk {
    std::vector<std::uint8_t> code;
    std::vector<Value> constants;
    std::vector<SourceSpan> locations;
};

The vectors must remain synchronized:

• code[index] identifies an instruction byte.
• locations[index] identifies the source associated with that byte.
• Constant operands index into constants.

This creates an important invariant:

Every emitted instruction byte must have corresponding source-location information.

That invariant allows runtime failures to be traced back to the original program.

The Virtual Machine Owns Execution State

A stack-based virtual machine maintains an instruction pointer and operand stack.

#include <cstddef>
#include <span>
#include <vector>

class VirtualMachine {
public:
    explicit VirtualMachine(
        const Chunk &chunk
    )
        : chunk_(chunk)
    {
    }

    RuntimeResult run();

private:
    const Chunk &chunk_;
    std::size_t instruction_pointer_ = 0;
    std::vector<Value> stack_;
};

The runtime borrows the bytecode chunk, so the chunk must outlive the virtual machine.

The operand stack is owned by the virtual machine because it exists only during execution.

These ownership relationships should be deliberate and documented.

Stack Effects Must Be Predictable

Each bytecode instruction has a stack effect.

CONSTANT 8     stack: [8]
CONSTANT 4     stack: [8, 4]
CONSTANT 2     stack: [8, 4, 2]
MULTIPLY       stack: [8, 8]
ADD            stack: [16]

MULTIPLY removes two operands and pushes one result. Its net stack effect is minus one.

The implementation must detect stack underflow before accessing operands.

RuntimeResult VirtualMachine::execute_add()
{
    if (stack_.size() < 2) {
        return runtime_error(
            "ADD requires two operands"
        );
    }

    Value right = stack_.back();
    stack_.pop_back();

    Value left = stack_.back();
    stack_.pop_back();

    stack_.push_back(
        add_values(left, right)
    );

    return RuntimeResult::success;
}

Valid bytecode generated by the compiler should satisfy the stack contract. The virtual machine should still defend its boundary because bytecode may eventually be loaded, inspected, transformed, or corrupted independently.

Execution Tracing Makes the Runtime Observable

A trace mode can display the instruction pointer, current instruction, and stack state.

ip=0000  CONSTANT 8       stack=[]
ip=0002  CONSTANT 4       stack=[8]
ip=0004  CONSTANT 2       stack=[8, 4]
ip=0006  MULTIPLY         stack=[8, 4, 2]
ip=0007  ADD              stack=[8, 8]
ip=0008  DEFINE_GLOBAL    stack=[16]
ip=0010  LOAD_GLOBAL      stack=[]
ip=0012  PRINT            stack=[16]
16
ip=0013  HALT             stack=[]

Tracing supports:

• Debugging the compiler's emitted bytecode
• Diagnosing runtime behavior
• Teaching the execution model
• Comparing optimized and unoptimized instruction sequences
• Detecting incorrect stack transitions

Observability should be designed into the runtime rather than added only after failures become difficult to explain.

The Compiler Driver Coordinates the Pipeline

The driver owns the stage transitions.

CompileResult compile(
    const SourceFile &source
)
{
    Lexer lexer(source.text);
    std::vector<Token> tokens =
        tokenize(lexer);

    Parser parser(tokens);
    SyntaxTree syntax =
        parser.parse_program();

    DiagnosticCollection diagnostics;
    analyze_semantics(
        syntax,
        diagnostics
    );

    if (diagnostics.has_errors()) {
        return CompileResult::failure(
            std::move(diagnostics)
        );
    }

    Chunk bytecode =
        generate_bytecode(syntax);

    return CompileResult::success(
        std::move(bytecode)
    );
}

The driver should not contain the internal logic of every stage. Its responsibility is orchestration:

• Establish inputs
• Invoke stages in order
• Preserve diagnostics
• Stop when required invariants fail
• Transfer valid outputs to the next stage

This structure allows each stage to be tested independently while preserving an end-to-end compilation path.

Test Boundaries, Not Only Functions

A language implementation needs tests at several levels.

Lexer tests verify token kinds, lexemes, and source spans.

Parser tests verify tree structure and precedence.

Semantic tests verify valid and invalid name or type relationships.

Bytecode tests verify emitted instruction sequences and constants.

Runtime tests verify results, failures, and stack behavior.

End-to-end tests verify complete programs:

source:
let result = 8 + 4 * 2;
print result;

expected output:
16

expected status:
success

Failure cases are equally important:

source:
print unknown_name;

expected diagnostic:
undeclared identifier 'unknown_name'

expected status:
compile failure

These tests make stage contracts observable and protect them as the implementation evolves.

Optimization Comes After Correctness

Optimization should not be the first objective of a new language implementation.

The early priorities are:

1. Correct tokenization
2. Predictable grammar
3. Valid syntax trees
4. Clear semantic rules
5. Stable bytecode contracts
6. Safe virtual-machine execution
7. Useful diagnostics
8. Repeatable tests

Once these foundations are dependable, measurement can guide optimization.

Possible later stages include:

• Constant folding
• Dead-code elimination
• Compact instruction encoding
• Faster symbol lookup
• Arena-based syntax allocation
• Bytecode verification
• Inline caching
• Register-based execution
• Native-code generation
• Garbage collection
• Just-in-time compilation

Optimization becomes safer when the unoptimized behavior is already well specified and thoroughly tested.

Building the Path From Systems to Runtimes

Compiler and runtime engineering brings together several systems disciplines:

• Memory ownership
• Data-structure design
• Parsing and formal structure
• Error handling
• Binary representation
• Execution models
• Performance measurement
• Testing and diagnostics
• Operating-system interaction

My PoiseLang work is being developed along this progression: establish a dependable C and C++ engineering foundation, define clear frontend boundaries, preserve source and diagnostic information, introduce intermediate representations, and advance toward executable runtime behavior through measured stages.

The objective is not only to produce a language that accepts input. It is to build an implementation whose transformations, ownership rules, failures, and execution behavior can be inspected and understood from source text to runtime state.