demo · 2026-10-06

pi.dev driving a self-hosted LLM gateway

A terminal coding agent given three bounded tasks on a real Python project. Everything below is generated from the project's own artifacts — including the verification commands, run at build time.

pi → LiteLLM gateway → model — no provider key in the client, all inference routed through one endpoint

Live verification output

test suitePASSED: all valid inputs parsed, all malformed inputs rejected

behaviour invariantBEHAVIOUR PRESERVED

fingerprint sha256b1452ab0bdd1ec9bdcb29a170519f9c5f1281bef3da4d347039a927238118db1

The invariant is the interesting part. The third task refactored one module into a four-file package. Correctness was judged by hashing the observable behaviour across a wide input space — thousands of parses, a formatter sweep, malformed inputs, and error cases — and requiring the hash to be unchanged. A refactor that looks right but behaves differently fails automatically.

Three tasks

✓

Fix a bug until tests pass

Parser silently accepted garbage — 10m bananas → 600.

Ran the suite first, added a whole-string validation guard, verified. Did not touch the tests.
✓

Add a feature to an explicit spec

Implement format_duration() with 9 exact cases + 2 error cases.

All 11 cases correct including the counter-intuitive 90 → '1m30s'. Handled errors unprompted.
✓

Multi-file refactor, behaviour-preserving

Split one module into a 4-file package without changing observable behaviour.

Byte-identical behaviour (sha256 fingerprint), correct relative imports, shared constants extracted once.

One honest caveat. An earlier run of task 2 used an ambiguous spec ("use the largest units") and left a failing test it had written itself — it did not push back on the contradiction. With an explicit spec listing every expected output, the same model produced a clean pass. The agent is only as good as the brief.

The artifacts

duration/__init__.py9 lines

"""Duration conversion package.

Re-exports parser and formatter functions.
"""

from .parser import parse_duration
from .formatter import format_duration

__all__ = ['parse_duration', 'format_duration']

duration/parser.py33 lines

"""Duration parsing functionality.

Return the total number of seconds in *text*.

>>> parse_duration("90s")
90
>>> parse_duration("1h30m")
5400
>>> parse_duration("2d")
172800
"""

from ._units import _TOKEN, _UNITS
import re


def parse_duration(text: str) -> int:
    if not text:
        raise ValueError("empty duration")

    # First validate the entire string matches the expected pattern
    if not re.match(r"^\s*(\d+[smhd])+\s*$", text):
        raise ValueError(f"invalid duration format: {text!r}")
    
    # Then parse individual components
    matches = _TOKEN.findall(text)
    if not matches:
        raise ValueError(f"no duration components found in {text!r}")

    total = 0
    for value, unit in matches:
        total += int(value) * _UNITS[unit]
    return total

duration/formatter.py33 lines

"""Duration formatting functionality.

Convert seconds into a decomposed duration string like "1d2h3m4s".

>>> format_duration(90)
'1m30s'
>>> format_duration(5400)
'1h30m'
>>> format_duration(93784)
'1d2h3m4s'
"""

from ._units import _UNITS


def format_duration(seconds: int) -> str:
    if not isinstance(seconds, int):
        raise ValueError("seconds must be integer")
    if seconds < 0:
        raise ValueError("seconds cannot be negative")

    if seconds == 0:
        return "0s"

    components = []
    for unit, divisor in [("d", 86400), ("h", 3600), ("m", 60)]:
        if seconds >= divisor:
            value, seconds = divmod(seconds, divisor)
            components.append(f"{value}{unit}")
    if seconds > 0 or not components:
        components.append(f"{seconds}s")

    return "".join(components)

duration/_units.py4 lines

import re

_UNITS = {"s": 1, "m": 60, "h": 3600, "d": 86400}
_TOKEN = re.compile(r"(\d+)([smhd])")

test_duration.py100 lines

"""Tests for duration module — stdlib only, no pytest required.

Run:  python3 test_duration.py
"""

from duration import parse_duration, format_duration

VALID = [
    ("90s", 90),
    ("5m", 300),
    ("2h", 7200),
    ("1h30m", 5400),
    ("1d2h3m4s", 93784),
    ("0s", 0),
]

MALFORMED = [
    "",
    "abc",
    "5x",
    "10m5",        # trailing bare number
    "1h30m!!",     # trailing garbage
    "10m bananas",
    "  ",          # whitespace only
]


def main() -> int:
    failures = []

    # Test format_duration
    FORMAT_TESTS = [
        (0, "0s"),
        (45, "45s"),
        (60, "1m"),
        (3600, "1h"),
        (86400, "1d"),
        (90, "1m30s"),
        (5400, "1h30m"),
        (3661, "1h1m1s"),
        (93784, "1d2h3m4s"),
    ]
    for seconds, expected in FORMAT_TESTS:
        try:
            got = format_duration(seconds)
        except Exception as e:                                  # noqa: BLE001
            failures.append(f"  format({seconds}) raised {type(e).__name__}: {e}")
            continue
        if got != expected:
            failures.append(f"  format({seconds}) -> {got!r}  expected {expected!r}")

    # Test format_duration edge cases
    try:
        format_duration(-1)
        failures.append("  format(-1) unexpectedly succeeded (should raise ValueError)")
    except ValueError:
        pass
    except Exception as e:                                      # noqa: BLE001
        failures.append(f"  format(-1) raised {type(e).__name__} (expected ValueError)")

    try:
        format_duration(1.5)
        failures.append("  format(1.5) unexpectedly succeeded (should raise ValueError)")
    except ValueError:
        pass
    except Exception as e:                                      # noqa: BLE001
        failures.append(f"  format(1.5) raised {type(e).__name__} (expected ValueError)")

    # Test parse_duration
    for text, expected in VALID:
        try:
            got = parse_duration(text)
        except Exception as e:                                  # noqa: BLE001
            failures.append(f"  {text!r:16s} raised {type(e).__name__}: {e}")
            continue
        if got != expected:
            failures.append(f"  {text!r:16s} -> {got}  expected {expected}")

    for text in MALFORMED:
        try:
            got = parse_duration(text)
        except ValueError:
            continue  # correctly rejected
        except Exception as e:                                  # noqa: BLE001
            failures.append(f"  {text!r:16s} raised {type(e).__name__} (expected ValueError)")
            continue
        failures.append(f"  {text!r:16s} -> {got}  expected rejection")

    if failures:
        print(f"FAILED ({len(failures)}):")
        for f in failures:
            print(f)
        return 1

    print("PASSED: all valid inputs parsed, all malformed inputs rejected")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

verify_fingerprint.py66 lines

"""Behaviour fingerprint: hashes the observable output across a wide input space.

Any refactor must leave this hash UNCHANGED. Run:
    python3 verify_fingerprint.py
It prints the hash; compare against FINGERPRINT below.
"""

import hashlib
import sys

FINGERPRINT = "b1452ab0bdd1ec9bdcb29a170519f9c5f1281bef3da4d347039a927238118db1"


def compute() -> str:
    # import INSIDE the function so an import error is reported, not a crash
    from duration import format_duration, parse_duration

    h = hashlib.sha256()

    for n in range(0, 400):
        for unit in "smhd":
            s = f"{n}{unit}"
            try:
                h.update(f"{s}={parse_duration(s)};".encode())
            except Exception as e:  # noqa: BLE001
                h.update(f"{s}!{type(e).__name__};".encode())

    for n in range(0, 200000, 137):
        h.update(f"{n}:{format_duration(n)};".encode())

    for s in ["", "abc", "5x", "10m5", "1h30m!!", "10m bananas", "  ", "1H", "-5s", "1.5h"]:
        try:
            h.update(f"{s}={parse_duration(s)};".encode())
        except Exception as e:  # noqa: BLE001
            h.update(f"{s}!{type(e).__name__};".encode())

    for v in [-5, "60", 1.5, None]:
        try:
            format_duration(v)
            h.update(f"fmt{v}=ok;".encode())
        except Exception as e:  # noqa: BLE001
            h.update(f"fmt{v}!{type(e).__name__};".encode())

    return h.hexdigest()


def main() -> int:
    try:
        got = compute()
    except Exception as e:  # noqa: BLE001
        print(f"IMPORT/RUN FAILED: {type(e).__name__}: {e}")
        return 2

    print(f"expected: {FINGERPRINT}")
    print(f"actual:   {got}")

    if got == FINGERPRINT:
        print("BEHAVIOUR PRESERVED")
        return 0

    print("BEHAVIOUR CHANGED")
    return 1


if __name__ == "__main__":
    sys.exit(main())

History

commits

d4dec4b refactor: split duration.py into a package (pi.dev, behaviour-preserving)
b9a1b46 feat: format_duration per explicit spec (pi.dev via LiteLLM gateway)
ded7e1e fix: reject malformed duration strings (pi.dev via LiteLLM gateway)
ae1835f baseline: failing tests + buggy parser
7600859 demo: duration parser with a bug

Generated from projects/pi-demo. The verification output above was produced by running the project's own test_duration.py and verify_fingerprint.py at build time — not transcribed by hand.

Live verification output

Three tasks

Fix a bug until tests pass

Add a feature to an explicit spec

Multi-file refactor, behaviour-preserving

The artifacts

History