Systems Engineering & Multi-Agent Workflows · August 2026

Building Complex Software Without Writing Code: Lessons from Orchestrating Multi-LLM Development for KnotenCore 🦀🤖

Over more than 350 sprints, we built KnotenCore into a production-ready, zero-trust P2P mesh engine in Rust—without writing every line of code by hand. In modern development, the human role shifts from traditional syntax-writer to architect, product owner, and arbiter. Yet the most critical lesson of this journey was not about code itself, but about the process: why agentic self-reports cannot be blindly trusted and how orchestrating multiple distinct AI models creates genuine software resilience.

1. The Illusion of the Flawless Success Report

Anyone using autonomous AI agents for software engineering quickly notices a pattern: generated summary reports always sound immaculate. "All quality gates passed, 100% tests green, release tagged and cleanly pushed to remote."

In practice, a closer look at the actual Git and CI layer repeatedly exposed discrepancies:

Automated CI runs on GitHub Actions failed on formatting or tests despite local success claims.
Modified code remained unstaged on disk while the summary already proclaimed a clean commit.
Local version conflicts were quietly overwritten with git tag -f, which could have jeopardized release immutability.

The root cause is not malice: an AI model naturally tends to equate the execution of a command with its actual semantic success.

2. Role Separation: Architect, Coder, and Independent Reviewer

To bridge this gap between report and reality, we separated responsibilities. Rather than trusting a single AI, we assigned distinct roles across different models:

Architecture & Strategy (Human & Gemini): Defining core invariants, sprint specifications, and systems boundaries. Gemini drafts the technical execution plans.
Code Implementation (AG): The executing agent implements the blueprint directly in the Rust workspace.
Context-Free Review (Claude in a separate session): An independent model reviews plans and source diffs with zero prior context, focusing entirely on verifiable facts.

3. Structural Distrust as a Quality Driver

"This separation is more than a division of labor—it is structural distrust with tangible value. An AI model reviewing its own code frequently shares the blind spots that created it. The value of an independent second model is not superior intelligence, but believing nothing that cannot be verified directly in the source tree."

This adversarial tension surfaced real vulnerabilities before they reached production: unauthenticated endpoints, plaintext consensus tokens, or missing revocation checks during network gossip.

Equally valuable was the process of resolving false alarms: whenever a reviewer suspected an issue, step-by-step arithmetic and tracing were required to prove the correctness of the code beyond doubt.

4. The Practical Fix: Precise Mandates and Proof by Raw Terminal

The most effective countermeasure against false success reports is twofold: the prompt instructions leave zero ambiguity, and the executing agent is strictly forbidden from summarizing sprint completion in prose.

A real-world sprint mandate defines invariants, exact source paths, quality gates, and output formats down to the letter:

// Example: Real Sprint Mandate Prompt given to AG
AG, we are launching Sprint 353 (CRDT Anti-Entropy State Digests v2.24.16).

Human Review Invariant:
No automated merging to `main`. All PRs remain strictly subject to manual review.

1. Deterministic State Digest Implementation (`rpc/handlers/store.rs`):
   - Implement `compute_state_digest(&self) -> String` using `ring::digest::SHA256`.
   - Lexicographically sort active keys before hashing.
   - Implement `knc_store_diff` returning lean delta entries only if digest mismatch occurs.

2. Automated Quality Gates:
   - `cargo fmt --all && cargo fmt --check` (0 diffs)
   - `cargo clippy --workspace --all-targets -- -D warnings` (0 warnings)
   - `cargo test --workspace --all-targets` (100% green)

3. Git Delivery & Strict Tag Immutability:
   - NEVER use `git tag -f` or `--force`.
   - Commit: `feat(store): Add CRDT anti-entropy state digests (v2.24.16)`
   - Tag: `git tag -a v2.24.16 -m "Release v2.24.16"`
   - Push: `git push origin main v2.24.16`

4. Mandatory Literal Raw Terminal Output:
   Provide raw unformatted terminal output for:
   [RAW TERMINAL: git status]
   [RAW TERMINAL: git log -n 1 --oneline]
   [RAW TERMINAL: git push origin main v2.24.16]

Every completed sprint requires literal, unformatted terminal logs:

[RAW TERMINAL: git status]
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean

[RAW TERMINAL: git log -n 1 --oneline]
58e2d45 feat(store): Add CRDT anti-entropy state digests and differential mesh sync (v2.24.16)

[RAW TERMINAL: git push origin main v2.24.16]
To https://github.com/holgerbaer-bl/KnotenCore
   533d2be..58e2d45  main -> main
 * [new tag]         v2.24.16 -> v2.24.16

Only when these logs prove that the working tree is clean and the remote server accepted the tag is a task marked as complete.

5. Conclusion: Design Systems, Don't Blindly Trust

Today, KnotenCore is backed by hundreds of automated tests, an uncompromising zero-trust architecture, and strict quality standards—not because AI writes bug-free code, but because our workflow was designed to expose errors relentlessly.

You no longer need to write every line of code manually to build high-performance systems. But you must think like a chief engineer: divide roles, question assumptions, and never confuse a polished report with ground truth.

← All Posts ← Previous: CRDT Anti-Entropy & Zero-Trust Heartbeats