Debugging AI Agents: Why Your Prompt Edits Aren’t Always Taking Effect in Conditional Loading Architectures.

Developers frequently encounter a perplexing challenge when working with AI agents: changes meticulously applied to a skill’s prompt, confirmed to be present in the runtime files, and followed by a system restart, often fail to alter the agent’s behavior. This isn’t an intermittent glitch; it’s a reliably reproducible phenomenon for a significant portion of queries. The immediate, intuitive response is to intensify prompt editing efforts, perhaps by adding more explicit instructions or reinforcing existing ones. However, as recent observations from a development team building an AI operations layer for an all-electric charter catamaran reveal, the true problem frequently lies elsewhere: the prompt being edited was simply not included in the request sent to the large language model (LLM) for specific interactions. This crucial insight underscores a fundamental characteristic of modern AI agent frameworks that rely on a base system prompt augmented by conditionally loaded skill bodies. If a rule or piece of information needs to apply to every query, it must reside within the "always-on" foundational layer, not in a skill that only triggers under certain conditions. Compounding this, a secondary, equally critical flaw often involves the absence of a proper deployment mechanism for these always-on foundational files, allowing them to silently drift out of sync with their source repository.

This architectural nuance is not unique to a single platform but is a framework-general issue. The development team encountered this particular challenge while working with a local Hermes agent, an open-source framework for building AI assistants. However, the same underlying principle applies to any system adopting the increasingly common pattern of a base prompt combined with modular SKILL.md files. Prominent examples include Claude Agent Skills, Gemini CLI skills, and VS Code agent skills, among others, all of which leverage a "progressive disclosure" model where certain prompt elements are loaded only when specific conditions are met. Understanding this distinction between always-on and conditionally loaded prompt layers is paramount for robust and predictable AI agent development.

The Confounding Problem: Agent Discrepancy

The genesis of this debugging saga began within an AI agent stack designed to manage various operational aspects of a sailing vessel. This stack featured a shared persona, defined in a SOUL.md file, which established the agent’s core identity, voice, and overarching behavioral guidelines. Layered on top were per-agent skills, such as Navigator, Engineer, and Logbook, each adding specialized capabilities and instructions relevant to its domain. The SOUL.md file, for instance, named the vessel and set the conversational tone, while individual skill files provided context-specific directives.

The specific incident involved an attempt to update the vessel’s name. The development team modified the vessel name within the Navigator skill’s body, specifically in the SKILL.md file associated with it. Following standard procedure, they deployed the updated skill and then verified that the change was correctly reflected in the runtime file on disk. A grep command targeting the deployed SKILL.md file confirmed the presence of the new vessel name:

$ grep -n "VESSEL_NAME_HERE|aboard" ~/.hermes/skills/naturali/navigator/SKILL.md
1:Navigator agent aboard s/v Naturali.

With the file appearing correct and the system restarted, the team proceeded to test the agent with a straightforward query: "how’s our depth?". To their surprise and frustration, the agent responded using the old vessel name, "Wrongboat," despite all efforts to update it:

> how's our depth?
Aboard s/v Wrongboat, depth is 8.2 metres below the keel, Captain.

This outcome was perplexing. The file on disk was unequivocally correct, yet the agent continued to output the outdated information. Repeated edits to the skill body, followed by redeployment and restarts, yielded no change for this particular type of query. The agent seemed impervious to the intended modification, creating a significant roadblock in ensuring the AI’s consistent and accurate representation of the vessel’s identity.

The Crucial Diagnosis: Inspecting the Actual Prompt

The breakthrough in understanding this persistent anomaly came when the development team shifted their diagnostic approach from theoretical assumptions about what the model should see to concrete evidence of what it actually received. Hermes, like many modern AI frameworks, provides mechanisms to dump the full request payload sent to the LLM for each call. This could be a debug log, a --print-prompt option, or a proxy capture. By examining the request dump for the problematic query, the true nature of the issue became apparent.

A grep operation on the most recent request dump revealed a critical discrepancy:

$ ls -t ~/.hermes/sessions/request_dump_*.json | head -1
/Users/me/.hermes/sessions/request_dump_1733270400.json

$ grep -o "ship's computer" ~/.hermes/sessions/request_dump_*.json | tail -1
ship's computer            # <-- base persona text IS present

$ grep -o "Navigator agent aboard" ~/.hermes/sessions/request_dump_*.json | tail -1
                           # <-- nothing. the skill body is NOT in the prompt

The output was unequivocal. For a simple query like "how’s our depth?", the base persona’s text, originating from SOUL.md, was indeed present in the prompt. However, the Navigator skill’s body, SKILL.md, which contained the updated vessel name, was entirely absent. This meant the query did not activate or "trigger" the Navigator skill, and consequently, its associated SKILL.md body was never loaded into the LLM’s context. The vessel name that the team had so carefully edited resided solely within that conditionally loaded skill body.

This behavior is a direct consequence of the "progressive disclosure" model employed by many modern agent frameworks. As explicitly stated in Anthropic’s Agent Skills documentation, these systems typically operate on a two-level loading mechanism:

• Level 1: Metadata (always loaded): This includes information like a skill’s name and description. Claude (and similarly, Hermes, Gemini, etc.) loads this metadata at startup and incorporates it into the system prompt for every interaction. This allows the model to be aware of available skills and decide when to invoke them.
• Level 2: Instructions (loaded when triggered): This encompasses the detailed instructions and context contained within a skill’s body (e.g., SKILL.md). This content only enters the context window if and when the model determines that a query matches a skill’s description and warrants its activation.

Therefore, while a skill’s name and description are always in context, the crucial body containing specific instructions, identity details, or behavioral rules only loads when that skill is explicitly triggered. Conversely, the base system prompt (like SOUL.md in this case) is consistently present in every single request. The implication is clear: any instruction that must always apply, regardless of whether a specific skill is invoked, must reside in this always-on foundational layer. Vessel identity, being a core attribute of the agent, is inherently an "always-apply" rule. Its placement in a conditional skill body was a miscategorization, leading to its intermittent absence from the prompt.

The "Wrongboat" output was merely a symptom exposing this deeper architectural misunderstanding. The principle extends beyond identity to any invariant rule that must consistently govern the agent’s behavior. This includes time and units discipline (e.g., always reporting depth in meters, never narrating raw UTC timestamps), and crucial anti-fabrication rules (e.g., never guessing data availability). If these rules are tucked away in skill bodies, they will silently fail to apply on any query that does not happen to trigger the specific skill containing them, leading to unpredictable and potentially erroneous agent responses.

Failed Attempts and Crucial Debugging Lessons

Before arriving at the correct diagnosis, the development team undertook several common but ultimately misguided debugging attempts, each reinforcing the critical lessons learned.

Attempt 1: Edit the Skill Body and Redeploy (Harder)

The initial, instinctive reaction was to re-edit the Navigator skill’s source file, ensuring the vessel name was correct, and then redeploy it. The output confirmed the successful deployment:

# fixed the name in the skill source, redeployed
$ scripts/deploy-navigator.sh
deploy-navigator: wrote ~/.hermes/skills/naturali/navigator/SKILL.md (vessel: Naturali)

However, subsequent queries continued to yield the old vessel name. This failure, despite the file being demonstrably correct on disk, highlighted that the issue wasn’t the content of the file itself, but its loading mechanism. One can edit a conditionally loaded body indefinitely, but it will never influence a query that does not trigger the corresponding skill. This phase burned valuable debugging cycles due to a misdirected focus.

Attempt 2: Assume Deployment Failure and Re-check the Disk

Faced with the persistence of the old name, the team then questioned the deployment process itself. Perhaps the file hadn’t truly updated on disk, or a caching issue was at play. A direct inspection of the runtime file using cat confirmed its correctness:

$ cat ~/.hermes/skills/naturali/navigator/SKILL.md | head -1
Navigator agent aboard s/v Naturali.

The runtime file was correct all along. This confirmation was a pivotal moment. It definitively ruled out a deployment failure or a simple file corruption issue. It should have immediately redirected the debugging thought process from "is this file correct?" to the more profound question: "is this file even being loaded?" These are fundamentally different questions, and the team had been attempting to answer the wrong one. The inability to distinguish between these two failure modes (incorrect prompt vs. un-loaded prompt) cost significant time.

Attempt 3: Duplicate Identity Rule Across Skill Bodies

A tempting, albeit flawed, solution considered was to duplicate the vessel name and identity statement into every skill body. The rationale was that at least some loaded skill would then carry the correct information, ensuring the name appeared when any skill was triggered. While this might "work" for queries that activate a skill, it presents several critical shortcomings:

1. Incomplete Coverage: Bare, social, or ambiguous queries that trigger no specific skill would still revert to the old, incorrect persona, as no skill body containing the name would be loaded.
2. Drift Trap: Creating N copies of an invariant value (the vessel name) across multiple files introduces a classic maintenance nightmare. Keeping these copies synchronized becomes a continuous, error-prone task, making "drift" (where one copy updates but others don’t) almost inevitable.
3. Symptom Treatment: This approach treats the symptom (the name not appearing) rather than the root cause (the rule’s incorrect placement). The vessel’s identity is not a per-skill attribute; it’s an always-on characteristic of the agent.

Ultimately, this approach was rejected because it violated the principle of a single source of truth and failed to address the fundamental architectural requirement for always-on rules.

The Definitive Fix: Strategic Prompt Layering and Robust Deployment

The successful resolution involved a two-pronged approach: correctly classifying and relocating always-on invariants, and implementing a robust, automated deployment mechanism for all critical prompt files.

Relocating Always-On Invariants to the Base Persona

The core of the fix was to move the vessel identity and other universally applicable rules from the conditional skill bodies into the SOUL.md file, the base persona that is loaded on every request. To ensure flexibility and maintainability, the vessel name was templated within SOUL.md:

<!-- SOUL.md – the base persona, loaded on EVERY request -->
You are the ship's computer aboard s/v VESSEL_NAME.
You address the user as "Captain."
Never speak a raw UTC timestamp in conversation.
Report a reading and its source path; never narrate whether data is
"live," from a "mock," or whether the vessel is "ashore" – you aren't
given that and must not guess it.

This ensures that regardless of which, if any, skill is triggered, the agent’s core identity and critical behavioral guidelines are consistently present in the prompt. Rules like addressing the user as "Captain," avoiding raw UTC timestamps, and refraining from fabricating data status are now guaranteed to apply to every interaction.

Implementing a Robust Deployment Workflow

The second, equally vital part of the solution addressed the silent drift problem. The SOUL.md file had previously been hand-placed into the runtime directory, a practice that led to it becoming stale as the vessel name changed over time in the source repository. To prevent this, a dedicated deploy step was created for SOUL.md, mirroring the robust deployment scripts already in place for individual skills. This script, scripts/deploy-soul.sh, dynamically substitutes the VESSEL_NAME placeholder with the correct name, sourced from a single, active vessel profile:

# scripts/deploy-soul.sh
vessel_name="$(resolve_vessel_name "$repo_root")"   # reads the active vessel profile
sed "s|VESSEL_NAME|$vessel_name|g" "$repo_root/SOUL.md" > "$HERMES_HOME/SOUL.md"

This automated process ensures that the SOUL.md file in the runtime environment is always up-to-date with the latest vessel information from the authoritative source. After deploying the updated SOUL.md with the correct vessel name, the agent’s response to the original query finally reflected the desired change:

> how's our depth?
Aboard s/v Naturali, depth is 8.2 metres below the keel, Captain.

The agent now correctly identified the vessel, even for a query that triggered no specific skill, because its identity was now consistently part of the always-on persona. The skill bodies, in turn, were refocused to contain only genuinely conditional instructions—operational hints relevant only when that specific skill is active. For example, the Navigator skill’s body was streamlined to:

<!-- skills/navigator/body.md – loaded only when Navigator triggers -->
For "how's our depth?", read environment.depth.belowKeel, not belowTransducer.

This optimized layering ensures token efficiency by loading only necessary context and maintains a clear separation of concerns between universal rules and task-specific guidance.

Broader Implications and Best Practices for AI Agent Development

The debugging experience with the Hermes agent on the Naturali catamaran offers several critical lessons applicable to all AI agent development, regardless of the specific framework.

The Categorization of Prompt Rules: Always-On vs. Conditional

The foremost lesson is the absolute necessity of meticulously categorizing every prompt rule as either "always-on" or "conditional." Developers must ask a fundamental question for each rule: "Does this instruction or piece of information need to hold true even for a query that triggers no specific skill?" If the answer is yes, it unequivocally belongs in the base prompt or foundational system instructions that are loaded for every interaction. This includes core identity statements, safety protocols, units of measurement discipline, conversational tone guidelines, and crucial anti-fabrication rules designed to prevent the agent from hallucinating or misrepresenting information. If a rule is only relevant within the scope of a particular task or capability, then the skill body is the appropriate, token-efficient home. Misplacing an always-on rule in a conditional skill body guarantees its intermittent failure, leading to unpredictable agent behavior and potentially critical errors in sensitive applications.

The Indispensability of Request Dumps

"The prompt says X, but the agent does Y" is a common developer lament that can arise from two distinct failure modes: the prompt content is incorrect, or the prompt isn’t loaded at all. From an external perspective, these two problems appear identical. The only way to definitively distinguish between them is to inspect the actual prompt payload sent to the LLM. Most frameworks offer some form of request dumping, debug logging, or prompt printing. The experience on the Naturali catamaran dramatically demonstrated how quickly two debugging cycles were wasted editing a file that was never even part of the request. A single grep of the dumped prompt would have revealed the true problem immediately. Incorporating prompt inspection into the standard debugging workflow for AI agents is not merely good practice; it is essential for efficiency and accuracy.

Automated Deployment for All Runtime Artifacts

The second major takeaway concerns deployment hygiene. Any runtime prompt artifact that influences agent behavior must have an associated, automated deployment step. The SOUL.md file, initially hand-placed, silently diverged from its source of truth in the repository. This "silent rot" meant the running agent was operating with outdated instructions, unbeknownst to the developers. Implementing a pre-commit Git hook, as demonstrated in the fix, ensures that any changes to relevant source files (like SOUL.md or deployment scripts) automatically trigger the redeployment of the corresponding runtime artifact. This practice guarantees that the runtime environment consistently reflects the state of the repository, preventing divergence and ensuring that changes are reliably applied. This extends beyond just prompt files to any configuration or data source that an agent relies upon.

The Power of a Single Source of Truth

Finally, the principle of a single source of truth is paramount in complex AI agent systems. The vessel name, in this case, was not hardcoded in multiple locations. Instead, it was resolved from an "active vessel profile," the same authoritative source that seeds the data-layer’s base values. This templating approach ensures that the vessel is named identically in both the agent’s internal data model and its external persona. Should the vessel’s name change, or if the agent needs to operate for a different boat, only one central profile needs to be updated, eliminating the need for tedious and error-prone prompt edits. Hardcoding values that appear in multiple layers is a recipe for inconsistency and future debugging headaches.

The development of the AI ops layer for the all-electric charter catamaran, Naturali, highlights the evolving challenges and best practices in building robust, reliable, and context-aware AI agents. By integrating a local LLM with a shared persona and specialized skills over a marine data bus, the project underscores the importance of precision in prompt engineering and deployment. The agent’s ability to consistently refer to the boat by its correct name, irrespective of the query’s complexity or the skills it invokes, is not just a matter of politeness but a critical component of its operational integrity. The open-source nature of the agent skills and deploy scripts, available at github.com/sailingnaturali/naturali-agents, offers a valuable resource for other developers navigating the intricate landscape of AI agent architectures. As AI systems become more integral to real-world operations, meticulous attention to these architectural details will be key to their success and trustworthiness.