Rick's Blog

Over the past few months developing TaskWeaver, we’ve stumbled through countless pitfalls and collected some truly amusing (and occasionally painful) stories. Here’s a small journal of those moments—for memory, and maybe to help someone else down the road.

🧵 1. The First “Task Chain Crash” Incident

It was around 1 a.m. and I was debugging a module called ChainPlanner, which analyzes user intentions and decomposes them into executable task steps. I had confidently finished writing the task dependency DAG generator, only to be greeted with a runtime error:
RecursionError: maximum recursion depth exceeded.

After digging around, I realized I had mistakenly made plan call itself internally, with no cycle detection logic. In other words, I had created an infinite loop in the task chain.

Our team later gave this bug a name: “Ouroboros.”

🔐 2. That Time We Almost Pushed an OpenAI Key…

An intern once submitted a PR that accidentally committed .env.dev, which contained one of our test OpenAI API keys.

Thankfully, we had already integrated GitHub’s secret scanner and immediately received an alert email. We revoked the old key, issued a new one, and added a pre-commit hook to ensure all .env* files were ignored.

Since then, we’ve never dared to cut corners again.

🧠 3. Trying a Prompt Chain DSL So the AI Could “Figure It Out”

We were obsessed for a while with building a “Prompt Chain DSL”—a declarative way to define task workflows, like this:

task:
  name: write_blog
  input: topic
  steps:
    - name: outline
      uses: gpt-4
    - name: fill_in
      uses: gpt-4
      depends_on: outline

It looked elegant, but once we tried to execute it, the problems started: context passing was messy, chain debugging was a nightmare, and model invocation sequencing was buggy. It felt like we were rebuilding a “stateless Redux store” for prompts.

Eventually, we simplified the DSL into a JSON-based intermediate plan, interpreted by our internal planner agent.

🧪 4. When Model Output Got Unstable, We Started Scoring It

To make model outputs more reliable, we trained a lightweight scoring system to evaluate task plans—checking for things like duplicate steps, logical jumps, etc.

We called it PlanGrader. It wasn’t some fancy LLM—just a hybrid scorer built with rules and a small model.

For a while, our favorite daily ritual was watching PlanGrader give a plan a 90+ score, and taking a celebratory screenshot like it was a game achievement.

🛠️ 5. Three Days of Debugging—Because a Prompt Missed the Word “Please”

At one point, we noticed a task plan behaving strangely. The user asked for a travel itinerary, but the plan generated was a blog post outline.

We checked logs, traces, prompt mocks… and finally found the culprit. In the prompt, we had:
“You are an AI assistant. Please generate task steps based on the user’s need.”
But the intern had forgotten the “Please,” making it:
“You are an AI assistant. Generate task steps based on the user’s need.”

That one missing word changed the model’s role entirely.

We had mixed feelings that day.

💡 Conclusion

TaskWeaver’s development has been an evolving journey. Many of the grand architectural ideas ended up defeated by tiny devils in the details. But it’s precisely these small stories that deepened our understanding of human-AI collaboration, prompt engineering, and agent planning.

Maybe that’s the charm of building AI tools: every bug is a new discovery.