How We Made Our Docs Test Themselves

What happened

LangChain faced the challenge of maintaining up-to-date code samples in their documentation due to rapid changes in APIs and dependencies. They used the Deep Agents CLI to automate the migration of inline code into standalone, testable code samples. This involved organizing code, adding setup and teardown, linting, and integrating with CI pipelines. The process was streamlined using a skill file (SKILL.md) that provided detailed instructions for the agent to follow.

Why it matters

This approach is significant for AI and software development teams because it ensures documentation remains accurate and functional, reducing the risk of outdated examples. It also improves the reliability of code samples by integrating them into CI pipelines, which helps catch errors early and maintain high-quality documentation.

• Automating the migration of inline code into testable examples is essential for maintaining accurate documentation.
• Using skills in Deep Agents can significantly reduce manual effort in documentation workflows.
• Integrating code samples into CI pipelines ensures they remain up-to-date and functional.
• The SKILL.md file provides a structured way to define and execute migration workflows for code samples.
• This approach is scalable and can be adapted to other documentation projects.

The bigger picture

The practical impact of this approach is improved documentation quality and reliability, which enhances user experience and reduces the risk of errors in code examples. It also streamlines the maintenance of documentation, making it easier for teams to keep up with rapid changes in technology.

By leveraging the Deep Agents CLI and skills, LangChain successfully automated the process of making code samples in documentation testable and CI-ready, addressing the challenge of stale code samples in fast-evolving fields like AI and LLMs.

Technical details

• Deep Agents CLI is a command-line interface that allows users to interact with agents.
• Skills are reusable instructions that guide agents to perform specific tasks.
• The SKILL.md file defines the workflow for migrating code samples, including directory structure, commands, and conventions.
• CI pipelines are used to regularly run tests on code samples and create tickets if tests fail.
• Markup tags like :snippet-start: and :snippet-end: are used to define code snippets in documentation.

FAQ