Introduction
Reliable and up-to-date code samples are crucial for effective technical documentation. However, manually creating and updating these samples to reflect the latest API and SDK changes is challenging for documentation teams, and outdated samples can harm clients.
A common best practice in the industry is to use standard specs (such as OpenAPI) to define API, which can help automate the process of generating code snippets in various languages. OpenAPI is a means to alleviate the problem of outdated code samples. While this addresses the issue of keeping samples up-to-date with the latest API changes, ensuring their reliability and correctness remains a concern.
With technical documentation potentially containing thousands of code samples in several languages, manual testing is not feasible. A better solution is to develop a dedicated testing framework that can automatically test code samples, ensuring users receive accurate and functional code.
Test framework for tech documentation samples
Framework Modules
- CLI Manager: Scans code, gathers info, and sends it to the service.
- Service: Docker-based testing, handles code samples information via endpoints.
- Mock server: Mocks OpenAPI spec using prism-cli.
- Queue system: Communication pipeline (redis) between service and workers.
- Workers: Docker images run language-specific code snippets.
Process
The CLI manager is responsible for collecting code snippets from a file system and communicating with the testing framework.
The testing framework employs a queue-based architecture to manage and distribute testing code samples.
Image 1. Queue-based architecture for testing framework
A queue serves as the in-memory database for pending code samples. Multiple worker test runners, each specifically designed for a particular programming language supported by the system, subscribe to this queue. This subscription model allows each runner to independently listen for and retrieve the code samples that require execution.
Each worker runner proceeds with its execution. During this phase, any API calls made by the code are directed to a dedicated mock server. This mock server is one of the core components of the framework, designed to simulate the behavior and responses of the actual API endpoints that the code would interact with in a live environment. By simulating the API responses, the testing framework ensures that tests are reliable, repeatable, and independent of the availability or state of external services.
Upon completion of the testing process, the results are sent to the user as a detailed report, which provides a summary of working and failing code samples.
It also contains information about the code samples that have failures during their execution and a small description of the error. It allows developers and other stakeholders to quickly identify and address any problematic areas within the codebase.
Benefits
- Test Any Language with Plug-and-Play Runners: The framework is designed to test code samples across a wide range of programming languages and can also be extended to add support for more languages by creating specific test runners for each one of them.
- Detailed and actionable error reporting: In the event of a failing test, the framework provides clear and actionable error messages, pinpointing the issue and facilitating rapid resolution.
- Mocking beyond service responses: Mocking a service response is a common strategy for unit testing, but it doesn’t fully test the integration with third parties, since the contract is assumed to be valid. This testing framework helps create a mock for the entire service, which can increase testing confidence by replicating the behavior of the API that users interact with.
- Cost-efficient testing environment: Having a dedicated testing environment or testing against production requires additional infrastructure and maintenance, which can lead to increased costs. By mocking the service layer, we can achieve the benefits of testing against a replica of the service without the additional costs.
- Avoid credential setup: One of the main challenges of working with production environment replicas is the need to setup credentials and authentication, which are user-dependent and thus not useful for testing purposes. Mocking the API service can help bypass this need, helping us focus on the behavior of the code samples themselves.
- Continuous monitoring: The framework can be set up to run as a monitoring tool that regularly checks code samples to ensure that they are still valid. This can help catch latent bugs and have an early alert when API changes break code samples, without the need for end users to report them.
- Usage in test-driven development (TDD): When making active development changes to the code generation tooling, having a testing framework can help guide design decisions and prevent breaking changes from happening, which in turn ensures the delivery of incremental value without the need for versioning.
Conclusion
By implementing a robust testing framework, the correctness and intended functionality of code samples within technical documentation can be assured, thereby empowering users and fostering trust in the provided resources. This framework not only enhances the reliability of the documentation but also significantly contributes to a positive user experience by minimizing potential errors and streamlining the adoption of the technology being documented.
If your company has a technical documentation site with thousands of code samples, contact us today and start testing smarter, not harder. We can help you create and test code snippets reliably.