Requirements document for on-device testing (#1161)

docs/api/device_runner.md

@@ -0,0 +1,103 @@
+# CHIP on-device testing
+
+_Requirements and high-level design_
+
+## Background
+
+The ability to run tests on actual and emulated hardware is paramount in
+embedded projects. CHIP is no exception. We want on-device testing to be a first
+class goal of CHIP architecture. On-device testing requirements apply both to
+Continuous Integration testing for main CHIP software stack development and to
+eventual CHIP product certification. This document explores the requirements and
+evaluates potential solutions.
+
+## Overview of requirements
+
+A good device test infrastructure is built on four pillars.
+
+### Pillar 1: Using a test framework
+
+A test framework provides a testing structure that developers can follow and
+potentially reduces some of the burden of test setup and teardown (less
+boilerplate). Support for state-oriented and asynchronous structuring of tests
+would be beneficial. Many test frameworks leverage scripting languages such as
+Python to simplify the quick development of tests and to leverage rich sets of
+libraries for device/systems access and results generation.
+
+### Pillar 2: Dispatching tests
+
+Tests can run on lab machines or on the developer's local workstation. Tests can
+be triggered manually by the developer or as a result of completion of a
+changeset built on a continuous integration (CI) server. CHIP involves multiple
+stakeholders, many of which will want to contribute to the testing efforts with
+lab capacity. The infrastructure therefore must be prepared for
+cross-organization test dispatch.
+
+To facilitate uniform dispatch of tests we will probably need a simple
+request/response protocol. Potentially HTTPS based and RESTful. Due to the long
+running nature of device tests the response for a test scheduling request could
+be a test ID, not the test result. That ID could be used to query the test
+status, subscribe for notifications on status changes and to pull the test
+results. Core aspects of such a scheme include the conventions for request
+artifacts contents and minimum expected results contents once the run is
+complete.
+
+### Pillar 3: Interacting with devices
+
+The test host environment has to reset devices, flash images on them, issue
+commands, monitor status and collect test results. It may also need to integrate
+both virtual (simulated) and real devices together. This can at first be done in
+an ad-hoc way per platform but eventually we can go into device access
+abstraction, i.e. define a common device testing interface which CHIP-compliant
+devices can expose. The test host has to be prepared for driving multiple
+devices at the same time for a single test, e.g. for tests that check
+communication between multiple devices.
+
+### Pillar 4: Collecting results
+
+Ideally, test results are output in standard formats and similar or analogous
+results between different devices and tests are output the same way. This
+ensures reusability of code that processes similar data while allowing
+aggregation of results across different dimensions. Failed tests must propagate
+errors from device platform layers all the way to the CHIP stack and present
+errors and potential stack traces in a standard result format. As the purpose of
+on-device tests is to capture bugs, it is important that the test outputs
+highlight the failure reason(s) and developers don't have to browse through
+thousands of lines of logs to find the one line that sheds light on why a test
+failed.
+
+## Priorities
+
+In the spirit of CHIP's charter, it would be great to see something taking-off
+as soon as possible, to support continuous testing of the evolving CHIP stack.
+We could then improve on that first iteration, even if we have to throw away
+some temporary concepts and code.
+
+Test dispatch (Pillar 2) arises as the highest priority, because all other
+pillars can have ad-hoc solutions. The first need is an interface between a
+CircleCI job and a test execution host at a participating organization. This
+would enable dispatching tests to a variety of existing in-house infrastructure,
+while retaining common request/response protocols to shield the CI system from
+implementation details of each lab.
+
+The next most important goal is to provide a test framework (Pillar 1). With a
+standard framework developers can start writing tests, even if those tests will
+be device specific and of ad-hoc input and output format. The general structure
+of tests will however be present and later the tests can be adapted to standard
+interactions (Pillar 3) and result formats (Pillar 4).
+
+Specifying result formats (Pillar 4) for the most common outputs
+(success/failure, failure reason, stack trace, memory and CPU usage time series,
+pcaps of network traffic, etc.) will be an ongoing effort. The simplest output
+formats can be specified together with the test framework.
+
+Lastly, we want to look into a common device interaction interface that would
+enable reusing tests between different devices.
+
+## Baseline hardware platforms for CHIP
+
+The TSG is targeting the following platforms/boards for early bringup:
+
+-   Nordic nRF52 board <TODO: REF>
+-   SiLabs XXXX board <TODO:REF>
+-   Espressif ESP32 XXXX board <TODO:REF>