Requirements document for on-device testing (#1161)

[szatmz]

Problem

Specification of on-device test protocols for development support and certification.

Summary of Changes

Test framework, dispatching, device interaction,
result collection are discussed.

• Status: first draft

fixes #1161

[gerickson]

We should always support Linux, macOS with BSD Sockets + LwIP (+ Network.framework for macOS) variants as well, at all times. If possible, we should also add iOS and Android, where it makes sense and where possible.

[chip-bot]

Size increase report for "Build Examples [ESP32]"

File	Section	File	VM
chip-wifi-echo.elf	.iram0.text	-5796	-5796
chip-wifi-echo.elf	.dram0.bss	0	-5912
chip-wifi-echo.elf	.dram0.data	-12244	-12244
chip-wifi-echo.elf	.flash.rodata	-86048	-86048
chip-wifi-echo.elf	[1672 Others]	-182848	-4
chip-wifi-echo.elf	.flash.text	-213336	-213336

Full report output

Bloat report for job 'Build Examples [ESP32]'

Files found only in the baseline:
    bloat_report.txt

Comparing master_binaries/esp32-build/chip-wifi-echo.elf and example_binaries/esp32-build/chip-wifi-echo.elf:

sections,vmsize,filesize
.xt.prop._ZNKSt9money_getIcSt19istreambuf_iteratorIcSt11char_traitsIcEEE10_M_extractILb0EEES3_S3_S3_RSt8ios_baseRSt12_Ios_IostateRSs,0,-1072
.xt.prop._ZNKSt9money_getIcSt19istreambuf_iteratorIcSt11char_traitsIcEEE10_M_extractILb1EEES3_S3_S3_RSt8ios_baseRSt12_Ios_IostateRSs,0,-1072
.xt.prop._ZNKSt9money_getIwSt19istreambuf_iteratorIwSt11char_traitsIwEEE10_M_extractILb0EEES3_S3_S3_RSt8ios_baseRSt12_Ios_IostateRSs,0,-1072
[Unmapped],0,-2407
.iram0.text,-5796,-5796
.dram0.bss,-5912,0
.dram0.data,-12244,-12244
.debug_aranges,0,-35448
.symtab,0,-80160
.flash.rodata,-86048,-86048
.debug_frame,0,-99396
.debug_ranges,0,-104968
.strtab,0,-131901
.shstrtab,0,-173575
[1672 Others],-4,-182848
.flash.text,-213336,-213336
.debug_abbrev,0,-215292
.debug_str,0,-501192
.debug_loc,0,-719458
.debug_line,0,-1034660
.debug_info,0,-3242359

[chip-bot]

Size increase report for "Build Examples [nRF]"

File	Section	File	VM

Full report output

Bloat report for job 'Build Examples [nRF]'

Files found only in the baseline:
    bloat_report.txt

Comparing master_binaries/nrf-build/chip-nrf52840-lock-example.out and example_binaries/nrf-build/chip-nrf52840-lock-example.out:

sections,vmsize,filesize
[Unmapped],0,2
.debug_line,0,-36
.debug_info,0,-65
.debug_macro,0,-328
.debug_str,0,-5989

[chip-bot]

Size increase report for "Build Examples [main-build]"

File	Section	File	VM

Full report output

Bloat report for job 'Build Examples [main-build]'

Files found only in the baseline:
    bloat_report.txt

Comparing master_binaries/main-build/chip-standalone-demo.out and example_binaries/main-build/chip-standalone-demo.out:

sections,vmsize,filesize
[Unmapped],0,3
.debug_line,0,-40
.debug_info,0,-102
.debug_macro,0,-586
.debug_str,0,-6019

[woody-apple]

@saurabhst @jelderton ?