Skip to content

Commit 5531985

Browse files
andy31415restyled-commitsandreilitvinbzbarsky-apple
authored and
pull[bot]
committed
Add compile-time code generation using zap for app-templates (#23962)
* Add compile-time code generation using zap for app-templates - add support for gn builds - add support for cmake builds - update esp32 build logic - update pre-generation script - split out app and idl codegen - update template used so zap_generate_all only generates IDLs for apps - updated CI unit tests * Fix tizen build dependencies for all-clusters and minimal * Remove one more unnedded include directory from tizen * Fix tv casting app include * Restyle * Bump up openiotsdk dockerfile to get zap-cli * Fix include for efr32 Lcd painter * Test out a zap installation workflow in darwin builds * Fix trailing slash * Multiline to make mac install more redable * make sure all subdirs are created when installing zap * Use sudo to create the zap install directory * unzip only zap-cli, to hopefully make the unzip faster * Install zap on all darwin CI runs * Add `"$(TEMP_DIR)/out/gen/src/controller/data_model/zapgen/"` to xcode project header search paths * Do not run clang-format for compile-time generated files * Update code generation documentation a bit * More documentation with examples of how to invoke and set pregeneration variables * Restyle * Fix typo in command for code pregen * Fix typo * Add a runnable check that is intended to codegen * make unit tests with golden images run (but not yet pass) * Update golden copies * Update documentation and restyle * Bump up build apps time... seems like bimodal timing distribution and this occasionally fails at 1h, leave to 1.5hrs for now * Fix spelling * make zap-cli --version run on mac as part of the install since I seem to get package conflict errors periodically * Try to use tempState for mac parallel runs * Restyle * Several code review changes * Fix typo in path for templates * Undo third party submodule changes * Move common python zap logic into one file, to have the zap path logic centralized and to be more easily be able to add version checks * Restyle * Add version check capabilities to zap execution and generate.py * Restyle * Update zap version and set skip_real_version based on merge with master * Also update darwin zap installation path * Update more paths with the new zap version * Restyle * Golden image update for regeneration * Do not restyle test outputs for zap generation tests * Restyle * Restyled by isort * Fix up one more include for obsolete paths * Update generated files for unit tests * Re-generate unit test output data * Fix telink contact sensor build * Update unit test data * Revert wrong diff marker * Bump CI docker to 0.6.31 to use the latest zap * Bump up image versions for non-chip-build images as well * Re-generate using latest zap from 0.6.31 image * Keep codegen for controller clusters, for darwin specifically * patch the matter xcode project to update the include path to use the darwin-spefici codegen for controller cluster paths * Restyle * Update to make template checker happy, update min version for zap execution * One more manual restyle attempt * Ran regen again for tests * Re-generated this time using CI steps for checking * See if CI can run zap tests on mac as well * Do not set skip real version for the version check * Add a better comment of why IMClusterCommandHandler is odd * Restyle * Start developing a simpler ZAP version update script as this is likely often used * Off by one fix and update search logic a bit * Add support for min version update as well * Update a comment and a min version * Comment updates * Another comment regarding version update * Fix gen files for unit tests. We should figure out why environment matters here ... this is odd * Update clang-format version comment: it is 15 not 14 now in pigweed * Force clang-format-15 or clang-format (assuming we use pigweed) * Regenerate in pigweed environment, so clang-format is consistently 15 * Report clang-format version when formatting, to understand any format differences * Re-gen all after clang-format-15 was applied * Remove zzz_generated from restyled.yaml since we clang-format already during zap gen * Zap regen app2 * Regenerated (after restyle changed permissions..) and updated logs to contain what files get formatted * Update CI image for coverage run * Serialize zap execution to work around zap darwin bugs * Fix copy and paste error for cmake: remove extra comma * Update version update to control what files to update: usage or docker * Update scripts/tools/zap/generate.py Co-authored-by: Boris Zbarsky <bzbarsky@apple.com> * Skip ZAP version check when ZAP_DEVELOPMENT_PATH is set * Restyle Co-authored-by: Restyled.io <commits@restyled.io> Co-authored-by: Andrei Litvin <andreilitvin@google.com> Co-authored-by: Boris Zbarsky <bzbarsky@apple.com>
1 parent 4d44b9e commit 5531985

File tree

380 files changed

+35539
-93679
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

380 files changed

+35539
-93679
lines changed

‎.github/workflows/bloat_check.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ jobs:
3030
runs-on: ubuntu-latest
3131

3232
container:
33-
image: connectedhomeip/chip-build:0.6.30
33+
image: connectedhomeip/chip-build:0.6.31
3434

3535
steps:
3636
- uses: Wandalen/wretry.action@v1.0.36

‎.github/workflows/build.yaml

+29-5
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ jobs:
3636
if: github.actor != 'restyled-io[bot]'
3737

3838
container:
39-
image: connectedhomeip/chip-build:0.6.30
39+
image: connectedhomeip/chip-build:0.6.31
4040
volumes:
4141
- "/tmp/log_output:/tmp/test_logs"
4242
options: --privileged --sysctl "net.ipv6.conf.all.disable_ipv6=0
@@ -138,7 +138,7 @@ jobs:
138138
if: github.actor != 'restyled-io[bot]'
139139

140140
container:
141-
image: connectedhomeip/chip-build:0.6.30
141+
image: connectedhomeip/chip-build:0.6.31
142142
volumes:
143143
- "/tmp/log_output:/tmp/test_logs"
144144
options: --privileged --sysctl "net.ipv6.conf.all.disable_ipv6=0
@@ -251,6 +251,14 @@ jobs:
251251
--pregen-dir ./zzz_pregenerated \
252252
build \
253253
"
254+
- name: Check no code generation in output
255+
run: |
256+
CNT=$(find -name "CHIPClusters.h" out | wc -l)
257+
if [ "${CNT}" != "0" ]; then
258+
echo "ERROR: found unexpected generated files:"
259+
find -name "CHIPClusters.h" out
260+
exit 1
261+
fi
254262
- name: Undo code pre-generation changes (make compile time codegen work again)
255263
run: |
256264
rm -rf ./zzz_pregenerated
@@ -290,7 +298,7 @@ jobs:
290298
if: github.actor != 'restyled-io[bot]'
291299

292300
container:
293-
image: connectedhomeip/chip-build:0.6.30
301+
image: connectedhomeip/chip-build:0.6.31
294302
volumes:
295303
- "/tmp/log_output:/tmp/test_logs"
296304
options: --sysctl "net.ipv6.conf.all.disable_ipv6=0
@@ -344,6 +352,12 @@ jobs:
344352
timeout-minutes: 200
345353
runs-on: macos-latest
346354
if: github.actor != 'restyled-io[bot]'
355+
356+
env:
357+
# NOTE: Generally kept in sync within the repo using
358+
# scripts/tools/zap/version_update.py
359+
ZAP_VERSION: v2023.01.09-nightly
360+
ZAP_INSTALL_PATH: /usr/local/zap/zap-v2023.01.09-nightly
347361

348362
steps:
349363
- uses: Wandalen/wretry.action@v1.0.36
@@ -374,6 +388,16 @@ jobs:
374388
ls -la /usr/local/Cellar/openssl@1.1
375389
OPEN_SSL_VERSION=`ls -la /usr/local/Cellar/openssl@1.1 | cat | tail -n1 | awk '{print $NF}'`
376390
ln -s /usr/local/Cellar/openssl@1.1/$OPEN_SSL_VERSION/lib/pkgconfig/* .
391+
- name: Install zap
392+
run: |
393+
sudo mkdir -p $ZAP_INSTALL_PATH
394+
sudo chown `whoami` $ZAP_INSTALL_PATH
395+
curl -L https://github.com/project-chip/zap/releases/download/${ZAP_VERSION}/zap-mac.zip \
396+
--output $ZAP_INSTALL_PATH/zap-mac.zip
397+
cd $ZAP_INSTALL_PATH
398+
unzip zap-mac.zip zap-cli
399+
rm zap-mac.zip
400+
./zap-cli --version
377401
- name: Bootstrap
378402
timeout-minutes: 25
379403
run: scripts/build/gn_bootstrap.sh
@@ -447,7 +471,7 @@ jobs:
447471
if: github.actor != 'restyled-io[bot]'
448472

449473
container:
450-
image: connectedhomeip/chip-build:0.6.30
474+
image: connectedhomeip/chip-build:0.6.31
451475
volumes:
452476
- "/tmp/log_output:/tmp/test_logs"
453477
options: --privileged --sysctl "net.ipv6.conf.all.disable_ipv6=0
@@ -478,4 +502,4 @@ jobs:
478502
- name: Run Build Coverage
479503
timeout-minutes: 20
480504
run: ./scripts/build_coverage.sh
481-
505+

‎.github/workflows/chef.yaml

+3-3
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ jobs:
2929
if: github.actor != 'restyled-io[bot]'
3030

3131
container:
32-
image: connectedhomeip/chip-build:0.6.30
32+
image: connectedhomeip/chip-build:0.6.31
3333
options: --user root
3434

3535
steps:
@@ -57,7 +57,7 @@ jobs:
5757
if: github.actor != 'restyled-io[bot]'
5858

5959
container:
60-
image: connectedhomeip/chip-build-esp32:0.6.30
60+
image: connectedhomeip/chip-build-esp32:0.6.31
6161
options: --user root
6262

6363
steps:
@@ -85,7 +85,7 @@ jobs:
8585
if: github.actor != 'restyled-io[bot]'
8686

8787
container:
88-
image: connectedhomeip/chip-build-nrf-platform:0.6.30
88+
image: connectedhomeip/chip-build-nrf-platform:0.6.31
8989
options: --user root
9090

9191
steps:

‎.github/workflows/cirque.yaml

+2-2
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ jobs:
2929
timeout-minutes: 90
3030

3131
env:
32-
DOCKER_RUN_VERSION: 0.6.30
32+
DOCKER_RUN_VERSION: 0.6.31
3333
GITHUB_CACHE_PATH: /tmp/cirque-cache/
3434

3535
runs-on: ubuntu-latest
@@ -38,7 +38,7 @@ jobs:
3838
# need to run with privilege, which isn't supported by job.XXX.contaner
3939
# https://github.com/actions/container-action/issues/2
4040
# container:
41-
# image: connectedhomeip/chip-build-cirque:0.6.30
41+
# image: connectedhomeip/chip-build-cirque:0.6.31
4242
# volumes:
4343
# - "/tmp:/tmp"
4444
# - "/dev/pts:/dev/pts"

‎.github/workflows/darwin-tests.yaml

+14
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,10 @@ jobs:
3939
env:
4040
BUILD_VARIANT: ${{matrix.build_variant}}
4141
LSAN_OPTIONS: detect_leaks=1 malloc_context_size=40 suppressions=scripts/tests/chiptest/lsan-mac-suppressions.txt
42+
# NOTE: Generally kept in sync within the repo using
43+
# scripts/tools/zap/version_update.py
44+
ZAP_VERSION: v2023.01.09-nightly
45+
ZAP_INSTALL_PATH: /usr/local/zap/zap-v2023.01.09-nightly
4246

4347
if: github.actor != 'restyled-io[bot]'
4448
runs-on: macos-latest
@@ -72,6 +76,16 @@ jobs:
7276
ls -la /usr/local/Cellar/openssl@1.1
7377
OPEN_SSL_VERSION=`ls -la /usr/local/Cellar/openssl@1.1 | cat | tail -n1 | awk '{print $NF}'`
7478
ln -s /usr/local/Cellar/openssl@1.1/$OPEN_SSL_VERSION/lib/pkgconfig/* .
79+
- name: Install zap
80+
run: |
81+
sudo mkdir -p $ZAP_INSTALL_PATH
82+
sudo chown `whoami` $ZAP_INSTALL_PATH
83+
curl -L https://github.com/project-chip/zap/releases/download/${ZAP_VERSION}/zap-mac.zip \
84+
--output $ZAP_INSTALL_PATH/zap-mac.zip
85+
cd $ZAP_INSTALL_PATH
86+
unzip zap-mac.zip zap-cli
87+
rm zap-mac.zip
88+
./zap-cli --version
7589
- name: Bootstrap
7690
timeout-minutes: 25
7791
run: scripts/build/gn_bootstrap.sh

‎.github/workflows/darwin.yaml

+16
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,12 @@ jobs:
3131
if: github.actor != 'restyled-io[bot]'
3232
runs-on: macos-latest
3333

34+
env:
35+
# NOTE: Generally kept in sync within the repo using
36+
# scripts/tools/zap/version_update.py
37+
ZAP_VERSION: v2023.01.09-nightly
38+
ZAP_INSTALL_PATH: /usr/local/zap/zap-v2023.01.09-nightly
39+
3440
steps:
3541
- uses: Wandalen/wretry.action@v1.0.36
3642
name: Checkout
@@ -52,6 +58,16 @@ jobs:
5258
ls -la /usr/local/Cellar/openssl@1.1
5359
OPEN_SSL_VERSION=`ls -la /usr/local/Cellar/openssl@1.1 | cat | tail -n1 | awk '{print $NF}'`
5460
ln -s /usr/local/Cellar/openssl@1.1/$OPEN_SSL_VERSION/lib/pkgconfig/* .
61+
- name: Install zap
62+
run: |
63+
sudo mkdir -p $ZAP_INSTALL_PATH
64+
sudo chown `whoami` $ZAP_INSTALL_PATH
65+
curl -L https://github.com/project-chip/zap/releases/download/${ZAP_VERSION}/zap-mac.zip \
66+
--output $ZAP_INSTALL_PATH/zap-mac.zip
67+
cd $ZAP_INSTALL_PATH
68+
unzip zap-mac.zip zap-cli
69+
rm zap-mac.zip
70+
./zap-cli --version
5571
- name: Bootstrap
5672
timeout-minutes: 25
5773
run: scripts/build/gn_bootstrap.sh

‎.github/workflows/doxygen.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ jobs:
8282

8383
runs-on: ubuntu-20.04
8484
container:
85-
image: connectedhomeip/chip-build-doxygen:0.6.30
85+
image: connectedhomeip/chip-build-doxygen:0.6.31
8686

8787
if: github.actor != 'restyled-io[bot]'
8888

‎.github/workflows/examples-ameba.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ jobs:
3535
if: github.actor != 'restyled-io[bot]'
3636

3737
container:
38-
image: connectedhomeip/chip-build-ameba:0.6.30
38+
image: connectedhomeip/chip-build-ameba:0.6.31
3939
options: --user root
4040

4141
steps:

‎.github/workflows/examples-bouffalolab.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ jobs:
3535
if: github.actor != 'restyled-io[bot]'
3636

3737
container:
38-
image: connectedhomeip/chip-build-bouffalolab:0.6.30
38+
image: connectedhomeip/chip-build-bouffalolab:0.6.31
3939
volumes:
4040
- "/tmp/bloat_reports:/tmp/bloat_reports"
4141
steps:

‎.github/workflows/examples-cc13x2x7_26x2x7.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ jobs:
3737
if: github.actor != 'restyled-io[bot]'
3838

3939
container:
40-
image: connectedhomeip/chip-build-ti:0.6.30
40+
image: connectedhomeip/chip-build-ti:0.6.31
4141
volumes:
4242
- "/tmp/bloat_reports:/tmp/bloat_reports"
4343
steps:

‎.github/workflows/examples-efr32.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ jobs:
3838
if: github.actor != 'restyled-io[bot]'
3939

4040
container:
41-
image: connectedhomeip/chip-build-efr32:0.6.30
41+
image: connectedhomeip/chip-build-efr32:0.6.31
4242
volumes:
4343
- "/tmp/bloat_reports:/tmp/bloat_reports"
4444
steps:

‎.github/workflows/examples-esp32.yaml

+10-2
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ jobs:
3434
if: github.actor != 'restyled-io[bot]'
3535

3636
container:
37-
image: connectedhomeip/chip-build-esp32:0.6.30
37+
image: connectedhomeip/chip-build-esp32:0.6.31
3838
volumes:
3939
- "/tmp/bloat_reports:/tmp/bloat_reports"
4040

@@ -93,6 +93,14 @@ jobs:
9393
build \
9494
--copy-artifacts-to out/artifacts \
9595
"
96+
- name: Check no code generation in output
97+
run: |
98+
CNT=$(find -name "CHIPClusters.h" out | wc -l)
99+
if [ "${CNT}" != "0" ]; then
100+
echo "ERROR: found unexpected generated files:"
101+
find -name "CHIPClusters.h" out
102+
exit 1
103+
fi
96104
- name: Undo code pregeneration changes
97105
run: |
98106
rm -rf ./zzz_pregenerated
@@ -140,7 +148,7 @@ jobs:
140148
if: github.actor != 'restyled-io[bot]'
141149

142150
container:
143-
image: connectedhomeip/chip-build-esp32:0.6.30
151+
image: connectedhomeip/chip-build-esp32:0.6.31
144152
volumes:
145153
- "/tmp/bloat_reports:/tmp/bloat_reports"
146154

‎.github/workflows/examples-infineon.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ jobs:
3535
if: github.actor != 'restyled-io[bot]'
3636

3737
container:
38-
image: connectedhomeip/chip-build-infineon:0.6.30
38+
image: connectedhomeip/chip-build-infineon:0.6.31
3939
volumes:
4040
- "/tmp/bloat_reports:/tmp/bloat_reports"
4141
steps:

‎.github/workflows/examples-k32w.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ jobs:
3737
if: github.actor != 'restyled-io[bot]'
3838

3939
container:
40-
image: connectedhomeip/chip-build-k32w:0.6.30
40+
image: connectedhomeip/chip-build-k32w:0.6.31
4141
volumes:
4242
- "/tmp/bloat_reports:/tmp/bloat_reports"
4343
steps:

‎.github/workflows/examples-linux-arm.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ jobs:
3434
if: github.actor != 'restyled-io[bot]'
3535

3636
container:
37-
image: connectedhomeip/chip-build-crosscompile:0.6.30
37+
image: connectedhomeip/chip-build-crosscompile:0.6.31
3838
volumes:
3939
- "/tmp/bloat_reports:/tmp/bloat_reports"
4040

‎.github/workflows/examples-linux-imx.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ jobs:
3434
if: github.actor != 'restyled-io[bot]'
3535

3636
container:
37-
image: connectedhomeip/chip-build-imx:0.6.30
37+
image: connectedhomeip/chip-build-imx:0.6.31
3838

3939
steps:
4040
- uses: Wandalen/wretry.action@v1.0.36

‎.github/workflows/examples-linux-standalone.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ jobs:
3434
if: github.actor != 'restyled-io[bot]'
3535

3636
container:
37-
image: connectedhomeip/chip-build:0.6.30
37+
image: connectedhomeip/chip-build:0.6.31
3838
volumes:
3939
- "/tmp/bloat_reports:/tmp/bloat_reports"
4040

‎.github/workflows/examples-mbed.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ jobs:
4040
if: github.actor != 'restyled-io[bot]'
4141

4242
container:
43-
image: connectedhomeip/chip-build-mbed-os:0.6.30
43+
image: connectedhomeip/chip-build-mbed-os:0.6.31
4444
volumes:
4545
- "/tmp/bloat_reports:/tmp/bloat_reports"
4646

‎.github/workflows/examples-mw320.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ jobs:
3737
if: github.actor != 'restyled-io[bot]'
3838

3939
container:
40-
image: connectedhomeip/chip-build:0.6.30
40+
image: connectedhomeip/chip-build:0.6.31
4141
volumes:
4242
- "/tmp/bloat_reports:/tmp/bloat_reports"
4343
steps:

‎.github/workflows/examples-nrfconnect.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ jobs:
3737
if: github.actor != 'restyled-io[bot]'
3838

3939
container:
40-
image: connectedhomeip/chip-build-nrf-platform:0.6.30
40+
image: connectedhomeip/chip-build-nrf-platform:0.6.31
4141
volumes:
4242
- "/tmp/bloat_reports:/tmp/bloat_reports"
4343

‎.github/workflows/examples-openiotsdk.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ jobs:
3535
if: github.actor != 'restyled-io[bot]'
3636

3737
container:
38-
image: connectedhomeip/chip-build-openiotsdk:0.6.30
38+
image: connectedhomeip/chip-build-openiotsdk:0.6.31
3939
volumes:
4040
- "/tmp/bloat_reports:/tmp/bloat_reports"
4141
options: --privileged

‎.github/workflows/examples-qpg.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ jobs:
3737
if: github.actor != 'restyled-io[bot]'
3838

3939
container:
40-
image: connectedhomeip/chip-build:0.6.30
40+
image: connectedhomeip/chip-build:0.6.31
4141
volumes:
4242
- "/tmp/bloat_reports:/tmp/bloat_reports"
4343
steps:

‎.github/workflows/examples-telink.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ jobs:
3535
if: github.actor != 'restyled-io[bot]'
3636

3737
container:
38-
image: connectedhomeip/chip-build-telink:0.6.30
38+
image: connectedhomeip/chip-build-telink:0.6.31
3939
volumes:
4040
- "/tmp/bloat_reports:/tmp/bloat_reports"
4141

‎.github/workflows/examples-tizen.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ jobs:
3333
if: github.actor != 'restyled-io[bot]'
3434

3535
container:
36-
image: connectedhomeip/chip-build-tizen:0.6.30
36+
image: connectedhomeip/chip-build-tizen:0.6.31
3737
options: --user root
3838
volumes:
3939
- "/tmp/bloat_reports:/tmp/bloat_reports"

‎.github/workflows/full-android.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ jobs:
3737
if: github.actor != 'restyled-io[bot]'
3838

3939
container:
40-
image: connectedhomeip/chip-build-android:0.6.30
40+
image: connectedhomeip/chip-build-android:0.6.31
4141
volumes:
4242
- "/tmp/log_output:/tmp/test_logs"
4343

0 commit comments

Comments
 (0)