Migrating regression suite to iree-test-suite #20194

geomin12 · 2025-03-07T20:38:51Z

Progress on iree-org/iree-test-suites#81 and #19115.

This moves regression test logic over to iree-test-suites and calls those regression test files with JSON configuration files from this repository.

Quality test JSON configuration files are now at tests/external/iree-test-suites/sharktank_models/quality_tests/
Benchmark test JSON configuration are now at tests/external/iree-test-suites/sharktank_models/benchmarks/
Test file configurations (such as mlir files) are now at tests/external/iree-test-suites/test_suite_files/

For previous PR reference: #20124

geomin12 · 2025-03-07T21:00:31Z

iree-test-suite PR 87 will fix the build issue :)

build_tools/pkgci/external_test_suite/sdxl_pipeline_bench_f16.mlir

docs/website/docs/developers/general/testing-guide.md

ScottTodd · 2025-03-07T20:53:21Z

tests/external/iree-test-suites/sharktank_models/quality_tests/sd3/clip_cpu.json

+    "inputs": [
+        {
+            "source": "https://sharkpublic.blob.core.windows.net/sharkpublic/sai/sd3-prompt-encoder/inference_input.0.bin",
+            "value": "1x77x2xi64"
+        },


(I think this is tracked in iree-org/iree-test-suites#81, fine to continue as-is for now)

These can be moved to git LFS in the test suite repo.

File paths should not have usernames in them (sai here)

Files should be editable by any developer (send a PR, get it approved, merge it), not just those with permission to work in this cloud project

ScottTodd · 2025-03-07T20:55:22Z

tests/external/iree-test-suites/sharktank_models/benchmarks/sdxl/vae_rocm.json

+        "mi300": 1.1,
+        "mi308": 1.1
+    },
+    "device": "hip",


(I think this is tracked in iree-org/iree-test-suites#81, fine to continue as-is for now)

This could maybe be represented via general compile_flags and run_flags instead of this potentially magic "device" setting that the test suite uses to change behavior.

I would also group the compile/run/benchmark flags in the config file before the benchmark expectations (golden_*)

Thinking of how developers interact with the benchmarks, we might also want to fold the metrics into a smaller set of files somehow, so developers don't need to search through files

which file had the golden_time_ms for this test failure?

Moving to an automatically managed database instead of manually edited source files would also solve that problem.

.github/workflows/pkgci_sharktank_regression.yml

geomin12 · 2025-03-07T23:49:30Z

Looks like there's an issue with iree-test-suite, will fix that

krzysz00 · 2025-03-08T23:37:12Z

Wanted to flag to make sure the recently added RDNA4 tests didn't mysteriously disappear on us

geomin12 · 2025-03-10T03:04:48Z

Wanted to flag to make sure the recently added RDNA4 tests didn't mysteriously disappear on us

hi, from the looks of it, the RDNA4 tests look safe to me! i'll keep note and make sure those new tests are not deleted!

geomin12 · 2025-04-02T16:58:39Z

@aviator19941 tagged so you'll get a notif when this goes through to add llama 🦙

ScottTodd

Cool! The setup here looks good and the logs are showing the right tests and benchmarks running and passing.

.github/workflows/pkgci_sharktank_regression.yml

.github/workflows/pkgci.yml

.github/workflows/pkgci_sharktank_regression.yml

aviator19941

LGTM, @ScottTodd can you do the final review?

ScottTodd

Nice! Can you make a pass through the PR description and add a bit more context, like

A link to the code in iree-test-suites
Instructions for where the new config files are located in this repository, for when benchmark numbers change and developers need to update them
Drop "via composite action"
Link to any related issues (I thought we had one filed for this migration)

.github/workflows/pkgci_sharktank.yml

.github/workflows/pkgci.yml

geomin12 · 2025-04-03T18:48:22Z

@ScottTodd merge this through whenever you can! (after the tests all pass too)

ScottTodd · 2025-04-03T19:31:51Z

Thanks. I made some edits to the PR description and will merge now, without waiting for checks. The CI has a 2 hour queue right now due to only having a single w7900 runner: https://github.com/iree-org/iree/actions?query=is%3Aqueued.

Progress on iree-org/iree-test-suites#81 and iree-org#19115. This moves regression test logic over to [iree-test-suites](https://github.com/iree-org/iree-test-suites) and calls those regression test files with JSON configuration files from this repository. * Quality test JSON configuration files are now at `tests/external/iree-test-suites/sharktank_models/quality_tests/` * Benchmark test JSON configuration are now at `tests/external/iree-test-suites/sharktank_models/benchmarks/` * Test file configurations (such as mlir files) are now at `tests/external/iree-test-suites/test_suite_files/` For previous PR reference: iree-org#20124

krzysz00 · 2025-04-09T22:40:17Z

Hi - rather late followup question: where'd test_unet.py go? I'm trying to test the performance impact of a change against SDXL and that used to be

# Compile the model, downloading parameters if needed
pytest ../iree/experimental/regression_suite/shark-test-suite-models/sdxl/test_unet.py
# Actually run the benchmark
env ROCR_VISIBLE_DEVICES=4 iree-benchmark-module --device=hip --device_allocator=caching --module=sdxl_unet_fp16_vmfbs/model.rocm_gfx942.vmfb --parameters=model=artifacts/sdxl_unet_fp16/real_weights.irpa --function=run_forward --input=1x4x128x128xf16 --input=2x64x2048xf16 --input=2x1280xf16 --input=2x6xf16 --input=1xf16 --input=1xi64 --benchmark_repetitions=10 --benchmark_min_warmup_time=3.0

adjusting for the architecture, data type, etc.

What's the replacement for this?

geomin12 · 2025-04-09T23:59:55Z

Hi - rather late followup question: where'd test_unet.py go? I'm trying to test the performance impact of a change against SDXL and that used to be

# Compile the model, downloading parameters if needed
pytest ../iree/experimental/regression_suite/shark-test-suite-models/sdxl/test_unet.py
# Actually run the benchmark
env ROCR_VISIBLE_DEVICES=4 iree-benchmark-module --device=hip --device_allocator=caching --module=sdxl_unet_fp16_vmfbs/model.rocm_gfx942.vmfb --parameters=model=artifacts/sdxl_unet_fp16/real_weights.irpa --function=run_forward --input=1x4x128x128xf16 --input=2x64x2048xf16 --input=2x1280xf16 --input=2x6xf16 --input=1xf16 --input=1xi64 --benchmark_repetitions=10 --benchmark_min_warmup_time=3.0

adjusting for the architecture, data type, etc.

What's the replacement for this?

hi! so the logic for compiling and running tests are here and benchmark tests are here. we run the tests inside the iree repo, and we pull data from these JSON configuration files: quality JSON configs , benchmark JSON config

in terms of architecture, we pull that data from this github action matrix

krzysz00 · 2025-04-10T16:30:44Z

Ok, so, further questions, because I realized part of my issue: why's iree/tests/external/iree-test-suites out of sync with https://github.com:/ree-org/iree-test-suites ?

krzysz00 · 2025-04-10T16:31:15Z

(I'm specifically looking at the quality_tests directory)

geomin12 · 2025-04-10T16:38:11Z

Ok, so, further questions, because I realized part of my issue: why's iree/tests/external/iree-test-suites out of sync with https://github.com:/ree-org/iree-test-suites ?

ah, so the JSON config in iree-test-suites are just for example. we only have the generic ubuntu github runners for iree-test-suites, so we can't run any rocm tests

Our thoughts for this is that any user can use iree-test-suites code (locally or in another repo) to test any JSON configuration, and the iree repo is just a "user" of the iree-test-suite code to test their own JSON configuration setup (sdxl, sd3, etc)

ScottTodd · 2025-04-10T16:41:45Z

We can also add some quality of life scripts for cloning the test suites repo and running tests using the configs from the iree repo. Or just more documentation.

ScottTodd · 2025-04-10T18:38:57Z

we only have the generic ubuntu github runners for iree-test-suites, so we can't run any rocm tests

BTW we do have mi300 runners from ossci accessible in iree-test-suites now. We should be able to add some rocm tests there for easier development of the test suite itself.

krzysz00 · 2025-04-10T19:51:48Z

To answer my own question

# make a directory, active the iree venv in it (with your iree build editably installed), install iree-test-suites/sharktank-models into said venv editably are the one-time steps
export BACKEND=rocm  
export TEST_OUTPUT_ARTIFACTS=$PWD  
pytest ../iree-test-suites/sharktank_models/quality_tests/ -rpfE --log-cli-level=info --test-file-directory=../iree/tests/external/iree-test-suites/sharktank_models/quality_tests --external-file-directory=../iree/build_tools/pkgci/external_test_suite

I'll note that said "external file directory" doesn't seem to exist, but specifying it doesn't hurt

geomin12 added 3 commits March 7, 2025 12:30

Adding test-suite migration

f877d8d

Adding test-suite migration

d079eea

here

fbfc597

geomin12 requested review from ScottTodd, benvanik and stellaraccident as code owners March 7, 2025 20:38

Fixing lint

b13701e

Updated version

9e63e93

ScottTodd reviewed Mar 7, 2025

View reviewed changes

ScottTodd added infrastructure Relating to build systems, CI, or testing infrastructure/benchmark Relating to benchmarking infrastructure labels Mar 7, 2025

PR comments

c64565b

geomin12 added 14 commits March 11, 2025 14:46

Fixing compile flags for sd3

5787e46

fixing merge conflict

eae8e99

Updating json files from merge and adding iree test files

641c249

try env variable

0dd2ae5

strangeeeee behavior, testing this

78f7f53

adding hard coded bc that is where it is stored

09225ab

trying this

4135749

trying this sha

c413787

fixing lfs

ce30c81

okay let us see if this works this is nuts

aa8e012

actually retrieving the lfs files

4dc6034

push again

22c2443

fixing error

1e03bb6

new sha bc i cannot workflow dispatch

181732b

geomin12 added 4 commits March 31, 2025 16:45

Trying without composite actions

a2aee63

merge conflict

b0c2673

Adding new sha for cpu benchmkark test

40918e0

Adding proper sha

68298ae

ScottTodd reviewed Apr 2, 2025

View reviewed changes

geomin12 added 6 commits April 2, 2025 11:18

PR comments

70dc6d4

fixing lint

beeb648

Adding cpu fix

a3b715b

Adding default for rocm_hip_w7900

b595d1f

Adding default workspace for iree test files

2bad4b1

Adding separate job for w7900

f8c7bfb

geomin12 requested a review from ScottTodd April 2, 2025 22:37

aviator19941 reviewed Apr 3, 2025

View reviewed changes

ScottTodd approved these changes Apr 3, 2025

View reviewed changes

.github/workflows/pkgci_sharktank.yml Outdated Show resolved Hide resolved

ScottTodd reviewed Apr 3, 2025

View reviewed changes

.github/workflows/pkgci.yml Outdated Show resolved Hide resolved

PR comment

feca05d

ScottTodd merged commit fcd847a into iree-org:main Apr 3, 2025
41 checks passed

geomin12 deleted the test-suite-migration-clean branch April 3, 2025 20:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrating regression suite to iree-test-suite #20194

Migrating regression suite to iree-test-suite #20194

geomin12 commented Mar 7, 2025 •

edited by ScottTodd

Loading

geomin12 commented Mar 7, 2025

ScottTodd Mar 7, 2025

ScottTodd Mar 7, 2025

geomin12 commented Mar 7, 2025

krzysz00 commented Mar 8, 2025

geomin12 commented Mar 10, 2025

geomin12 commented Apr 2, 2025

ScottTodd left a comment

aviator19941 left a comment

ScottTodd left a comment

geomin12 commented Apr 3, 2025 •

edited

Loading

ScottTodd commented Apr 3, 2025

krzysz00 commented Apr 9, 2025

geomin12 commented Apr 9, 2025

krzysz00 commented Apr 10, 2025

krzysz00 commented Apr 10, 2025

geomin12 commented Apr 10, 2025

ScottTodd commented Apr 10, 2025

ScottTodd commented Apr 10, 2025

krzysz00 commented Apr 10, 2025

Migrating regression suite to iree-test-suite #20194

Migrating regression suite to iree-test-suite #20194

Conversation

geomin12 commented Mar 7, 2025 • edited by ScottTodd Loading

geomin12 commented Mar 7, 2025

ScottTodd Mar 7, 2025

Choose a reason for hiding this comment

ScottTodd Mar 7, 2025

Choose a reason for hiding this comment

geomin12 commented Mar 7, 2025

krzysz00 commented Mar 8, 2025

geomin12 commented Mar 10, 2025

geomin12 commented Apr 2, 2025

ScottTodd left a comment

Choose a reason for hiding this comment

aviator19941 left a comment

Choose a reason for hiding this comment

ScottTodd left a comment

Choose a reason for hiding this comment

geomin12 commented Apr 3, 2025 • edited Loading

ScottTodd commented Apr 3, 2025

krzysz00 commented Apr 9, 2025

geomin12 commented Apr 9, 2025

krzysz00 commented Apr 10, 2025

krzysz00 commented Apr 10, 2025

geomin12 commented Apr 10, 2025

ScottTodd commented Apr 10, 2025

ScottTodd commented Apr 10, 2025

krzysz00 commented Apr 10, 2025

geomin12 commented Mar 7, 2025 •

edited by ScottTodd

Loading

geomin12 commented Apr 3, 2025 •

edited

Loading