Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: adding videos to CI report artifact #8990

Merged
merged 2 commits into from
Mar 21, 2025
Merged

test: adding videos to CI report artifact #8990

merged 2 commits into from
Mar 21, 2025

Conversation

jordanl17
Copy link
Member

@jordanl17 jordanl17 commented Mar 19, 2025

Description

  • Includes videos in the playwright report artifact generated during CI
  • Creates a comment on each PR with a link to download the report artifact

What to review

Testing

Scroll down and see the comment, or click here.
Open up the report, or try this one generated already, and navigate to one of the flakey tests to see that a video recording is available

Notes for release

N/A - relates only to DX on Studio

Copy link

vercel bot commented Mar 19, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
page-building-studio ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 20, 2025 3:27pm
performance-studio ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 20, 2025 3:27pm
test-studio ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 20, 2025 3:27pm
2 Skipped Deployments
Name Status Preview Comments Updated (UTC)
studio-workshop ⬜️ Ignored (Inspect) Visit Preview Mar 20, 2025 3:27pm
test-next-studio ⬜️ Ignored (Inspect) Mar 20, 2025 3:27pm

Copy link
Contributor

No changes to documentation

Copy link
Contributor

github-actions bot commented Mar 19, 2025

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 42.98% 55546 / 129224
🔵 Statements 42.98% 55546 / 129224
🔵 Functions 47.43% 2783 / 5867
🔵 Branches 79.57% 10428 / 13105
File CoverageNo changed files found.
Generated in workflow #32583 for commit 7576031 by the Vitest Coverage Report Action

Copy link
Contributor

github-actions bot commented Mar 19, 2025

⚡️ Editor Performance Report

Updated Thu, 20 Mar 2025 15:40:18 GMT

Benchmark reference
latency of sanity@latest
experiment
latency of this branch
Δ (%)
latency difference
article (title) 16.1 efps (62ms) 22.2 efps (45ms) -17ms (-27.4%)
article (body) 72.2 efps (14ms) 63.3 efps (16ms) +2ms (-/-%)
article (string inside object) 18.3 efps (55ms) 22.7 efps (44ms) -11ms (-19.3%)
article (string inside array) 16.9 efps (59ms) 20.8 efps (48ms) -11ms (-18.6%)
recipe (name) 30.3 efps (33ms) 30.3 efps (33ms) +0ms (-/-%)
recipe (description) 33.3 efps (30ms) 33.3 efps (30ms) +0ms (-/-%)
recipe (instructions) 99.9+ efps (5ms) 99.9+ efps (5ms) +0ms (-/-%)
synthetic (title) 18.5 efps (54ms) 19.2 efps (52ms) -2ms (-3.7%)
synthetic (string inside object) 18.5 efps (54ms) 20.4 efps (49ms) -5ms (-9.3%)

efps — editor "frames per second". The number of updates assumed to be possible within a second.

Derived from input latency. efps = 1000 / input_latency

Detailed information

🏠 Reference result

The performance result of sanity@latest

Benchmark latency p75 p90 p99 blocking time test duration
article (title) 62ms 68ms 97ms 342ms 1821ms 13.7s
article (body) 14ms 15ms 17ms 126ms 181ms 5.3s
article (string inside object) 55ms 58ms 70ms 270ms 1260ms 9.6s
article (string inside array) 59ms 63ms 69ms 181ms 1531ms 9.5s
recipe (name) 33ms 35ms 41ms 50ms 55ms 9.6s
recipe (description) 30ms 31ms 33ms 53ms 0ms 6.2s
recipe (instructions) 5ms 7ms 8ms 18ms 0ms 3.1s
synthetic (title) 54ms 61ms 70ms 263ms 958ms 13.7s
synthetic (string inside object) 54ms 61ms 78ms 374ms 1121ms 8.6s

🧪 Experiment result

The performance result of this branch

Benchmark latency p75 p90 p99 blocking time test duration
article (title) 45ms 77ms 91ms 383ms 763ms 12.1s
article (body) 16ms 17ms 25ms 147ms 213ms 6.0s
article (string inside object) 44ms 46ms 55ms 90ms 428ms 8.0s
article (string inside array) 48ms 51ms 53ms 159ms 362ms 7.6s
recipe (name) 33ms 35ms 37ms 75ms 47ms 9.3s
recipe (description) 30ms 32ms 39ms 65ms 45ms 6.3s
recipe (instructions) 5ms 7ms 7ms 10ms 0ms 3.1s
synthetic (title) 52ms 57ms 64ms 375ms 1398ms 13.4s
synthetic (string inside object) 49ms 51ms 57ms 449ms 1072ms 8.7s

📚 Glossary

column definitions

  • benchmark — the name of the test, e.g. "article", followed by the label of the field being measured, e.g. "(title)".
  • latency — the time between when a key was pressed and when it was rendered. derived from a set of samples. the median (p50) is shown to show the most common latency.
  • p75 — the 75th percentile of the input latency in the test run. 75% of the sampled inputs in this benchmark were processed faster than this value. this provides insight into the upper range of typical performance.
  • p90 — the 90th percentile of the input latency in the test run. 90% of the sampled inputs were faster than this. this metric helps identify slower interactions that occurred less frequently during the benchmark.
  • p99 — the 99th percentile of the input latency in the test run. only 1% of sampled inputs were slower than this. this represents the worst-case scenarios encountered during the benchmark, useful for identifying potential performance outliers.
  • blocking time — the total time during which the main thread was blocked, preventing user input and UI updates. this metric helps identify performance bottlenecks that may cause the interface to feel unresponsive.
  • test duration — how long the test run took to complete.

Copy link
Contributor

github-actions bot commented Mar 20, 2025

📊 Playwright Test Report

Download Full E2E Report

This report contains test results, including videos of failing tests.

echo "url=$URL" >> $GITHUB_OUTPUT

- name: Add PR Comment with Report Link
uses: thollander/actions-comment-pull-request@fabd468d3a1a0b97feee5f6b9e499eab0dd903f6 # v2
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is roughly the same step that was used when we created a GH PR comment for the playwright component tests - which can be seen here

@jordanl17 jordanl17 enabled auto-merge (squash) March 20, 2025 18:16
Copy link
Contributor

@pedrobonamin pedrobonamin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great, thank you @jordanl17 !

@jordanl17 jordanl17 merged commit a5d7296 into next Mar 21, 2025
61 checks passed
@jordanl17 jordanl17 deleted the test/e2e-videos branch March 21, 2025 07:17
bjoerge added a commit that referenced this pull request Mar 21, 2025
* next: (81 commits)
  fix(presentation): use `perspectiveStack` when resolving main document (#9005)
  fix(core): hide discard changes action when published (#9009)
  fix(sanity): treat releases as inactive if releases tool not present (#9001)
  test(e2e): add unarchive / unpublish tests to releases (#8979)
  test: adding videos to CI report artifact (#8990)
  fix(cli): use outpath as package name for apps (#9003)
  fix(core): drop explicit `withCredentials` on requests (#8996)
  chore: add presentation to regular workspace (#8997)
  fix(core): use publishedId for search intent links (#8980)
  chore: fix vitest snapshot client error (#8993)
  chore: add vitest typecheck support (#8994)
  fix(core): stickyParams for `__unsafe_disableScopedSearchParams` (#8987)
  fix: no releases screen shows the create release dialog (#8988)
  fix(presentation): previewUrl is required (#8995)
  fix(deps): update dependency @portabletext/editor to ^1.40.3 (#8992)
  fix(deps): update dependency @portabletext/block-tools to ^1.1.14 (#8991)
  feat(core): add version to documents when creating with a release pinned (#8974)
  fix(test): update flaky navbar test (#8984)
  chore: add simple e2e presentation test (#8982)
  chore(deps): update dev-non-major (#8983)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants