[BUG]: Cypress integration does not work with the --specs option

[fcano-ut]

Tracer Version(s)

5.35.0

Node.js Version(s)

20.18

Bug Report

We do test parallelization, running tests with code like this:

CYPRESS_CACHE_FOLDER=$CYPRESS_CACHE cypress run -C ./cypress/config/cypress.specs.config.ts -b chrome --config baseUrl=http://localhost:3000,trashAssetsBeforeRuns=false --spec $TESTS

$TESTS is an array with a series of comma-separated files to run on each matrix task.

After configuring the Cypress integration, we don't seem to get report on individual tests. instead everything is grouped into "suites"

The suite's name is that of the first test ran for the given job, so data is arbitrarily grouped into "suites" whose name only represent the first test ran.

Also I'd expect the "tests" tab to be populated, but it's not.

Feels like the plugin is only working "partially" and not doing everything it should be doing 🤔

Reproduction Code

No response

Error Logs

I've tried running with DD_TRACE_DEBUG: true and DD_TRACE_LOG_LEVEL: error, which adds some warnings:

Warning: Package '@opentelemetry/sdk-trace-node' was loaded before dd-trace! This may break instrumentation.
Warning: Package 'lodash' was loaded before dd-trace! This may break instrumentation.
Warning: Package 'bluebird' was loaded before dd-trace! This may break instrumentation.
Warning: Package 'cypress' was loaded before dd-trace! This may break instrumentation.
Warning: Please ensure dd-trace is loaded before other modules.

tput: No value for $TERM and no -T specified

Tracer Config

All DD imports were added at the beginning of the files

# support file
import 'dd-trace/ci/cypress/support';
// ...

# specs/plugins.ts

import ddTraceCiPlugin from 'dd-trace/ci/cypress/plugin';
// ...

module.exports = (on, config) => {
  ddTraceCiPlugin(on, config);
  // ... other plugins

These variables were added to the github action that runs tests

          DD_CIVISIBILITY_AGENTLESS_ENABLED: true
          DD_API_KEY: ${{ secrets.DD_API_KEY }}
          DD_SITE: datadoghq.com
          DD_TRACE_DEBUG: true
          DD_TRACE_LOG_LEVEL: error

Operating System

GitHub Runner (Ubuntu)

Bundling

Unsure

[juan-fernandez]

hey @fcano-ut ! Thanks for the report.

Can you confirm that you followed instructions in https://docs.datadoghq.com/tests/setup/javascript/?tab=cypress#cypress-version-10-or-later?

There are at least two files you need to modify:

cypress.config.js, which has the global configuration for cypress:

const { defineConfig } = require('cypress')

module.exports = defineConfig({
  e2e: {
    setupNodeEvents: require('dd-trace/ci/cypress/plugin'), // this is the DD bit
    supportFile: 'cypress/support/e2e.js'
  }, 
  // .. config continues here
})

Additionally you need to add this line to your supportFile (in this example, this would be cypress/support/e2e.js):

require('dd-trace/ci/cypress/support')
// Also supported:
// import 'dd-trace/ci/cypress/support'
// Your code can be after this line
// Cypress.Commands.add('login', (email, pw) => {})

From your comment it seems the supportFile config ('dd-trace/ci/cypress/support') is missing

[fcano-ut]

I'm sorry @juan-fernandez , I got confused when copying the examples, but definitely this change is made in the support file:

import 'dd-trace/ci/cypress/support';
// ...

As a side-note, we might not be blocked by this bug. I'm not sure yet. We have a Knapsack Pro subscription and I'll try to parallelize tests using the Cypress integration, which might end up fixing our problem 🤔

[fcano-ut]

Temporarily all I can say is that using Knapsack Pro, I can bypass this issue. However I still have the same problem where the "tests" tab is empty, only filtering by "suite" shows the list of tests. The Fail or Success status of "suites" is not being reported.

I think it's an entirely different problem.
I'm not sure if it might be related to the fact that the library seems to assume that I run my tests as commit checks while in fact I run them on PR synchronize events.

I'm having this issue:

Error: Git plugin error executing command
    at /__w/orders/orders/node_modules/dd-trace/packages/dd-trace/src/log/writer.js:69:46
    at withNoop (/__w/orders/orders/node_modules/dd-trace/packages/dd-trace/src/log/writer.js:21:3)
    at onError (/__w/orders/orders/node_modules/dd-trace/packages/dd-trace/src/log/writer.js:69:18)
    at Channel.publish (node:diagnostics_channel:143:9)
    at Object.error (/__w/orders/orders/node_modules/dd-trace/packages/dd-trace/src/log/index.js:101:20)
    at sanitizedExec (/__w/orders/orders/node_modules/dd-trace/packages/dd-trace/src/plugins/util/git.js:64:9)
    at unshallowRepository (/__w/orders/orders/node_modules/dd-trace/packages/dd-trace/src/plugins/util/git.js:152:28)
    at /__w/orders/orders/node_modules/dd-trace/packages/dd-trace/src/ci-visibility/exporters/git/git_metadata.js:287:5
    at /__w/orders/orders/node_modules/dd-trace/packages/dd-trace/src/ci-visibility/exporters/git/git_metadata.js:120:5
    at IncomingMessage.<anonymous> (/__w/orders/orders/node_modules/dd-trace/packages/dd-trace/src/exporters/common/request.js:96:11)
    at IncomingMessage.emit (node:events:530:35)
    at IncomingMessage.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:[169](https://github.com/usertesting/orders/actions/runs/13262822245/job/37024062356?pr=41664#step:16:170)8:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21)
Error: Command failed: git rev-parse --abbrev-ref --symbolic-full-name @{upstream}
fatal: HEAD does not point to a branch

My guess is that the on: push trigger always checks out repos with a branch, but pull requests check out repos in detached head mode.

I think this might be a different issue, but probably the original issue is also still valid. It's not blocking us anymore.

Sorry I'm mixing two issues into one. I guess what I'm trying to say is that likely these are actually separate issues:

• the "suites" filter groups using the name of the first test passed in the --specs list as the "suite" name
• the "tests" filter has no data (this is a separate issue and might be caused by these git errors I'm seeing)

[juan-fernandez]

hey @fcano-ut, thanks for the extra context.

The git error you're seeing is unlikely to lead into errors in the reporting of test data.

My guess is that the on: push trigger always checks out repos with a branch, but pull requests check out repos in detached head mode.

this shouldn't lead to test data missing either.

For the next step I'd ask you to run the suite with DD_TRACE_DEBUG: true (not setting DD_TRACE_LOG_LEVEL so that everything is logged), but that's going to leak data, so it's not recommended.

Is it possible for you to create a small reproducible case in a public repository?

[fcano-ut]

Sorry about the late reply. I do see some useful logs with DD_TRACE_DEBUG enabled.

A reproduction step of the repo would only work with a Knapsack Pro key, it's not so easy right now to create one, I would have to go through company bureaucracy to even get access to a key. There's also suddenly a lot of other priorities for me to be working on right now so I might have to let this one hang for a while...

If I reach through support I guess it would be faster since I would be able to share the debug logs with them. I'm assuming the only data that would leak are the test assertions themselves, that they could keep confidential. I didn't see any tokens in the output... 🤔

[juan-fernandez]

Sorry about the late reply. I do see some useful logs with DD_TRACE_DEBUG enabled.

A reproduction step of the repo would only work with a Knapsack Pro key, it's not so easy right now to create one, I would have to go through company bureaucracy to even get access to a key. There's also suddenly a lot of other priorities for me to be working on right now so I might have to let this one hang for a while...

If I reach through support I guess it would be faster since I would be able to share the debug logs with them. I'm assuming the only data that would leak are the test assertions themselves, that they could keep confidential. I didn't see any tokens in the output... 🤔

yeah going through support might be the best course of action here.

No tokens in the output indeed, just test data