[SUREFIRE-2232] StatelessXmlReporter: handle failed test result without a throwable #716

dr29bart · 2024-01-11T16:05:49Z

A regression bug appeared in 3.0.0-M6:

A testNG test class has a listener which updates results from SUCCESS to FAILURE:

@Override
public void onTestSuccess(ITestResult result) {
    result.setStatus(ITestResult.FAILURE);
    result.getTestContext().getPassedTests().removeResult(result);
    result.getTestContext().getFailedTests().addResult(result);
}

Surefire fails to process a failed test result without a throwable and reports 0 total tests.

ForkStarter IOException: java.util.NoSuchElementException.
org.apache.maven.plugin.surefire.booterclient.output.MultipleFailureException: java.util.NoSuchElementException
	at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.<init>(ThreadedStreamConsumer.java:59)
	at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer.<init>(ThreadedStreamConsumer.java:107)
	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:546)
	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:285)
	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250) 
...
Suppressed: java.util.NoSuchElementException
		at java.base/java.util.StringTokenizer.nextToken(StringTokenizer.java:347)
		at org.apache.maven.plugin.surefire.report.StatelessXmlReporter.getTestProblems(StatelessXmlReporter.java:454)
		at org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClassWithoutRerun(StatelessXmlReporter.java:221)
		at org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClass(StatelessXmlReporter.java:211)
		at org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:161)
		at org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:85)
		at org.apache.maven.plugin.surefire.report.TestSetRunListener.testSetCompleted(TestSetRunListener.java:193)
		at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:143)

The fix is to set a hardcoded value - UndefinedException as error type in case stack trace is empty.

checklist

Following this checklist to help us incorporate your contribution quickly and easily:

Make sure there is a JIRA issue filed
for the change (usually before you start working on it). Trivial changes like typos do not
require a JIRA issue. Your pull request should address just this issue, without
pulling in other changes.
Each commit in the pull request should have a meaningful subject line and body.
Format the pull request title like [SUREFIRE-XXX] - Fixes bug in ApproximateQuantiles,
where you replace SUREFIRE-XXX with the appropriate JIRA issue. Best practice
is to use the JIRA issue title in the pull request title and in the first line of the
commit message.
Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
Run mvn clean install to make sure basic checks pass. A more thorough check will
be performed on your pull request automatically.
You have run the integration tests successfully (mvn -Prun-its clean install).

If your pull request is about ~20 lines of code you don't need to sign an
Individual Contributor License Agreement if you are unsure
please ask on the developers list.

To make clear that you license your contribution under
the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

I hereby declare this contribution to be licenced under the Apache License Version 2.0, January 2004
In any other case, please file an Apache Individual Contributor License Agreement.

michael-o · 2024-01-11T16:20:01Z

...efire-common/src/main/java/org/apache/maven/plugin/surefire/report/StatelessXmlReporter.java

-                    ppw.addAttribute("type", new StringTokenizer(stackTrace).nextToken());
+                    ppw.addAttribute(
+                            "type",
+                            isBlank(stackTrace) ? "UndefinedException" : new StringTokenizer(stackTrace).nextToken());


Why don't you reuse the approach from line 451?

Do you mean indexOf approach or to use value as is?

I didn't want to leave a chance for type to be null. What non-null value can be used here?

The stackTrace StatelessXmlReporter#439 var may be null:

public String getStackTrace(boolean trimStackTrace) { StackTraceWriter w = original.getStackTraceWriter(); return w == null ? null : (trimStackTrace ? w.writeTrimmedTraceToString() : w.writeTraceToString()); }

At the same time surefire's XSD requires type

XSD

<xs:element name="error" nillable="true" minOccurs="0" maxOccurs="1"> <xs:complexType> <xs:simpleContent> <xs:extension base="xs:string"> <xs:attribute name="message" type="xs:string"/> <xs:attribute name="type" type="xs:string" use="required"/> </xs:extension> </xs:simpleContent> </xs:complexType> </xs:element>

The goal is to handle stack traces like:

// has message junit.framework.ComparisonFailure: Expected :fail at foo Actual :faild at foo // does not have message java.lang.AssertionError at org.junit.Assert.fail(Assert.java:87) at org.junit.Assert.fail(Assert.java:96) // or stack trace is missing (empty String)

Please, let me know if this alternative is more preferable:

Option A

if (report.getStackTraceWriter() != null) { //noinspection ThrowableResultOfMethodCallIgnored SafeThrowable t = report.getStackTraceWriter().getThrowable(); if (t != null) { int delimiter = StringUtils.indexOfAny(stackTrace, ":", "\t", "\n", "\r", "\f"); String type = delimiter == -1 ? stackTrace : stackTrace.substring(0, delimiter); ppw.addAttribute("type", Objects.toString(type, "UndefinedException")); } }

Option B

SafeThrowable t = report.getStackTraceWriter().getThrowable(); if (t != null) { if (t.getMessage() != null) { int delimiter = stackTrace.indexOf(":"); String type = delimiter == -1 ? stackTrace : stackTrace.substring(0, delimiter); ppw.addAttribute("type", type); } else { ppw.addAttribute( "type", isBlank(stackTrace) ? stackTrace : new StringTokenizer(stackTrace).nextToken()); } }

Option C

SafeThrowable t = report.getStackTraceWriter().getThrowable(); if (t != null) { if (t.getMessage() != null) { int delimiter = stackTrace.indexOf(":"); String type = delimiter == -1 ? stackTrace : stackTrace.substring(0, delimiter); ppw.addAttribute("type", type); } else { int delimiter = StringUtils.indexOfAny(stackTrace, "\t", "\n", "\r", "\f"); String type = delimiter == -1 ? stackTrace : stackTrace.substring(0, delimiter); ppw.addAttribute("type", type); } } </details>

I looked at this again and I want to address for 3.3.0.

I have option D for you: Since this is an oversight and strackTrace can be truly null of t.getMessage() is null, we should not write non-sense or empty to type, but leave it out. Rather no output than bad output. My proposal: Since we know that type can be null, I can remove the requried in the schema and bump to 3.1.0.

WDYT?

I can remove the requried in the schema and bump to 3.1.0.

I like this idea. it is the easiest option to implement.

Let me then prepare a PR with that change and you can build up on it.

michael-o · 2024-06-05T20:23:54Z

@dr29bart First PR: #745

michael-o · 2024-06-05T20:24:36Z

I need to refresh my schema foo with https://www.xfront.com/SchemaVersioning.html and https://www.xfront.com/Versioning.pdf

michael-o · 2024-06-06T21:00:22Z

@dr29bart FYI: #746

michael-o

@dr29bart All set, please rework your PR to option D.

…out a throwable This closes apache#716

michael-o · 2024-06-11T08:56:15Z

@dr29bart I took the liberty to change your PR.

...e-common/src/test/java/org/apache/maven/plugin/surefire/report/StatelessXmlReporterTest.java

michael-o

For me, this is fine now. Are you OK as well?

dr29bart · 2024-06-11T19:41:41Z

For me, this is fine now. Are you OK as well?

yes, I am OK to merge.

…iled result without a throwable This closes apache#716

michael-o reviewed Jan 11, 2024

View reviewed changes

michael-o requested changes Jun 8, 2024

View reviewed changes

michael-o force-pushed the SUREFIRE-2232 branch from 009dd2f to 36a1d92 Compare June 11, 2024 08:55

michael-o pushed a commit to dr29bart/maven-surefire that referenced this pull request Jun 11, 2024

[REGRESSION] StatelessXmlReporter fails to process failed result with…

36a1d92

…out a throwable This closes apache#716

michael-o self-requested a review June 11, 2024 08:56

michael-o reviewed Jun 11, 2024

View reviewed changes

...e-common/src/test/java/org/apache/maven/plugin/surefire/report/StatelessXmlReporterTest.java Outdated Show resolved Hide resolved

michael-o self-requested a review June 11, 2024 19:39

michael-o approved these changes Jun 11, 2024

View reviewed changes

[SUREFIRE-2232] [REGRESSION] StatelessXmlReporter fails to process fa…

e6287dd

…iled result without a throwable This closes apache#716

michael-o force-pushed the SUREFIRE-2232 branch from fec9fc8 to e6287dd Compare June 11, 2024 19:43

asfgit closed this in e6287dd Jun 11, 2024

asfgit merged commit e6287dd into apache:master Jun 11, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SUREFIRE-2232] StatelessXmlReporter: handle failed test result without a throwable #716

[SUREFIRE-2232] StatelessXmlReporter: handle failed test result without a throwable #716

dr29bart commented Jan 11, 2024

michael-o Jan 11, 2024

dr29bart Jan 11, 2024

michael-o Jun 5, 2024

dr29bart Jun 5, 2024

michael-o Jun 5, 2024

michael-o commented Jun 5, 2024

michael-o commented Jun 5, 2024

michael-o commented Jun 6, 2024

michael-o left a comment

michael-o commented Jun 11, 2024

michael-o left a comment

dr29bart commented Jun 11, 2024

[SUREFIRE-2232] StatelessXmlReporter: handle failed test result without a throwable #716

[SUREFIRE-2232] StatelessXmlReporter: handle failed test result without a throwable #716

Conversation

dr29bart commented Jan 11, 2024

michael-o Jan 11, 2024

Choose a reason for hiding this comment

dr29bart Jan 11, 2024

Choose a reason for hiding this comment

michael-o Jun 5, 2024

Choose a reason for hiding this comment

dr29bart Jun 5, 2024

Choose a reason for hiding this comment

michael-o Jun 5, 2024

Choose a reason for hiding this comment

michael-o commented Jun 5, 2024

michael-o commented Jun 5, 2024

michael-o commented Jun 6, 2024

michael-o left a comment

Choose a reason for hiding this comment

michael-o commented Jun 11, 2024

michael-o left a comment

Choose a reason for hiding this comment

dr29bart commented Jun 11, 2024