Run tests targeting .NET Standard

[CharliePoole]

In resolving issue #1182, I experimented by actually allowing an assembly targeting .NET Standard 2.0 to be loaded by the engine. In the case of that issue, it was not a test assembly and loading it resulted in a clean error message saying that no tests were found. That's a big improvement over crashing, but I think we may be able to go further.

From the initial release of .NET Core, and .NET Standard, it has been stated that all tests must target a Core platform rather than a version of .NET Standard. I'm wondering if that's really true. Obviously, the agent running the tests must target a specific platform, but I'm now thinking that it should be easy enough to write, .NET Standard tests when the assembly under test targets .NET Standard.

There are, of course, reasons why someone may not want to do this. They may prefer to test using each actual platform, which will be used in production. That's what we now support. But I see no reason to forbid testing under .NET Standard in cases where that's what somebody wants to do. I can even envision situations where most tests are contained in a .NET Standard assembly, with a few extra tests created in a separate assembly targeting multiple platforms.

Before I add this feature, I'd like to hear whether anyone sees problems with supporting it as a feature. The fix to #1182, which loaded but didn't run tests targeting .NET Standard 2.0, only took a few lines. I'll continue experiment a bit more to determine whether any problems arise from actually running them.

I want to hear from anyone who reads this but especially @rprouse, @jnm2, @mikkel and @OsirisTerje :-)

[CharliePoole]

I have run a simple test targeting netstandard2.0 and 2.1, with no further changes to the engine code. The test did no more than Assert.Pass() so we would want to have further tests. The automatic selection of a 3.1 agent works well for both cases. They can also run under net5.0, net6.0 and net7.0, by use of the --framework option.

UPDATE: Using NUnit 3.12, I was able to execute tests targeting netstandard1.6 as well.

So it looks as if this is easy to do at the moment. Still asking if we should, however. :-)

[rprouse]

I'd be curious if there is something in Testing within Visual Studio or running from dotnet test that prevents us from running .NET Standard tests? When we first made this decision, it was following the guidance to get dotnet test working originally.

I also wonder if doing so sets unrealistic expectations? As a user, I might run my netstandard1.6 tests using the console and whatever the default agent/target are and assume that it is okay on all platforms which may not be the case? By specifying the target platforms, we are being explicit in which platforms we have tested and support.

I am likely overthinking this though 😄

[CharliePoole]

One small advantage of allowing it is that we don't have to make special provision for handling it in the code as we now do. It's not much, however, and would be outweighed by any subsequent difficulties we encounter in executing the tests.

On your other point, I'd say that anyone who assumes running tests on one platform means it will run on others is not using good professional judgement. It might be necessary to point out the trade-off involved in the docs. If we definitely don't want to support it, then there needs to be a strategy for dealing with the fact that it currently works, albeit accidentally and not fully tested.

[CharliePoole]

Regarding running under Visual Studio, I think that's not a problem because AFAIK VS never passes a .NET Standard assembly to be executed. @OsirisTerje is that correct? I don't know what prevents it under dotnet test but I see no reason why we couldn't have a feature, which dotnet test doesn't support. We already have lots!

One possibility would be to call it "experimental" and possibly require the user to specify the runtime to use on the command-line.

[manfred-brands]

We have many common libraries that (used to) target .NETStandard2.0 as being the lowest common denominator. The corresponding Test projects target multiple frameworks. One downside of that is that we compile those up to 4 times for all supported runtime frameworks: net48, netcoreapp3.1, net6.0 and net7.0. If we could compile the test project also only for .netstandard2.0 but test it with the supported frameworks it would save some compile time.
Downside would be that unless we can make it work in VS, debugging NUnit tests is then not possible without resorting to Debugger.Launch.

However, the APIs for .netstandard2.0 and .net7.0 are diverging. We either cannot use the new overloads (like Stream.Write(ReadOnlySpan) or we need conditional compilation. Also the NUnit Platform attribute seems to have stopped at net-4.5, so making test conditional on runtime framework is not simple.

Another problem with .netstandard is that it allows using API calls that are in the standard, but simple not supported in .netcore, e.g.: Thread.Abort. Even if the tests are conditional, it still allows a user to call it only to get a PlatformNotSupported exception at runtime. The SupportedOSPlatformAttribute doesn't help if we compile for .netstandard2.0. We therefore moved to specific runtime versions of our common libraries.

[CharliePoole]

@manfred-brands Do you think the option should be available to use .NET Standard tests, or should it be disabled?

[manfred-brands]

Do you think the option should be available to use .NET Standard tests, or should it be disabled?

If it doesn't harm, then keep it, but do not use a default framework so it is obvious for the caller it tests a particular framework only.

[CharliePoole]

@manfred-brands I like that idea but I'll have to think about how the user should express the default. It may need to be separate from the existing --framework option if the user is running multiple assemblies together.

[CharliePoole]

Dropping this idea, which has not generated a lot of interest.

[cliffAtOsaic]

@CharliePoole thank you for starting this thread... I was trying everything to find out why my tests weren't running and nothing else I could find was helping me. I changed the test projects from netstandard2.0 to net6.0 and suddenly they started running again!

I do think this should be fixed or AT LEAST throw a build or test error.