Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dotnet tool restore started failing on macOS with NU3037 and NU3028 errors after 11th February #46857

Open
martincostello opened this issue Feb 14, 2025 · 31 comments · Fixed by #47321
Assignees
Labels
Area-Tools Priority:1 Work that is critical for the release, but we could probably ship without
Milestone

Comments

@martincostello
Copy link
Member

Describe the bug

Since February 13th (👻), the CI in Polly when running on macOS has stopped working with the error shown below.

The strange thing is this doesn't seem to correlate with any specific code change to Polly itself. We had a successful build on February 11th after updating to the .NET 9.0.200 SDK here, so it doesn't seem to correlate nicely with a change in SDK behaviour.

The runner version doesn't appear to differ either, so that would presumably rule out a change in GitHub Actions:

Current runner version: '2.322.0'
Operating System
  macOS
  14.7.2
  23H311

I don't have access to a physical MacBook but two colleagues can replicate the failure on their machines by cloning the repo and running dotnet tool restore.

The things I don't really understand are:

  • Why now, when the countersignature timestamp expired last year?
  • Why only on macOS?

I've experimented with a few things in this PR to try and get more information, but with no success: App-vNext/Polly#2496. Check out the commit history and workflow run logs for more detail. For example, setting DOTNET_NUGET_SIGNATURE_VERIFICATION=false hasn't helped.

Seems like either it's failing on macOS when it shouldn't or it should be consistently failing on macOS, Linux and Windows if there is a genuine certificate trust issue.

I've also reported this here against the tool itself in case they've genuinely revoked a certificate or something, but again if that's the case I'd expect it to fail on all OSs: NuGetPackageExplorer/NuGetPackageExplorer#1698

To Reproduce

Clone App-vNext/Poll on a MacBook and run dotnet tool restore from the root of the repository.

Exceptions (if any)

> dotnet tool restore --no-http-cache --verbosity diagnostic
[NuGet Manager] [Info]   GET https://api.nuget.org/v3/registration5-gz-semver2/cake.tool/index.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3/registration5-gz-semver2/cake.tool/index.json 75ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3-flatcontainer/cake.tool/index.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3-flatcontainer/cake.tool/index.json 30ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3-flatcontainer/cake.tool/5.0.0/cake.tool.5.0.0.nupkg
[NuGet Manager] [Info]   OK https://api.nuget.org/v3-flatcontainer/cake.tool/5.0.0/cake.tool.5.0.0.nupkg 3ms
Verified that the NuGet package "cake.tool.5.0.0" has a valid signature.
[NuGet Manager] [Info]   GET https://api.nuget.org/v3/registration5-gz-semver2/docfx/index.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3/registration5-gz-semver2/docfx/index.json 39ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3/registration5-gz-semver2/docfx/page/0.0.1-alpha-1517061409/2.10.2.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3/registration5-gz-semver2/docfx/page/0.0.1-alpha-1517061409/2.10.2.json 31ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3/registration5-gz-semver2/docfx/page/2.11.0/2.40.0.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3/registration5-gz-semver2/docfx/page/2.11.0/2.40.0.json 30ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3/registration5-gz-semver2/docfx/page/2.40.1/2.63.1.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3/registration5-gz-semver2/docfx/page/2.40.1/2.63.1.json 38ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3/registration5-gz-semver2/docfx/page/2.64.0/2.78.2.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3/registration5-gz-semver2/docfx/page/2.64.0/2.78.2.json 30ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3-flatcontainer/docfx/index.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3-flatcontainer/docfx/index.json 29ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3-flatcontainer/docfx/2.78.2/docfx.2.78.2.nupkg
[NuGet Manager] [Info]   OK https://api.nuget.org/v3-flatcontainer/docfx/2.78.2/docfx.2.78.2.nupkg 2ms
Verified that the NuGet package "docfx.2.78.2" has a valid signature.
[NuGet Manager] [Info]   GET https://api.nuget.org/v3/registration5-gz-semver2/dotnet-validate/index.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3/registration5-gz-semver2/dotnet-validate/index.json 38ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3-flatcontainer/dotnet-validate/index.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3-flatcontainer/dotnet-validate/index.json 29ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3-flatcontainer/dotnet-validate/0.0.1-preview.304/dotnet-validate.0.0.1-preview.304.nupkg
[NuGet Manager] [Info]   OK https://api.nuget.org/v3-flatcontainer/dotnet-validate/0.0.1-preview.304/dotnet-validate.0.0.1-preview.304.nupkg 3ms
Failed to validate package signing.
Verifying dotnet-validate.0.0.1-preview.304
Signature type: Author
  Subject Name: CN=NuGet Package Explorer (.NET Foundation), O=NuGet Package Explorer (.NET Foundation), L=Redmond, S=Washington, C=US, SERIALNUMBER=603 389 068
  SHA256 hash: 7726640A0DCAB8252F40C73B2FE5C5F38C137E756C0B62521C07DF390288CDAB
  Valid from: 4/25/2021 12:00:00 AM to 7/22/2024 11:59:59 PM
Signature type: Repository
  Subject Name: CN=NuGet.org Repository by Microsoft, O=NuGet.org Repository by Microsoft, L=Redmond, S=Washington, C=US
  SHA256 hash: 5A2901D6ADA3D18260B9C6DFE2[133](https://github.com/App-vNext/Polly/actions/runs/13326053490/job/37219456977?pr=2496#step:5:134)C95D74B9EEF6AE0E5DC334C8454D1477DF4
  Valid from: 2/16/2021 12:00:00 AM to 5/15/2024 11:59:59 PM
warn : NU3018: The repository countersignature found a chain building issue: RevocationStatusUnknown: An incomplete certificate revocation check occurred.
error: NU3037: The repository countersignature validity period has expired.
error: NU3028: The repository countersignature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.
Package signature validation failed.

Further technical details

.NET SDK:
 Version:           9.0.200
 Commit:            90e8b202f2
 Workload version:  9.0.200-manifests.b4a8049f
 MSBuild version:   17.13.8+cbc39bea8
Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  14.7
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /Users/runner/.dotnet/sdk/9.0.200/
.NET workloads installed:
There are no installed workloads to display.
Configured to use loose manifests when installing new manifests.
Host:
  Version:      9.0.2
  Architecture: arm64
  Commit:       80aa709f5d
.NET SDKs installed:
  7.0.102 [/Users/runner/.dotnet/sdk]
  7.0.202 [/Users/runner/.dotnet/sdk]
  7.0.[30](https://github.com/App-vNext/Polly/actions/runs/13326053490/job/37219456977?pr=2496#step:5:31)6 [/Users/runner/.dotnet/sdk]
  7.0.410 [/Users/runner/.dotnet/sdk]
  8.0.101 [/Users/runner/.dotnet/sdk]
  8.0.204 [/Users/runner/.dotnet/sdk]
  8.0.303 [/Users/runner/.dotnet/sdk]
  8.0.405 [/Users/runner/.dotnet/sdk]
  8.0.406 [/Users/runner/.dotnet/sdk]
  9.0.102 [/Users/runner/.dotnet/sdk]
  9.0.200 [/Users/runner/.dotnet/sdk]
.NET runtimes installed:
  Microsoft.AspNetCore.App 7.0.2 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.4 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.9 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.20 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.1 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.4 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.7 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.12 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.13 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.1 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.2 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 7.0.2 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.4 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.9 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.20 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.1 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.4 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.7 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.12 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.13 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.1 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.2 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
Other architectures found:
  None
Environment variables:
  DOTNET_ROOT       [/Users/runner/.dotnet]
global.json file:
  /Users/runner/work/Polly/Polly/global.json
Learn more:
  https://aka.ms/dotnet/info
Download .NET:
  https://aka.ms/dotnet/download
Copy link
Contributor

Thanks for creating this issue! We believe this issue is related to NuGet tooling, which is maintained by the NuGet team. Thus, we closed this one and encourage you to raise this issue in the NuGet repository instead. Don’t forget to check out NuGet’s contributing guide before submitting an issue!

If you believe this issue was closed out of error, please comment to let us know.

Happy Coding!

@baronfel
Copy link
Member

Tool restore isn't owned by NuGet so I've reopened this and tagged it correctly.

@baronfel baronfel reopened this Feb 14, 2025
@martincostello martincostello marked this as a duplicate of NuGet/Home#14105 Feb 14, 2025
@baronfel
Copy link
Member

The 'verify signing' error message is coming from the SDK directly:

await VerifySigning(nupkgPath, repository);

Based on the repository we downloaded the package from, we try to validate the package signature, but I'm wondering if a) we're doing this correctly and b) if we're adhering to the DOTNET_NUGET_SIGNATURE_VERIFICATION environment variable that users have documentation about.

cc @nkolev92 for feedback on the signature validation process, and I guess @marcpopMSFT for thoughts on who we should get to dig into this more on our end?

@baronfel
Copy link
Member

A few other notes:

  • we have an override to skip verification
  • that override is technically exposed for global tool installation, but not for local tool installation
  • it feels like, much like we have the unified 'restore action config' to describe installation sources, interactivity, etc, we should also be treating signature verification for tools in a unified way.

@marcpopMSFT
Copy link
Member

@joeloff is the signing expert but he's out for the moment. @edvilme @Forgind have done some experience in the tools space.

@marcpopMSFT
Copy link
Member

Also pinging @dtivel who was dealing with signature checks for nuget packages previously.

@joeloff
Copy link
Member

joeloff commented Feb 18, 2025

Repository thumbprints are periodically updated. the package in question is from 2021, so you may need to include updated thumbprints in your nuget.config. I might be totally wrong here, but take a look at this post: https://devblogs.microsoft.com/nuget/the-nuget-org-repository-signing-certificate-will-be-updated-as-soon-as-april-8th-2024/

@martincostello
Copy link
Member Author

I could try that out, but why would it only affect macOS?

@nkolev92
Copy link
Contributor

The 'verify signing' error message is coming from the SDK directly:

sdk/src/Cli/dotnet/NugetPackageDownloader/NuGetPackageDownloader.cs

Line 129 in fcba596

await VerifySigning(nupkgPath, repository);
Based on the repository we downloaded the package from, we try to validate the package signature, but I'm wondering if a) we're doing this correctly and b) if we're adhering to the DOTNET_NUGET_SIGNATURE_VERIFICATION environment variable that users have documentation about.

cc @nkolev92 for feedback on the signature validation process, and I guess @marcpopMSFT for thoughts on who we should get to dig into this more on our end?

I think @dtivel might be more helpful here.

@alexrp
Copy link

alexrp commented Feb 24, 2025

FWIW, this is breaking macOS CI across most of my .NET projects because I rely on dotnet tool restore so I can use dotnet cake afterwards. I've had to revert to 9.0.102 as a result. (That said, the actual offending package seems to be SourceLink in my case.)

@nagilson
Copy link
Member

https://github.com/dotnet/sdk/blob/main/src/Layout/redist/trustedroots/codesignctl.pem @dtivel It looks like this needs to be updated since it hasnt been since June. I thought this was supposed to happen automagically?

@nagilson
Copy link
Member

nagilson commented Feb 25, 2025

For servicing 9.0.200:
In addition, we would skip sign check on Linux

if (!_verifySignatures && !_validationMessagesDisplayed)
{
if (VerbosityGreaterThanMinimal())
{
_reporter.WriteLine(LocalizableStrings.NuGetPackageSignatureVerificationSkipped);
}
_validationMessagesDisplayed = true;
}
if (!_verifySignatures)
{
return;
}
in the dotnet tool codepath since we don't do that here and probably have the option to disable this validation from the dotnet tool command path.

@nagilson nagilson added this to the 9.0.3xx milestone Feb 25, 2025
@nagilson nagilson added Priority:1 Work that is critical for the release, but we could probably ship without and removed untriaged Request triage from a team member labels Feb 25, 2025
@nagilson nagilson removed this from the 9.0.3xx milestone Feb 25, 2025
@nagilson nagilson modified the milestones: 9.0.2xx, 9.0.3xx Feb 25, 2025
@dtivel
Copy link
Contributor

dtivel commented Feb 25, 2025

NuGet package signing verification is disabled on macOS by default, and it will remain that way for the foreseeable future. If you choose to enable it anyway, YMMV. It is not a supported scenario. See https://learn.microsoft.com/en-us/dotnet/core/tools/nuget-signed-package-verification#macos for details.

Verification is disabled on macOS for two key reasons:

  1. Apple maintains a block list of banned certificates. (Microsoft does too, so nothing special.) This block list contains the intermediate CA related to Symantec timestamping, the same timestamping used to timestamp older NuGet.org packages. When NuGet performs certificate chain verification on the timestamping certificate, verification fails because macOS enforces the block list. Even in contextual trust evaluations --- when you bring your own trusted roots, as NuGet does ---, macOS's distrust model overrides our contextual trust model decisions. Perhaps, it may be in part due to the blocked certificate being an intermediate and not a root and .NET's contextual trust model only supporting custom roots.
  2. Apple removed from macOS support for checking revocation via CRL. This means certificates with only CRL (and not OCSP) will be unrevokable on macOS.

https://github.com/dotnet/sdk/blob/main/src/Layout/redist/trustedroots/codesignctl.pem @dtivel It looks like this needs to be updated since it hasnt been since June. I thought this was supposed to happen automagically?

I keep them up to date manually. When the Trusted Root Program team has updates relevant for code signing, I update them here. There haven't been updates to code signing roots relevant to our scenarios since June.

@dtivel
Copy link
Contributor

dtivel commented Feb 25, 2025

@joeloff, @nagilson, @marcpopMSFT, was signature verification enabled by default on macOS during dotnet tool restore in 9.0.200?

@martincostello
Copy link
Member Author

For context, this wasn't explicitly enabled. It just stopped working around the date described above without anything being changed in my code.

Forcing it off with the env var doesn't fix it either. It's just flat out broken now for some reason.

@dtivel
Copy link
Contributor

dtivel commented Feb 25, 2025

Got it, @martincostello. Thank you for confirming. It sounds like there was a change that enabled it by default on macOS.

@joeloff
Copy link
Member

joeloff commented Feb 27, 2025

@joeloff, @nagilson, @marcpopMSFT, was signature verification enabled by default on macOS during dotnet tool restore in 9.0.200?

Yes, I believe that was the case.

@martincostello
Copy link
Member Author

If that's the case, the confusing part for me is why did it fail days after I merged the 9.0.200 SDK update?

I'd have thought we'd have caught the problem through it failing CI in the PR (or in main immediately after merging if there was an inconsistency).

@baronfel
Copy link
Member

Yeah, that's been the part that's been puzzling me too, @martincostello.

@robin-aws
Copy link

@joeloff, @nagilson, @marcpopMSFT, was signature verification enabled by default on macOS during dotnet tool restore in 9.0.200?

Yes, I believe that was the case.

But was this intentional or a regression? If it's still true that signature verification doesn't work reliably on macOS, it sounds like a regression.

@BouleDeGommme
Copy link

I'm having the same issue when building my app on the runner image macos-14 via azure pipelines since yesterday. Looks like DOTNET_NUGET_SIGNATURE_VERIFICATION is enabled in it's new version.

Will your fix be installed on it or do I have modifications to do to avoid this problem ?

@baronfel
Copy link
Member

baronfel commented Mar 6, 2025

@BouleDeGommme we'll need to make a fix and release a new version, and then the Azure Pipelines images will need to install that version. No timelines as of yet but we have devs working on it now.

@Forgind
Copy link
Member

Forgind commented Mar 6, 2025

It sounds like macOS signature verification has never worked consistently. I wouldn't be surprised if we happened to enable it at a time when it was a bit more permissive, and Apple made a change to make it more secure, breaking this scenario.

I think we should restrict signature verification to windows and linux by default. That'll be a small PR, so I'll go ahead and make it; then we can decide if we want it.

@nagilson nagilson assigned edvilme and nagilson and unassigned Forgind Mar 6, 2025
@nagilson
Copy link
Member

nagilson commented Mar 6, 2025

@edvilme is working on a PR; discussed offline.

@mattspeterson
Copy link

I'm having the same issue when running dotnet tool install:

> authtest dotnet tool install -g Microsoft.Web.LibraryManager.Cli
Failed to validate package signing.

Verifying Microsoft.Web.LibraryManager.Cli.2.1.175

Signature type: Author
  Subject Name: CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
  SHA256 hash: AA12DA22A49BCE7D5C1AE64CC1F3D892F150DA76140F210ABD2CBFFCA2C18A27
  Valid from: 9/29/2020 7:00:00 PM to 10/5/2023 7:00:00 AM

Signature type: Repository
  Subject Name: CN=NuGet.org Repository by Microsoft, O=NuGet.org Repository by Microsoft, L=Redmond, S=Washington, C=US
  SHA256 hash: 5A2901D6ADA3D18260B9C6DFE2133C95D74B9EEF6AE0E5DC334C8454D1477DF4
  Valid from: 2/15/2021 6:00:00 PM to 5/15/2024 6:59:59 PM

error: NU3037: The author primary signature validity period has expired.
error: NU3028: The author primary signature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.
error: NU3037: The repository countersignature validity period has expired.
error: NU3028: The repository countersignature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.

Package signature validation failed.

Apologies if I missed it in the thread above, but is there a workaround for now?

@BouleDeGommme
Copy link

BouleDeGommme commented Mar 10, 2025

@mattspeterson For my pipeline using macos-14 image, i added a task installing .NET 7 ->
- task: UseDotNet@2 displayName: 'Install .NET Core sdk 7.x' inputs: version: 7.x

It fixed my problem for now.

@Forgind
Copy link
Member

Forgind commented Mar 10, 2025

This should be fixed by #47321 whenever that ships

@martincostello
Copy link
Member Author

What was the missing piece of the puzzle on why this took a few days to kick in after I updated to the 9.0.200 SDK, rather than immediately?

@rishav-karanjit
Copy link

Is there an ETA for a release which includes #47321?

@Forgind
Copy link
Member

Forgind commented Mar 11, 2025

What was the missing piece of the puzzle on why this took a few days to kick in after I updated to the 9.0.200 SDK, rather than immediately?

After reading about it for a bit, I floated that Apple may have changed how they verified signatures, as apparently they've done that in the past. I think @joeloff mentioned having found something clearer? More generally, it sounds like it's a bad idea to be trying to verify signatures on mac.

@Forgind
Copy link
Member

Forgind commented Mar 11, 2025

Is there an ETA for a release which includes #47321?

This is a perfectly valid question that I don't have a great answer to. I think I heard that it missed the upcoming release by a hair, so maybe in a month? I'm not really sure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area-Tools Priority:1 Work that is critical for the release, but we could probably ship without
Projects
None yet
Development

Successfully merging a pull request may close this issue.