[NET9] dotnet test hangs during build

[indy-singh]

Describe the bug

When calling dotnet test the build step hangs on random projects.

C:\Users\indysingh\Desktop>dotnet-stack.exe report -p 20288
Thread (0x1958):
  CPU_TIME
  System.Private.CoreLib.il!System.Threading.WaitHandle.WaitOneNoCheck(int32,bool,class System.Object,value class WaitHandleWaitSourceMap)
  System.Private.CoreLib.il!System.Threading.WaitHandle.WaitOne()
  Microsoft.Build.il!Microsoft.Build.Execution.BuildSubmission.Execute()
  MSBuild.il!Microsoft.Build.CommandLine.MSBuildApp.ExecuteBuild(class Microsoft.Build.Execution.BuildManager,class Microsoft.Build.Execution.BuildRequestData)
  MSBuild.il!Microsoft.Build.CommandLine.MSBuildApp.BuildProject(class System.String,class System.String[],class System.String,class System.Collections.Generic.Dictionary`2<class System.String,class System.String>,class System.Collections.Generic.Dictionary`2<class System.String,class System.String>,class Microsoft.Build.Framework.ILogger[],value class Microsoft.Build.Framework.LoggerVerbosity,class Microsoft.Build.CommandLine.DistributedLoggerRecord[],int32,bool,class System.IO.TextWriter,class System.IO.TextWriter,bool,class System.Collections.Generic.ISet`1<class System.String>,class System.Collections.Generic.ISet`1<class System.String>,class System.Collections.Generic.ISet`1<class System.String>,bool,class Microsoft.Build.Logging.ProfilerLogger,bool,bool,value class Microsoft.Build.Execution.ProjectIsolationMode,class Microsoft.Build.Graph.GraphBuildOptions,bool,bool,bool,class System.String[],class System.String,bool,class Microsoft.Build.Execution.BuildResult&,class System.String[])
  MSBuild.il!Microsoft.Build.CommandLine.MSBuildApp.Execute(class System.String[])
  MSBuild.il!Microsoft.Build.CommandLine.MSBuildApp.Main(class System.String[])
  Microsoft.DotNet.Cli.Utils.il!Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(class System.String[])
  dotnet.il!Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingApp.Execute()
  dotnet.il!Microsoft.DotNet.Tools.RestoringCommand.Execute()
  dotnet.il!Microsoft.DotNet.Tools.Test.TestCommand.ForwardToMsbuild(class System.CommandLine.ParseResult,class System.String[],class System.String)
  dotnet.il!Microsoft.DotNet.Tools.Test.TestCommand.Run(class System.CommandLine.ParseResult)
  System.CommandLine.il!System.CommandLine.Invocation.InvocationPipeline.Invoke(class System.CommandLine.ParseResult)
  System.CommandLine.il!System.CommandLine.ParseResult.Invoke()
  dotnet.il!Microsoft.DotNet.Cli.Program.ProcessArgs(class System.String[],value class System.TimeSpan,class Microsoft.DotNet.Cli.Telemetry.ITelemetry)
  dotnet.il!Microsoft.DotNet.Cli.Program.Main(class System.String[])

To Reproduce

I really wish I could give you a step by step reproduction.

It's incredibly intermittent the exact command run 10 times will either fail consistently or succeed consistently.

dotnet test --tl:off --nologo --diag diag.log

diag*.log does have things like this:-

TpTrace Warning: 0 : 12940, 1, 2024/12/13, 21:19:04.426, 136452277481, vstest.console.dll, TestPluginDiscoverer: Failed to get types from assembly 'Microsoft.Diagnostics.NETCore.Client, Version=0.2.10.41601, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. Error: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Microsoft.Bcl.AsyncInterfaces, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.
   at System.Reflection.RuntimeModule.GetDefinedTypes()
   at System.Reflection.RuntimeModule.GetTypes()
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginDiscoverer.GetTestExtensionsFromAssembly[TPluginInfo,TExtension](Assembly assembly, Dictionary`2 pluginInfos, String filePath) in /_/src/Microsoft.TestPlatform.Common/ExtensionFramework/TestPluginDiscoverer.cs:line 159
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.


TpTrace Information: 0 : 12940, 1, 2024/12/13, 21:19:04.443, 136452451394, vstest.console.dll, Failed to load assembly Microsoft.VisualStudio.TestPlatform.Fakes, Version=17.0.0.0, Culture=neutral. Reason:System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.Fakes, Version=17.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.
File name: 'Microsoft.VisualStudio.TestPlatform.Fakes, Version=17.0.0.0, Culture=neutral, PublicKeyToken=null'
   at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext, RuntimeAssembly requestingAssembly, Boolean throwOnFileNotFound)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at Microsoft.VisualStudio.TestPlatform.Common.Utilities.FakesUtilities.LoadTestPlatformAssembly() in /_/src/Microsoft.TestPlatform.Common/Utilities/FakesUtilities.cs:line 273

But that is seemingly a red-herring as that happens on a good run too.

It's like something gets "stuck" and remains stuck until we do a dotnet build beforehand and then do dotnet test --no-build that always works.

Things we tried

• Eliminate transitive dependencies (because it has caused issues for us in the past)
• Upgrading nuget packages to latest version (things like TestPlaform etc)
• Ruled out application specific hangs (we have now seen the hang in a relatively small app compared to a large app - small and large referring to the number of tests)

Exceptions (if any)

None that we can see so far.

Further technical details

.NET SDK:
 Version:           9.0.100
 Commit:            59db016f11
 Workload version:  9.0.100-manifests.c6f19616
 MSBuild version:   17.12.7+5b8665660

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.20348
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\9.0.100\

.NET workloads installed:
There are no installed workloads to display.
Configured to use loose manifests when installing new manifests.

Host:
  Version:      9.0.0
  Architecture: x64
  Commit:       9d5a6a9aa4

.NET SDKs installed:
  6.0.418 [C:\Program Files\dotnet\sdk]
  9.0.100 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.26 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.31 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.26 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.31 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 6.0.26 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 9.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

[indy-singh]

I should mention that the target project(s) are targeting net8 not net9.

Cheers,
Indy

[indy-singh]

Other things we tried:-

• Rolling back to VS17.11.4 (we recently upgraded to 17.12) - this is controlled by DSC
• Rolling back net9 hosting bundle (https://dotnet.microsoft.com/en-us/download/dotnet/thank-you/runtime-aspnetcore-9.0.0-windows-hosting-bundle-installer) - this is controlled by DSC

It still hangs.

Cheers,
Indy

[indy-singh]

This is going to sound crazy, but when I don't use --tl:off it never fails, I have no idea why. Time to dig into that.

Cheers,
Indy

[indy-singh]

I've been poking around and it looks like they have been previous issues around the terminal logger.

microsoft/vstest#5091

I'm aware that is the new default beginning with net9.

Cheers,
Indy

[indy-singh]

I'm not sure if this is the sdk or the dotnet test command now, so I've opened a sister ticket:

microsoft/vstest#10760