dotnet test -graph fails once plugin is installed

[AArnott]

dotnet test -graph generally works for .NET SDK projects. But once the local MSBuildCache plugin is referenced, dotnet test -graph quits working.
Since -graph is typically added to Directory.Build.rsp as part of adopting a cache plugin for a repo, this will defeat even plain vanilla looking dotnet test.

Repro

md testrepro
cd testrepro
dotnet new xunit
dotnet test -graph # WORKS
dotnet add package Microsoft.MSBuildCache.Local --prerelease
dotnet test -graph # FAILS