This project provides plugin implementations for the experimental MSBuild Project Cache, which enables project-level caching within MSBuild.
Important
Currently MSBuildCache assumes that the build is running in a clean repo. Incremental builds, e.g. local developer builds, are not supported. Target scenarios include PR builds and CI builds.
This feature requires Visual Studio 17.9 or later.
To enable caching, simply add a <PackageReference> for the desired cache implementation and set various properties to configure it.
For repos which build C# code, add a <PackageReference> to Microsoft.MSBuildCache.SharedCompilation for shared compilation support.
Here is an example if you're using NuGet's Central Package Management and Azure Pipelines:
Directory.Packages.props:
<PropertyGroup>
<!-- In Azure pipelines, use Pipeline Caching as the cache storage backend. Otherwise, use the local cache. -->
<MSBuildCachePackageName Condition="'$(TF_BUILD)' != ''">Microsoft.MSBuildCache.AzurePipelines</MSBuildCachePackageName>
<MSBuildCachePackageName Condition="'$(MSBuildCachePackageName)' == ''">Microsoft.MSBuildCache.Local</MSBuildCachePackageName>
<!-- Replace this with the latest version -->
<MSBuildCachePackageVersion>...</MSBuildCachePackageVersion>
</PropertyGroup>
<ItemGroup>
<GlobalPackageReference Include="$(MSBuildCachePackageName)" Version="$(MSBuildCachePackageVersion)" />
<GlobalPackageReference Include="Microsoft.MSBuildCache.SharedCompilation" Version="$(MSBuildCachePackageVersion)" />
</ItemGroup>For repos using C++, you will need to add the projects to a packages.config and import the props/targets files directly.
Directory.Build.props:
<PropertyGroup>
<PackagesConfigFile>packages.config</PackagesConfigFile>
<PackagesConfigContents>$([System.IO.File]::ReadAllText("$(PackagesConfigFile)"))</PackagesConfigContents>
<MSBuildCachePackageVersion>$([System.Text.RegularExpressions.Regex]::Match($(PackagesConfigContents), 'Microsoft.MSBuildCache.*?version="(.*?)"').Groups[1].Value)</MSBuildCachePackageVersion>
<MSBuildCachePackageRoot>$(NugetPackageDirectory)\$(MSBuildCachePackageName).$(MSBuildCachePackageVersion)</MSBuildCachePackageRoot>
</PropertyGroup>
<Import Project="$(MSBuildCachePackageRoot)\build\$(MSBuildCachePackageName).props" />Directory.Build.targets:
<Import Project="$(MSBuildCachePackageRoot)\build\$(MSBuildCachePackageName).targets" />These settings are common across all plugins, although different implementations may ignore or handle them differently. Refer to the specific plugin documentation for details.
| MSBuild Property Name | Setting Type | Default value | Description |
|---|---|---|---|
$(MSBuildCacheLogDirectory) |
string |
"MSBuildCacheLogs" | Base directory to use for logging. If a relative path, it's assumed relative to the repo root. |
$(MSBuildCacheCacheUniverse) |
string |
"default" | The cache universe is used to isolate the cache. This can be used to bust the cache, or to isolate some types of builds from other types. |
$(MSBuildCacheMaxConcurrentCacheContentOperations) |
int |
64 | The maximum number of cache operations to perform concurrently |
$(MSBuildCacheLocalCacheRootPath) |
string |
"\MSBuildCache" (in repo's drive root) | Base directory to use for the local cache. |
$(MSBuildCacheLocalCacheSizeInMegabytes) |
int |
102400 (100 GB) | The maximum size in megabytes of the local cache |
$(MSBuildCacheIgnoredInputPatterns) |
Glob[] |
Files which are part of the repo which should be ignored for cache invalidation | |
$(MSBuildCacheIgnoredOutputPatterns) |
Glob[] |
*.assets.cache; *assemblyreference.cache |
Files to ignore for output collection into the cache. Note that if output are ignored, the replayed cache entry will not have these files. This should be used for intermediate outputs which are not properly portable |
$(MSBuildCacheIdenticalDuplicateOutputPatterns) |
Glob[] |
Files to allow duplicate outputs, with identical content, across projects. Projects which produce files at the same path with differing content will still produce an error. Note: this setting should be used sparingly as it impacts performance | |
$(MSBuildCacheRemoteCacheIsReadOnly) |
bool |
false | Whether the remote cache is read-only. This can be useful for scenarios where the remote cache should only be read from but not produced to. |
$(MSBuildCacheAsyncCachePublishing) |
bool |
true | Whether files are published asynchronously to the cache as opposed to delaying the completion signal to MSBuild until publishing is complete. Note: Publishing will be awaited before the overall build completes. |
$(MSBuildCacheAsyncCacheMaterialization) |
bool |
true | Whether files are materialized on disk asynchronously from the cache as opposed to delaying the completion signal to MSBuild until publishing is complete. Note: Materialization will be awaited when a depending project requires the files and before the overall build completes. |
$(MSBuildCacheAllowFileAccessAfterProjectFinishProcessPatterns) |
Glob[] |
\**\vctip.exe |
Processes to allow file accesses after the project which launched it completes, ie accesses by a detached process. Note: these accesses will not be considered for caching. |
$(MSBuildCacheAllowFileAccessAfterProjectFinishFilePatterns) |
Glob[] |
Files to allow to be accessed by a process launched by a project after the project completes, ie accesses by a detached process. Note: these accesses will not be considered for caching. | |
$(MSBuildCacheAllowProcessCloseAfterProjectFinishProcessPatterns) |
Glob[] |
\**\mspdbsrv.exe |
Processes to allow to exit after the project which launched it completes, ie detached processes. |
$(MSBuildCacheGlobalPropertiesToIgnore) |
string[] |
CurrentSolutionConfigurationContents; ShouldUnsetParentConfigurationAndPlatform; BuildingInsideVisualStudio; BuildingSolutionFile; SolutionDir; SolutionExt; SolutionFileName; SolutionName; SolutionPath; _MSDeployUserAgent, as well as all proeprties related to plugin settings |
The list of global properties to exclude from consideration by the cache |
$(MSBuildCacheGetResultsForUnqueriedDependencies) |
bool |
false | Whether to try and query the cache for dependencies if they have not previously been requested. This option can help in cases where the build isn't done in graph order, or if some projects are skipped. |
$(MSBuildCacheTargetsToIgnore) |
string[] |
GetTargetFrameworks;GetNativeManifest;GetCopyToOutputDirectoryItems;GetTargetFrameworksWithPlatformForSingleTargetFramework |
The list of targets to ignore when determining if a build request matches a cache entry. This is intended for "information gathering" targets which do not have side-effect. eg. a build with /t:Build and /t:Build;GetTargetFrameworks should be considered to have equivalent results. Note: This only works "one-way" in that the build request is allowed to have missing targets, while the cache entry is not. This is to avoid a situation where a build request recieves a cache hit with missing target results, where a cache hit with extra target results is acceptable. |
When configuring settings which are list types, you should always append to the existing value to avoid overriding the defaults:
<PropertyGroup>
<MSBuildCacheIdenticalDuplicateOutputPatterns>$(MSBuildCacheIdenticalDuplicateOutputPatterns);**\foo.txt</MSBuildCacheIdenticalDuplicateOutputPatterns>
</PropertyGroup>Once configured, to execute builds with caching, simply run:
msbuild /graph /m /reportfileaccesses
It's also recommended to set some parameters like /p:MSBuildCacheLogDirectory=$(LogDirectory)\MSBuildCache as part of the MSBuild call, as opposed to with MSBuild properties in your Directory.Build.props for example.
Arbitrary MSBuild targets can be cached, and with Microsoft.Build.RunVSTest, you can attach running vstest-based unit tests with the "Test" target.
Once Microsoft.Build.RunVSTest, or some other target hooked to the Test target, you can get cache hits for tests by adding (/t:Build;Test), eg:
msbuild /graph /m /reportfileaccesses /t:Build;Test
This not only provides the benefits of caching unit test execution, but also executes tests concurrently with other, unrelated, projects in the graph.
This implementation uses Azure Pipeline Caching as the cache storage, which is ideal for repos using Azure Pipelines for their builds.
Please refer to the cache isolation and security pipeline caching provides to understand the security around this cache implementation.
It is expected that this plugin is used within an Azure Pipeline. The SYSTEM_ACCESSTOKEN environment variable will need to be explicitly passed to the task calling MSBuild since it's considered a secret. In classic pipelines this is done by checking "Allow scripts to access the OAuth token" in the job. In yml pipelines, this is done by passing the env var explicitly using the $(System.AccessToken) variable:
- script: MSBuild /graph /restore /reportfileaccesses /bl:$(Build.ArtifactStagingDirectory)\Logs\msbuild.binlog /p:Configuration=Release
displayName: Build
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)$(System.AccessToken) by default does not have the scope relevant to Pipeline Caching. The scope can be enabled by setting the EnablePipelineCache pipeline variable. Additionally, the scope will get automatically added if the Cache task is used somewhere in the pipeline.
This implementation uses the local file system for the cache. In particular, it uses BuildXL's LocalCache. This is recommended for locally testing and debugging caching for your repo. This implementation can also be useful if you have stateful build agents where the cache can be reused across builds.
This implementation uses Azure Blob Storage as the cache storage.
Warning
This implementation does not yet have a robust security model. All builds using this will need write access to the storage resource, so for example an external contributor could send a PR which would write/overwrite arbitrary content which could then be used by CI builds. Builds using this plugin must be restricted to trusted team members. Use at your own risk.
These settings are available in addition to the Common Settings:
| MSBuild Property Name | Setting Type | Default value | Description |
|---|---|---|---|
$(MSBuildCacheCredentialsType) |
string |
"Interactive" | Indicates the credential type to use for authentication. Valid values are "Interactive", "ConnectionString", "ManagedIdentity", "TokenCredential" |
$(MSBuildCacheBlobUri) |
Uri |
Specifies the uri of the Azure Storage Blob. | |
$(MSBuildCacheManagedIdentityClientId) |
string |
Specifies the managed identity client id when using the "ManagedIdentity" credential type | |
$(MSBuildCacheInteractiveAuthTokenDirectory) |
string |
"%LOCALAPPDATA%\MSBuildCache\AuthTokenCache" | Specifies a token cache directory when using the "Interactive" credential type |
$(MSBuildCacheIgnoreDotNetSdkPatchVersion) |
bool |
false | Whether to ignore the patch version when doing cache lookups. This trades off some correctness for the sake of getting cache hits when the SDK version isn't exactly the same. The default behavior is to consider the exact SDK version, eg. "8.0.404". With this setting set to true, it will instead use something like "8.0.4XX". Note that the major version, minor version, and feature bands are still considered. |
When using the "ConnectionString" credential type, the connection string to the blob storage account must be provided in the MSBCACHE_CONNECTIONSTRING environment variable. This connection string needs both read and write access to the resource.
When using the "TokenCredential" credential type, an access token must be provided in the MSBCACHE_ACCESSTOKEN environment variable. Alternately, if using the programmatic project cache API, a TokenCredential may be provided in the plugin's constructor.
This package enables accurate file access reporting for the Roslyn Compiler Server. Because the compiler server (vbcscompiler) launches as a detached process, its file accesses are not observed by MSBuild. This package manually reports these files accesses to the plugin.
In the future, this feature will be directly supported by Roslyn, at which point this package will no longer be needed.
This package contains much of the shared logic for the plugins, including handling file accesses and cache fingerprinting. For those wanting to use a different cache backend than the ones associated with the available plugins, referencing this package can be a good option to avoid having to duplicate the rest of the logic involved with caching.
See CONTRIBUTING.md
Microsoft.MSBuildCache is licensed under the MIT license.
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.