Add AllowWorkDirectoryRepositories knob

[aleksandrlevochkin]

Description

Added AllowWorkDirectoryRepositories knob that allows to set a relative path to the _work directory during checkout.

Related issues: #1506, #4192

Based on this PR by @echalone #3475

Testing

• Added unit tests
• Tested locally with different values for AZP_AGENT_ALLOW_WORK_DIRECTORY_REPOSITORIES variable

[PaulVrugt]

It is awesome that this is being added.

• Will scaleset agent pools be able to use this?
• it would be even more awesome of we can change the default base path for checkouts. Otherwise will we have to update all the checkout steps of all our pipelines in our repo (which we have a lot), and make sure that every time we add a new pipeline we don't forget to set the checkout path

[aleksandrlevochkin]

@PaulVrugt

1. Yes, I think it should be working with scaleset agent pools
2. Thanks for your suggestion. We might consider this feature for the future, but not in the scope of this PR

[echalone]

[NiuBClass666]

I see this feature got merged into master. May I ask if it meant this will be released soon?

[echalone]

I see this feature got merged into master. May I ask if it meant this will be released soon?

I don't know for sure of course, but since it got merged into master right after they created the 3.229 release branch I guess it will come with release 3.231 around end of November or in December.

[echalone]

By the way, I've already compiled the Windows x64 agent with this feature (commit a3067b7 from master merger) myself and we're using it for our tests with the newly structure pipelines that use multiple centralized repositories. Everything works as expected. There's the bug with the wrong detection of the primary repository if the self repository isn't checked out first of course (see #4217), but that's not urgent since all you need to do to fix it is check out the self repository first. There's also the fun behaviour of Build.SourcesDirectory which will always point to \s in a multi-checkout scenario and System.DefaultWorkingDirectory which will always point to the same directory as Build.SourcesDirectory. So people have to keep that in mind, but we've already adapted the documentation accordingly a few years ago, so the behaviour is clearly documented. There's of course my feature request and pull request for the adaptable working directory by just choosing one of the repositories as the working directory (#3479), but that would need to be added to the yaml definition of the Azure DevOps Server as well. However, as long as you know where you've checked out to and just use that exact path when calling scripts or similar that shouldn't be a problem either. The only problem I encountered is with the SonarQube extension which just assumes the System.DefaultWorkingDirectory as the projectBaseDir when the MSBuild scanner is used, but that's their problem really, since they should have made that adaptable.

tl;dr: everything works as expected and as documented

[mjthurlkill]

Glad to see there has been some discussion around this issue and some movement.
I just ran into this issue a couple days ago with a new little pipeline, so I'll go through the comments and see if there is a resolution for my issue.

In the latest case, it is very simple 3 step pipeline using a single repo. The .yml is currently just in my private branch.
I push the initial changes to the yml, run the pipeline, ADO sends it to an on-prem agent, it clones the full 30gb git repo.
I push some other small changes to the yml, run the pipeline, ADO sends it to the same agent, it again clones the full 30gb (instead of just pulling the recent changes).

Of course, pulling the changes in git in another copy of the repo on one of my machines pulls down just the change to the yml very quickly. So appears to be an issue with ADO keeping track of which repo has been used in which workdir rather than a git issue. Though it does seem to keep track to a certain degree. When running the same job multiple times, ADO does select the same agent for the job each time unless there is alot of activity and the agent gets assigned a different job before this job is run again.

Anyway, I'll check the comments and see if there are new settings or clues for how to address this.

[mjthurlkill]

Part of the problem I had above was a result of a recent change to ADO where, for new pipelines, it uses shallow by default.
Shallow seems like more trouble than its worth. I would start transitioning to partial clones.

[noahzalapi]

Hello, I am having trouble using this knob - would you be able to share documentation or a detailed walkthrough of how to activate this feature?

As a background, I work in an environment where we will easily run 8+ different pipelines in the same repository on one machine. Because of this, we are now seeing our agent work directory take upwards of 400 GB, which obviously we want to decrease :D

Currently I am stuck getting this error:

##[error]Input path '../RelativeFolder/' should resolve to a directory under '/LocalStorage/agent/_work/7', current resolved path '/LocalStorage/agent/_work/RelativeFolder'.

Here is my file structure (with specific information obfuscated under general examples):

user@example-agent-1:/LocalStorage/agent/_work$ ll
total 4
drwxr-xr-x.  6 user users   52 Oct  8 09:56 1
drwxr-xr-x.  6 user users   52 Oct  9 11:38 7
drwxr-xr-x.  6 user users   52 Oct  2 18:47 8
drwxr-xr-x.  3 user group   17 Aug 28 22:38 Media
drwxr-xr-x.  2 user users    6 Oct  9 11:49 RelativeFolder
drwxr-xr-x.  4 user group   81 Sep 28 01:07 SourceRootMapping
drwxr-xr-x. 19 user group 4096 Oct  4 09:21 _tasks
drwxr-xr-x.  2 user group    6 Oct  9 11:40 _temp
drwxr-xr-x.  4 user group   29 Sep 27 16:42 _tool

This is how I am specifying the path to the folder:

steps:
  - checkout: self
    path: ../Repositories/SDK2
    lfs: true
    clean: true
    submodules: true

At this point I have tried to activate this feature 2 different ways:

1. Used AllowWorkDirectoryRepositories knob:
   1. Added AllowWorkDirectoryRepositories to .env file
   2. Stopped agent with sudo ./svc.sh stop
   3. Reloaded env variables with ./env.sh
   4. Started agent again sudo ./svc.sh start
2. Used the config flag --allowWorkDirectoryRepositories:
   1. Stop and uninstall agent: sudo ./svc.sh stop then sudo ./svc.sh uninstall
   2. Remove config: ./config.sh remove
   3. Reconfigure box (no other keys used): ./config.sh --allowWorkDirectoryRepositories
   4. Run agent install and start agent: sudo ./svc.sh install then sudo ./svc.sh start

System info:

• OS: Rocky Linux 8.10
• Agent version: 3.243.0

Any help you can give would be greatly appreciated! It could very well be that I am missing a step in the process of activating, I just wouldn't know where that is :P

[aleksandrlevochkin]

Hi @noahzalapi, thank you for your question. As per my understanding the .env file is meant to be a snapshot of specific variables listed here:

azure-pipelines-agent/src/Misc/layoutroot/env.sh

Lines 3 to 18 in 8c756a7

	varCheckList=(
	'LANG'
	'JAVA_HOME'
	'ANT_HOME'
	'M2_HOME'
	'ANDROID_HOME'
	'GRADLE_HOME'
	'NVM_BIN'
	'NVM_PATH'
	'VSTS_HTTP_PROXY'
	'VSTS_HTTP_PROXY_USERNAME'
	'VSTS_HTTP_PROXY_PASSWORD'
	'LD_LIBRARY_PATH'
	'PERL5LIB'
	'AGENT_TOOLSDIRECTORY'
	)

If you want to add a custom variable such as AZP_AGENT_ALLOW_WORK_DIRECTORY_REPOSITORIES, you will need to take a different approach that is described here.

Basically, after running sudo ./svc.sh install the runsvc.sh script is created (it will be located in the same directory alongside other files such as svc.sh), and you can modify this file by adding this line:

export AZP_AGENT_ALLOW_WORK_DIRECTORY_REPOSITORIES='true'

here:

azure-pipelines-agent/src/Misc/layoutbin/runsvc.sh

Line 13 in 8c756a7

# insert anything to setup env when running as a service

Then you can run ./runsvc.sh and your pipeline example should work. Please, let me know if you experience any issues in this process.

[noahzalapi]

Hey @aleksandrlevochkin - thank you for your response!

Adding this line to my runsvc.sh file did in fact work and I was able to see it use the new relative folder. For reference, since the svc.sh calls the runsvc.sh script, I was able to start the agent via the standard sudo ./svc.sh start command and it worked just fine.

Thank you again for all your help! :)