Offload publishing to separate jobs

[jorgee]

POC implementation of the offloading file publication to jobs.

publishOffload = true to activate the offload

Currently tested environments

• file copy and move with local executor
• s3 files awsbatch with s5cmd or fusion

Know issue/limitation:

• AWS Batch compute environment InstanceRole must have permissions to write the publication bucket and directory
• Copies with fusion are returning 0 even when copy is failure.

To Do:

• retries
• group tasks
• add test with different providers
• add documentation

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	2c5c185
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/6765e3707516e400082241bd

[pditommaso]

Looks nice, just left minor linting-like comments 😄

[pditommaso]

In principle else is not needed

[jorgee]

I have added the group of tasks and the retries. I have tried to do the same as @bentsherman did for task grouping but it was not working with Amazon. At the end, I have implemented a simpler solution where every time a file copy/move is offloaded I store the command together with and id and once a batch of commands are generated, a task is invoked with the list of commands. The code of the task is in the copy-group-template.sh. This scripts run all the commands in parallel, manages the retries and prints the exit of each command in the stdout together with the id. At the end of the task execution, I check the stdout output to verify if the publication command has failed or finished correctly and produces a warning or an error depending on the failOnError flag.

I have also modify the order of how the session is stop because there was a race condition between the publications and the end task monitor. Some publication tasks were submitted after the shutting down the monitor. So, now it first shutdown the the publishThreadpool and then it terminates the task monitor.