Development environment guide #5486

christopher-hakkaart · 2024-11-08T13:22:02Z

Added a new guide for creating development environment page. Added tabs to help split Windows/macOS/Linux.

Development environment

VS Code
Extensions
- Nextflow
- nf-core
- Remote development
Docker
Git
WSL

Updates:

Moved from Getting started to Tutorials
Dev containers section has been removed

@netlify /developer-env

Signed-off-by: Christopher Hakkaart <[email protected]>

netlify · 2024-11-08T13:22:18Z

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	`b315a5a`
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/67518ddbf4db290008847429
😎 Deploy Preview	https://deploy-preview-5486--nextflow-docs-staging.netlify.app/developer-env
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

ewels

Very nice!

docs/developer-env.md

pditommaso · 2024-11-08T13:46:27Z

docs/developer-env.md

+### Development Containers extension
+
+The VS Code Dev Containers extension lets you use a container as a full-featured development environment. It allows you to open any folder inside (or mounted into) a container and take advantage of VS Code's full feature set.
+
+The Dev Containers extension is included as a part of the [Remote Development extension pack](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack). See {ref}`remote-development-ext` for installation instructions.
+
+### Create and run a dev container
+
+A `devcontainer.json` file in your project directory instructs VS Code how to access, create, and configure a development container. It can be used to run an application or to separate tools, libraries, or runtimes.
+
+The `devcontainer.json` can be used to:
+
+- Install additional tools in the container.
+- Automatically install extensions.
+- Forward or publish additional ports.
+- Set runtime arguments.
+- Reuse or extend your existing Docker Compose setup.
+- Add more Advanced container configuration.
+
+To create a dev container with an existing image:
+
+1. Create `.devcontainer/devcontainer.json` in the root of your project.
+1. Add an image with the Nextflow tooling and VS Code customizations to the `devcontainer.json`. For example:
+
+    ```json
+    {
+        "name": "Nextflow Dev Container",
+        "image": "nfcore/gitpod:latest",
+        "remoteUser": "vscode",
+        "runArgs": ["--privileged"],
+
+        // Configure tool-specific properties.
+        "customizations": {
+            // Configure properties specific to VS Code.
+            "vscode": {
+                // Set *default* container specific settings.json values on container create.
+                "settings": {
+                    "python.defaultInterpreterPath": "/opt/conda/bin/python"
+                },
+
+                // Add the IDs of extensions you want installed when the container is created.
+                "extensions": ["ms-python.python", "ms-python.vscode-pylance", "nf-core.nf-core-extensionpack", "nextflow.nextflow"]
+            }
+        }
+    }
+    ```
+
+    :::{note}
+    Instead of using a prebuilt image, a custom Dockerfile may also live in the `.devcontainer` folder. You can replace the image property in `devcontainer.json` with dockerfile and utilize the custom container. See [Create a Dev Container](https://code.visualstudio.com/docs/devcontainers/create-dev-container) for more information.
+    :::
+
+1. Enter **Dev Containers: Reopen in Container** in the VS Code Command Palette and reopen your project. You should now see the name of the container ("Nextflow Dev Container" if using the above example) in the bottom left corner of VS Code.
+
+:::{note}
+Dev Containers can also be used by GitHub Codespaces in VS Code or the browser. See [GitHub Codespaces](https://code.visualstudio.com/docs/remote/codespaces) for more information.


not sure this makes much sense in the context of nextflow workflow in which each process can have its own container. Our best practice is Conda + Wave for container reproducibility and portability

Fair point. I'm not committed to keeping this. It's very thin on details and can get complicated quickly. I'll comment it out.

I want to push back on p's argument. The intent of a devcontainer isn't to have a single container with all processes for your pipeline (I agree conda + wave is best-practice for that). Instead the devcontainer has all of the pre-requisites like conda, nextflow, nf-core, nf-test, etc installed and updated to their latest versions.

Devcontainers works really well for the following use cases:

People who are new to developing in Nextflow and don't have any of these tools installed on their computer

Folks who only occasionally develop Nextflow pipelines and don't want all of those tools cluttering up their system when they're not working on nextflow stuff.

Teams who want to ensure everyone is working off the same version of all the base tools.

Especially with devcontainers being supported by both gitpod and Codespaces there's also major benefits to being able to boot up into a repo-specific cloud environment quite easily. (part of why I'd love Data Studios to support devcontainers.json down the line too).

Yeah agree with this. I think that devcontainers for the pipeline development environment is not incompatible with using conda / wave etc for actually running the pipeline.

Signed-off-by: Christopher Hakkaart <[email protected]>

pditommaso · 2024-11-08T16:22:09Z

Not sure this is material for nextflow documentation, this is more material for tutorial or blog post

kenibrewer · 2024-11-09T00:24:57Z

Not sure this is material for nextflow documentation, this is more material for tutorial or blog post

I've had four people in the last week (1 at summit and 3 at ASHG) tell me that they had a hard time figuring out how to install Nextflow from the documentation. Two of them gave up and quit and never used Nextflow, one of them needed personal help from me, and one was running an ancient version of nextflow that came pre-installed as a workaround.

Those user experiences make me think this is a very important addition to the main docs. I also think it would be better featured prominently as one of the first pages new users encounter on the docs page. A separate tutorial page or blog post will be harder to find and won't get continued ongoing attention to make sure we keep it up to date.

Signed-off-by: Christopher Hakkaart <[email protected]>

pditommaso · 2024-11-11T11:40:41Z

I've had four people in the last week ..

Understand and it makes sense. However, I'm contrary having dev containers in this context. Provided it could help, it should not make part of the best practices we are promoting for container usage both for dev and production deployment

justinegeffen

This is awesome!

Please don't be alarmed at the number of comments - the content is great - just a few editorial tweaks here and there. Happy to workshop the feedback if it's unclear. :)

docs/developer-env.md

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Signed-off-by: Christopher Hakkaart <[email protected]>

ewels

Content is looking great, though I think that the VSCode part can be condensed a bit to help with general readability and maybe yank the devcontainers into a blog post. Otherwise, comments are very minor.

Great work! 👏🏻

docs/developer-env.md

Signed-off-by: Christopher Hakkaart <[email protected]>

christopher-hakkaart · 2024-12-05T11:26:28Z

I've removed the dev containers section, removed repeating text boxes, removed bullet point examples, and move the whole page down to the tutorials section.

christopher-hakkaart added 12 commits November 5, 2024 14:29

First draft developer environment

33103ac

Signed-off-by: Christopher Hakkaart <[email protected]>

Merge branch 'nextflow-io:master' into docs-user-guide

1d199a2

Adding WSL skeleton and fixes

a1dfb9f

Signed-off-by: Christopher Hakkaart <[email protected]>

Adding dev containers section

633f71b

Signed-off-by: Christopher Hakkaart <[email protected]>

Finish draft

ef32325

Signed-off-by: Christopher Hakkaart <[email protected]>

Fix minor formatting issue

49b7f6f

Signed-off-by: Christopher Hakkaart <[email protected]>

Improve clarity

493b2e9

Signed-off-by: Christopher Hakkaart <[email protected]>

Improve clarity again

9a5a1fd

Signed-off-by: Christopher Hakkaart <[email protected]>

Fix bullet points

72cb41d

Signed-off-by: Christopher Hakkaart <[email protected]>

Try new phrasing

57e0024

Signed-off-by: Christopher Hakkaart <[email protected]>

Try new phrasing

d75d3bb

Signed-off-by: Christopher Hakkaart <[email protected]>

Merge branch 'nextflow-io:master' into docs-user-guide

6a58a07

ewels reviewed Nov 8, 2024

View reviewed changes

pditommaso reviewed Nov 8, 2024

View reviewed changes

christopher-hakkaart added 2 commits November 8, 2024 15:46

Incorporate comments from draft review

ba8175d

Signed-off-by: Christopher Hakkaart <[email protected]>

Incorporate comments from draft review

89d87e3

Signed-off-by: Christopher Hakkaart <[email protected]>

Uncomment dev containers and add Gitpod note

9cb8f7b

Signed-off-by: Christopher Hakkaart <[email protected]>

justinegeffen reviewed Nov 11, 2024

View reviewed changes

christopher-hakkaart and others added 8 commits November 11, 2024 20:46

Update docs/developer-env.md

fdd9ca2

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

73b66a7

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

c4d733e

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

a7cc667

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

f3b10be

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

6686425

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

55db7ab

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

9f9147e

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

christopher-hakkaart and others added 11 commits November 11, 2024 20:48

Update docs/developer-env.md

2a7ebc2

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

177f2a9

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

7304074

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

1a8e1e5

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

e6349f1

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

b3c00c5

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

d046d2f

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

e4832f7

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

2dd42b9

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Apply suggestions from code review

461a157

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Add suggestions from review

c839def

Signed-off-by: Christopher Hakkaart <[email protected]>

bentsherman added dependencies Pull requests that update a dependency file docs and removed dependencies Pull requests that update a dependency file labels Nov 13, 2024

ewels reviewed Nov 13, 2024

View reviewed changes

christopher-hakkaart added 5 commits December 4, 2024 13:20

Merge branch 'master' into docs-user-guide

62f2404

Signed-off-by: Christopher Hakkaart <[email protected]>

Fix conflict after merge

eb915d3

Signed-off-by: Christopher Hakkaart <[email protected]>

Move page, remove points, remove code blocks

cf5602f

Signed-off-by: Christopher Hakkaart <[email protected]>

Remove commented text

9eb8650

Signed-off-by: Christopher Hakkaart <[email protected]>

Merge branch 'master' into docs-user-guide

b315a5a

christopher-hakkaart marked this pull request as ready for review December 5, 2024 11:26

christopher-hakkaart requested a review from a team as a code owner December 5, 2024 11:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Development environment guide #5486

Development environment guide #5486

christopher-hakkaart commented Nov 8, 2024 •

edited

Loading

netlify bot commented Nov 8, 2024 •

edited

Loading

ewels left a comment

pditommaso Nov 8, 2024

christopher-hakkaart Nov 8, 2024

kenibrewer Nov 8, 2024

ewels Nov 8, 2024

pditommaso commented Nov 8, 2024

kenibrewer commented Nov 9, 2024

pditommaso commented Nov 11, 2024

justinegeffen left a comment

ewels left a comment

christopher-hakkaart commented Dec 5, 2024

Development environment guide #5486

Are you sure you want to change the base?

Development environment guide #5486

Conversation

christopher-hakkaart commented Nov 8, 2024 • edited Loading

netlify bot commented Nov 8, 2024 • edited Loading

✅ Deploy Preview for nextflow-docs-staging ready!

ewels left a comment

Choose a reason for hiding this comment

pditommaso Nov 8, 2024

Choose a reason for hiding this comment

christopher-hakkaart Nov 8, 2024

Choose a reason for hiding this comment

kenibrewer Nov 8, 2024

Choose a reason for hiding this comment

ewels Nov 8, 2024

Choose a reason for hiding this comment

pditommaso commented Nov 8, 2024

kenibrewer commented Nov 9, 2024

pditommaso commented Nov 11, 2024

justinegeffen left a comment

Choose a reason for hiding this comment

ewels left a comment

Choose a reason for hiding this comment

christopher-hakkaart commented Dec 5, 2024

christopher-hakkaart commented Nov 8, 2024 •

edited

Loading

netlify bot commented Nov 8, 2024 •

edited

Loading