Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: autogen component manifests #3633

Draft
wants to merge 18 commits into
base: main
Choose a base branch
from

Conversation

sicoyle
Copy link
Contributor

@sicoyle sicoyle commented Dec 16, 2024

Description

I got tired of manually updating metadata.yaml files for components. This is err prone, if people even remember to update it at all, and Dapr components are only as good as their metadata files since that's what we use to generate the metadata bundle.

This PR is in draft mode bc I'm still working through the components and want feedback on where I've taken things:

  • I am using build tags with the go generator to generate these using a make target and cli cmd I've added to handle parsing the metadata.go files.
  • Each component that leverages this auto generation must implement the MetadataBuilder interface I've added. That allows me to use ast to parse the file functions that I'm using the interface for to guarantee they're all there to parse.
  • Still working through all components, and marking the ones that need a lot of refactor with a go generate line that echos TODO so it is clear which still need to be done. This will allow a follow up issue to be created to track those. This also allows us to ensure that all can be marked with the TODO print out or are auto-generated.
  • The standardized metadata.go per component is also used to encourage and leverage self documenting code where I read the comments as the metadata field descriptions.
  • I've also added custom tags on struct fields to help fill in if a field is required or not, and this includes customization to support input/output bindings having diff required fields. While I do not want to heavily rely upon encoding tags, they MUST be added for the special case of bindings. Some fields are input only required or vice-versa so this must be indicated and therefore parsed to be put into the resulting manifest.
  • I've left a TODO comment on updating the schema version to v2. I think we should do this, but want confirmation. While the metadata is still the same, I feel like v2 is appropriate given now we are going to be autogenerating the metadata files through code. I'm thorough, but not 100% perfect 😄
  • Due to this being WIP still, ignore my comments/prints pls!
  • Also, docs should be updated to just pull these generated manifests to display in the UI, and we need to update the directory structure to specific components to a sandardized format so I can autogenerate reference URLs based on component type and name. cc @msfussell just a heads up.

This is an example of a before and after for S3:
BEFORE

schemaVersion: v1
type: bindings
name: aws.s3
version: v1
status: stable
title: AWS S3
urls:
  - title: Reference
    url: https://docs.dapr.io/reference/components-reference/supported-bindings/s3/
binding:
  output: true
  operations:
    - name: create
      description: Create blob
    - name: get
      description: Get blob
    - name: delete
      description: Delete blob
    - name: list
      description: List blob
builtinAuthenticationProfiles:
  - name: aws
metadata:
  - name: endpoint
    description: AWS endpoint for the component to use, to connect to S3-compatible services or emulators. Do not use this when running against production AWS.
    type: string
    example: http://localhost:4566
  - name: bucket
    description: The name of the S3 bucket to write to.
    required: true
    type: string
    example: bucket
  - name: decodeBase64
    description: Configuration to decode base64 file content before saving to bucket storage. (In case of saving a file with binary content).
    type: bool
    default: "false"
    example: "true"
  - name: encodeBase64
    description: Configuration to encode base64 file content before returning the content. (In case of opening a file with binary content).
    type: bool
    default: "false"
    example: "true"
  - name: forcePathStyle
    description: 'Currently Amazon S3 SDK supports virtual-hosted-style and path-style access. When false (the default), uses virtual-hosted-style format, i.e.: `https://<your bucket>.<endpoint>/<key>`. When true, uses path-style format, i.e.: `https://<endpoint>/<your bucket>/<key>`.'
    type: bool
    default: "false"
    example: "true"
  - name: disableSSL
    description: Allows to connect to non-`https://` endpoints.
    type: bool
    default: "false"
    example: "true"
  - name: insecureSSL
    description: When connecting to `https://` endpoints, accepts self-signed or invalid certificates.
    type: bool
    default: "false"
    example: "true"

AFTER

schemaVersion: v1
type: bindings
name: aws.s3
version: v1
status: stable
title: AWS S3
urls:
  - title: Reference
    url: https://docs.dapr.io/reference/components-reference/supported-bindings/s3/
binding:
  output: true
  operations:
    - name: create
      description: Create blob
    - name: get
      description: Get blob
    - name: delete
      description: Delete blob
    - name: list
      description: List blob
builtinAuthenticationProfiles:
  - name: aws
metadata:
  - name: endpoint
    description: AWS endpoint for the component to use, to connect to S3-compatible services or emulators. Do not use this when running against production AWS.
    type: string
    example: http://localhost:4566
  - name: bucket
    description: The name of the S3 bucket to write to.
    required: true
    type: string
    example: bucket
  - name: decodeBase64
    description: Configuration to decode base64 file content before saving to bucket storage. (In case of saving a file with binary content).
    type: bool
    default: "false"
    example: "true"
  - name: encodeBase64
    description: Configuration to encode base64 file content before returning the content. (In case of opening a file with binary content).
    type: bool
    default: "false"
    example: "true"
  - name: forcePathStyle
    description: 'Currently Amazon S3 SDK supports virtual-hosted-style and path-style access. When false (the default), uses virtual-hosted-style format, i.e.: `https://<your bucket>.<endpoint>/<key>`. When true, uses path-style format, i.e.: `https://<endpoint>/<your bucket>/<key>`.'
    type: bool
    default: "false"
    example: "true"
  - name: disableSSL
    description: Allows to connect to non-`https://` endpoints.
    type: bool
    default: "false"
    example: "true"
  - name: insecureSSL
    description: When connecting to `https://` endpoints, accepts self-signed or invalid certificates.
    type: bool
    default: "false"
    example: "true"

You can see in this case, the metadata is the same; however, for some components I'm making corrections on things.

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #[issue number]

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

  • Code compiles correctly
  • Created/updated tests
  • Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

Signed-off-by: Samantha Coyle <[email protected]>
Signed-off-by: Samantha Coyle <[email protected]>
Signed-off-by: Samantha Coyle <[email protected]>
Signed-off-by: Samantha Coyle <[email protected]>
Signed-off-by: Samantha Coyle <[email protected]>
Signed-off-by: Samantha Coyle <[email protected]>
Signed-off-by: Samantha Coyle <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant