v2.2.0

Release tag: v2.2.0

Upgrade Notes

No steps are required to upgrade to this release. There are 3 new environment variables. All are optional. See more information about them in the Bug Fixes and Patches section.

There may be a delay before seeing the new version of the definition that v2.0.0 generates as requests for recompute go onto a queue and the existing definition is returned. This allows recomputes to happen in the background and reduces the impact on API requests.

What’s changed

Minor Changes

• Add the ability to queue definition recomputation by @qtomlinson in #1242

When a major release happens, every definition for a period of time will be recomputed based on those updated. The recent v2.0.0 update saw throughput drop significantly leading to timeouts for some requests.

To address this issue, a queue for recomputes. The new process is:

• request a definition
• if definition needs to be recomputed, it is put on the recompute queue
• return the original definition

Once the recompute completes, any definition requests will return the updated definition

Bug Fixes and Patches

• Add heap stats logging to Service by @harrider in #1238

Added conditional logging to log the heap usage stats
In support of this, two environment variables were added to control heap stats logging and interval.

LOG_NODE_HEAPSTATS - string value of true or false (case insensitive) to enable the heap stats logging

LOG_NODE_HEAPSTATS_INTERVAL_MS - string value of a number (in milliseconds) representing the interval to log heap stats at

• Adjust logging and allow configuration of batch size for dequeueMultiple @qtomlinson in #1258

When pulling from the recompute queue, multiple messages are pulled at once. If this is too high, it can reduce the efficiency of processing and consume more resources than it would need otherwise. An env var was added to provide control over this value. When debugging it can be changed to determine if it has an impact on performance.

DEFINITION_UPGRADE_DEQUEUE_BATCH_SIZE - string value of a number representing the number of messages that are dequeued at once from the recomputation queue (default=16)

DEFINITION_UPGRADE_PROVIDER - string value of versionCheck or upgradeQueue that controls the upgrade behavior (default=versionCheck)

DEFINITION_UPGRADE_QUEUE_PROVIDER - string value of memory or azure that Controls the queuing of upgrades (default=memory)

DEFINITION_UPGRADE_QUEUE_CONNECTION_STRING - string value with the connection string to the Azure Storage Queue. (default=value of HARVEST_AZBLOB_CONNECTION_STRING)

DEFINITION_UPGRADE_QUEUE_NAME - string value designating the name of the upgrade queue (default=definitions-upgrade)

Full Changelog: https://github.com/clearlydefined/service/compare/v2.1.0…v2.2.0

v2.1.0

Release tag: v2.1.0

Upgrade Notes

No steps are required to upgrade to this release. Curators will see a difference in behavior in that curations that include NOASSERTION will not be accepted.

What’s changed

Minor Changes

• Optimize recomputing definition by @qtomlinson in #1232

Originally recomputation pulls all raw tool results for all versions of the tools. This was a time consuming process. The optimization only pulls the most recent raw tool results for each tool.

Bug Fixes and Patches

• Fix curation validation by @qtomlinson in #1223

Full Changelog: https://github.com/clearlydefined/service/compare/v2.0.1…v2.1.0

v2.0.1

What's Changed

Bug Fix

• set version to v2.0.1 in prep for bug release including deploy with operations v3.1.2 by @elrayle in #1219

Full Changelog: v2.0.0...v2.0.1

v2.0.0

Release tag: v2.0.0

Upgrade Notes

No steps are required to upgrade to this release as a user of ClearlyDefined. There are no changes to the API.

The change of most interest is the addition of support for scancode LicenseRefs and the update to scancode v32.1.0.

All major changes are related to changes in newly created definitions based on changes in the crawler data output by license tool updates and license extraction process.

Note: Requests for definitions will result in a recomputation of the definition to include the changes described in this release. Definition requests do not initiate a harvest request when a definition already exists. In that case, the caller must make a harvest request through the service API in order to update raw tool results from which the definition will be constructed. Note as well that harvesting takes significant time. There will be a delay from the time the harvest request is made before the results will be reflected in a definition request.

What’s changed

Major Changes

• Update definition schema to 1.7.0 by @qtomlinson in #1144

Forces definitions with older schema to be recalculated the next time they are requested. This is required for the data changes including the addition of support for scancode LicenseRefs.

Support scancode v32.1.0 and non-SPDX licenses using LicenseRef

ScanCode major versions 31 and 32 introduced pretty drastic changes to its output format which required significant changes to our summarizing logic. Multiple PRs brought in the support for LicenseRefs identified by ScanCode.

What this means for you?

When a license is identified as NOASSERTION or OTHER, it is possible that ScanCode can identify the license as something other than one of the SPDX licenses. Several possibilities each with a different solution...

1. ScanCode has already identified a non-SPDX license - In this case, simply requesting the definition will initiate a recompute of the definition which will replace the current license with the ScanCode LicenseRef
2. ScanCode has not identified a non-SPDX license with the previous version os ScanCode - In this case, a /harvest request is required to get ScanCode to run again. Ultimately, once harvesting is completed, the definition will be re-generated. If a new LicenseRef was identified, it will be part of the re-computed definition.
3. ScanCode cannot identify a license - You won't know this in advance meaning the step to take is to send a /harvest request. The result of the re-computed definition will be the license is unchanged. Sending additional /harvest and /definitions requests will not change the results or the definition.

PRs for LicenseRef support

• Add new summarizer for recent ScanCode versions (e.g. v32.1.0) by @lumaxis in #1056
• Update to SPDX v0.1.9 to support LicenseRef mapping in scanner and parser by @qtomlinson in #1205
• Update license normalization process to support LicenseRef by @lumaxis in #1148
• maintain precedence when joining Scancode license expressions by @lumaxis in #1087

When joining license expressions with AND: 'MIT OR Apache-2.0', 'GPL', precedence should be preserved in the result. The joined expression was incorrectly constructed as GPL AND MIT OR Apache-2.0. It is now correctly constructed with precedence as GPL AND (MIT OR Apache-2.0).

Additional data related changes

• Update to SPDX v0.1.8 to avoid adding unnecessary brackets in stringify by @qtomlinson in #1203

This update brings in SPDX PR clearlydefined/spdx#30

The expressions "LGPL-2.1-only OR MIT OR BSD-3-Clause" and "LGPL-2.1-only OR BSD-3-Clause AND MIT" are valid and simplified forms of SPDX expressions. Refer to the SPDX specification for more information (https://spdx.github.io/spdx-spec/v2-draft/SPDX-license-expressions/#d4-composite-license-expressions)

Minor Changes

• Update license mapping with latest ScanCode LicenseDB data by @github-actions) in #1137, #1200

Bug Fixes and Patches

Development related

• Add source location in definitions for sourcearchive packages by @qtomlinson in #1108
• Fixed origins api for pypi components throwing 500 error when invalid group id is provided by @yashkohli88 in #1172
• Fixed origins api for maven components throwing 500 error when invalid group id is provided by @yashkohli88 in #1176)

DevOps

• Add separate CI job for testing Docker build by @lumaxis in #1126
• docs: add SECURITY.md by @nickvidal in https://github.com/clearlydefined/service/pull//1143
• update deploy scripts to v2.0.0 by @ in https://github.com/clearlydefined/service/pull//1154
• Update Deployment Pipeline to v3.0.0, App Version and Build Sha available on GET / and logs by @elrayle and @ljones140 in #1199
• Reduce logging which is our biggest Azure cost by @ljones140 in #1201
• Update .npmrc so that @clearlydefined packages come from GitHub package registry by @ljones140 in #1204
• auto-start test workflow when PR into prod by @elrayle in #1208
• Update deployment pipelines 3.1.0 by @ljones140 in #1209
• Update deployment pipelines 3.1.1 by @ljones140 in #1210

Dependencies

• Bump express from 4.17.3 to 4.19.2 by @dependabot in #1080
• Bump clearlydefined/operations from 1.0.0 to 1.1.0 by @dependabot in #1123
• Bump github/codeql-action from 3.24.6 to 3.25.8 by @dependabot #1131
• Bump braces and patch-package by by @dependabot in #1136
• Updated deprecated dependency request-promise-native by @yashkohli88 in #1163

Full Changelog: v1.3.1...v2.0.0

v1.3.1

Patch release to update prod deploy workflow to use [email protected]. See operations v1.1.0 release notes for more information.

Changes: v1.3.0...v1.3.1

v1.3.0

Release Highlights

Release tag: v1.3.0

There is one change of interest:

• Conda was added as a package manager source. Details on usage are provided below under the Add Conda support section.

Upgrade Notes

No Action Required. Optionally, you can start requesting harvests for Conda packages. See details below.

What’s changed

Changes: v1.2.0...v1.3.0

Minor Changes

Add Conda support

There is one significant change in this release to add support for Conda package manager. It is classified as minor because it is additive. It does not impact the functioning of previously supported package managers.

• Add Conda Support (#1019) (@lamarrr)

Coordinates syntax:

• type (required) - identifies to use the Conda provider (values: conda | condasource)
• provider (required) - channel on which the package will be crawled. (values: conda-forge | anaconda-main | anaconda-r)
• namespace (optional) - architecture and OS of the package to be crawled (e.g. win64, linux-aarch64). If no architecture is specified, any architecture is chosen.
• package name (required): name of the package
• revision (optional): package version and optional build version (format: (${version} | )-(${buildversion} | )) (e.g. 0.3.0, 0.3.0-py36hffe2fc). If it is a conda coordinate type, the build version of the package is usually a conda-specific representation of the build tools and environment configuration, and build iteration of the package (e.g. for a Python 3.9 environment, buildversion is py39H443E). If none is specified, the latest one will be selected using the package's timestamp.

Examples:

• conda/conda-forge/linux-aarch64/numpy/1.13.0
• condasource/conda-forge/linux-aarch64/numpy/1.13.0
• conda/conda-forge/-/numpy/1.13.0/
• conda/conda-forge/linux-aarch64/numpy/-py36

You can find additional information in the crawler v1.1.0 release notes.

Bug Fixes and Patches

Development related

• Update license mapping with latest ScanCode LicenseDB data (#1049) (@lumaxis)
• Add source location in definitions for sourcearchive packages (#1108) (@qtomlinson)
• Fix originConda route (#1113) (@qtomlinson)
• [conda] fixed service component bug (#1112) (@lamarrr, @lumaxis)

DevOps

• Update test workflow (#1114) (@lumaxis)
• use shared dev deploy workflow in operations (#1092) (@elrayle)
• Add now-required env vars to docs and minimal.env.json (#1083) (@lumaxis)
• Improve formatting and linting setup (#1058) (@lumaxis)

Dependencies

• Bump peter-evans/create-pull-request from 6.0.4 to 6.0.5 (#1110) (@dependabot[bot])
• Bump the docker-actions group with 2 updates (#1105) (@dependabot[bot])
• Bump the azure-actions group with 2 updates (#1104) (@dependabot[bot])
• Bump peter-evans/create-pull-request from 6.0.1 to 6.0.4 (#1102) (@dependabot[bot])
• Bump azure/login from 1.6.1 to 2.0.0 (#1071) (@dependabot[bot])
• Bump docker/build-push-action from 5.1.0 to 5.2.0 (#1070) (@dependabot[bot])
• Add groups to Dependabot config (#1086) (@lumaxis, @github-actions[bot])

v1.2.0

Release Highlights

Release tag: v1.2.0

There are two changes of interest:

• improved coordinate checking for PyPI
• addition of an action that automatically updates the LicenseDB data

The remaining changes impact the development and deploy processes.

Upgrade Notes

There are no required actions for this upgrade.

What’s changed

Changes: v1.1.0...v1.2.0

Minor Changes

Improve coordinate checking for PyPI

• Use dependency injection in PypiCoordinatesMapper (#1072) (@qtomlinson)
• Restrict PyPI coordinates name in PypiCoordinatesMapper (#1065) (@qtomlinson)

Action to automatically update the LicenseDB data

• Delete azure-pipelines.yml (#1057) (@lumaxis)
• Update Actions workflow to pull in LicenseDB data (#1050) (@lumaxis)

Display information related to the deploy in the default endpoint

The sha of the deployed code is displayed at the default endpoint. This is useful for debugging.

Example:

{ "status": "OK", "sha": "89555eb2a172d4c804d1f4541377e300de77ca63" }

• get latest release version from GitHub API for prod deploy (#1055) (@elrayle)
• get version from package.json instead of release (#1053) (@elrayle)

Bug Fixes and Patches

Development related

• Improve formatting and linting setup (#1058) (@lumaxis)
• Add integration tests (#1042) (@qtomlinson)
• Use URL object to generate request URL (#1066) (@lumaxis)

Move deploy process to GitHub Actions

• update prod deploy action to use prod azure credentials (#1052) (@elrayle)
• add GitHub action for deploy of production service (#1051) (@elrayle)
• remove docker secrets from deploy action (#1048, #1073, #1074) (@elrayle, @lumaxis)

Dependency updates

• Bump github/codeql-action from 3.24.5 to 3.24.6 (#1068) (@dependabot[bot])
• Bump peter-evans/create-pull-request from 6.0.0 to 6.0.1 (#1067) (@dependabot[bot])
• Bump github/codeql-action from 3.24.0 to 3.24.5 (#1062) (@dependabot[bot])
• Bump ip from 2.0.0 to 2.0.1 (#1061) (@dependabot[bot])

v1.1.0

Release v1.1.0 is a minor release.

Release Highlights

• updates to Mongo indexing
• deprecate TRIMMED_DEFINITION_MONGO_COLLECTION_NAME

Upgrade Notes

There are no required changes to move to this version. There is a recommended change related to a deprecation. See Deprecations section for more information.

updates to Mongo indexing

API affected: get definitions with sort fields set to descending.

For example:

https://api.clearlydefined.io/definitions?type=composer&provider=packagist&namespace=10up&name=wpsnapshots&sort=namespace&sortDesc=true

Sorting order is determined by:
sortDesc=false – sort ascending (default)
sortDesc=true – sort descending

Behavior in v1.0.0

In this version, the sort honored sortDesc for any field identified as a sort field. Within multiple matches of those sort fields that share the same value, the results were being sorted ascending by coordinates regardless of the value of sortDesc. This is incorrect behavior.

New Behavior in v1.1.0

The new behavior sorts the identified sort fields and the coordinates in the same direction as identified by the sortDesc parameter.

It is recommended, but not required, that you delete the indices that are NOT documented in the _createIndexes method in clearlydefined/docker_dev_env_experiment/service/providers/stores/abstractMongoDefinitionStore.js

If you accidentally delete too many indices, they will be regenerated the next time you restart the service. NOTE: This can take a long time.

• Reduce indexes (@qtomlinson)
• Update documentation (@qtomlinson)

Deprecations

Deprecating TRIMMED_DEFINITION_MONGO_COLLECTION_NAME. If you are using this configuration, you should update the configs to use the name DEFINITION_MONGO_TRIMMED_COLLECTION_NAME instead.

• Rename trimmed definition variable (@qtomlinson)
• deprecate config TRIMMED_DEFINITION_MONGO_COLLECTION_NAME (@elrayle)

What’s Changed

v1.0.0…v1.1.0

Bug Fixes and Patches

• Fix azure pipeline error (@qtomlinson)
• Minor fixes (@qtomlinson)
• Expose revision in definition.find as a sort option (@qtomlinson)
• Target Node.js v18 (@JamieMagee)
• Regenerate lockfile from v1 to v3 (@JamieMagee)
• Formatting change (@qtomlinson)
• Update Environment Variables (@qtomlinson)
• Update SPDX related versions (@qtomlinson)
• Fix curation review url to point to the corresponding server (@qtomlinson)

Documentation

• Update documentation (@qtomlinson)

v1.0.0

Release v1.0.0 is a re-release of the current production service which was last released Feb 28, 2023. The purpose of this release is to establish a known baseline as the starting point for the transition to using Semantic Versioning for the released versions. Future releases will have a Docker image stored in GitHub Packages.

Release Highlights

Release tag: v1.0.0

NOTE: The version in package.json differs from the release tag because it was previously set and could not be changed.

Breaking Changes

none

Upgrade Notes

No Action Required

What’s changed

This release is identical to the code that has been the production release since Feb 28, 2023.

previous-release:

• sha: 98f3f4745abf26d48a846aa2ea990402f782bd80
• date: 12-5-2022

Changes: previous-release...v1.0.0