Skip to content

Releases: milvus-io/milvus

milvus-2.5.1

27 Dec 06:01
a67b572
Compare
Choose a tag to compare

v2.5.1

Release date: December 26, 2024

Milvus version Python SDK version Node.js SDK version Java SDK version
2.5.1 2.5.2 2.5.2 2.5.2

Milvus 2.5.1 focuses on a series of bug fixes addressing memory loading, RBAC listings, query node balancing, and sealed segment indexing, while also improving the Web UI and interceptors. We highly recommend upgrading to 2.5.1 for enhanced stability and reliability.

Improvement

  • Update web UI collection and query pages. (#38701)

Bug fixes

  • Fixed OOM issues by adding a memory factor to loading estimations. (#38722)
  • Fixed privilege group expansion when listing policies in RootCoord. (#38760)
  • Fixed issues with listing privilege groups and collections. (#38738)
  • Fixed the balancer to avoid repeatedly overloading the same query node. (#38724)
  • Fixed unexpected balance tasks triggered after QueryCoord restarts. (#38725)
  • Fixed load config updates not applying to loading collections. (#38737)
  • Fixed zero read count during data import. (#38695)
  • Fixed Unicode decoding for JSON keys in expressions. (#38653)
  • Fixed interceptor DB name for alterCollectionField in 2.5. (#38663)
  • Fixed empty index parameters for sealed segments when using BM25 brute force search. (#38752)

milvus-2.4.19

27 Dec 01:14
fec92e9
Compare
Choose a tag to compare

v2.4.19

Release Date: December 26, 2024

Milvus version Python SDK version Java SDK version Node.js SDK version
2.4.19 2.4.13 2.4.9 2.4.9

Milvus 2.4.19 focuses on fixing issues in RBAC, Balancer, and loading processes, while also introducing several performance enhancements in areas such as PartitionKey Deletion and hybrid searching. We highly recommend upgrading to this version for increased stability and reliability.

Improvements

  • Expression templates are introduced to accelerate hybrid searches. (#38624)
  • Additional metrics are provided for improved deletion monitoring. (#38746)
  • L0 file generation is restricted to specific partition for partitionKey deletion to reduce amplification. (#38232)

Bug fixes

  • Fixed OOM issues by adding a memory factor to loading estimations. (#38721)
  • Fixed privilege group expansion when listing policies in RootCoord. (#38759)
  • Fixed access log retention, skipping empty log rotation. (#38661)
  • Fixed the balancer to avoid repeatedly overloading the same query node. (#38720)
  • Fixed issues with listing privilege groups and collections. (#38698)
  • Fixed load config updates not applying to loading collections. (#38596)
  • Fixed unexpected balance tasks triggered after QueryCoord restarts. (#38714)
  • Fixed zero read count during data import. (#38696)
  • Fixed Unicode decoding for JSON keys in expressions. (#38652)

milvus-2.5.0

23 Dec 03:45
7fe2cd0
Compare
Choose a tag to compare

v2.5.0

Release date: December 23, 2024

Milvus version Python SDK version Node.js SDK version Java SDK version
2.5.0 2.5.1 2.5.2 2.5.2

Milvus 2.5.0 brings significant advancements to enhance usability, scalability, and performance for users dealing with vector search and large-scale data management. With this release, Milvus integrates powerful new features like term-based search, clustering compaction for optimized queries, and versatile support for sparse and dense vector search methods. Enhancements in cluster management, indexing, and data handling introduce new levels of flexibility and ease of use, making Milvus an even more robust and user-friendly vector database.

Key Features

Full Text Search

Milvus 2.5 supports full text search implemented with Sparse-BM25! This feature is an important complement to Milvus's strong semantic search capabilities, especially in scenarios involving rare words or technical terms. In previous versions, Milvus supported sparse vectors to assist with keyword search scenarios. These sparse vectors were generated outside of Milvus by neural models like SPLADEv2/BGE-M3 or statistical models such as the BM25 algorithm.

Powered by Tantivy, Milvus 2.5 has built-in analyzers and sparse vector extraction, extending the API from only receiving vectors as input to directly accepting text. BM25 statistical information is updated in real time as data is inserted, enhancing usability and accuracy. Additionally, sparse vectors based on approximate nearest neighbor (ANN) algorithms offer more powerful performance than standard keyword search systems.

For details, refer to Analyzer Overview and Full Text Search.

Cluster Management WebUI (Beta)

To better support massive data and rich features, Milvus's sophisticated design includes various dependencies, numerous node roles, complex data structures, and more. These aspects can pose challenges for usage and maintenance.

Milvus 2.5 introduces a built-in Cluster Management WebUI, reducing system maintenance difficulty by visualizing Milvus's complex runtime environment information. This includes details of databases and collections, segments, channels, dependencies, node health status, task information, slow queries, and more.

Text Match

Milvus 2.5 leverages analyzers and indexing from Tantivy for text preprocessing and index building, supporting precise natural language matching of text data based on specific terms. This feature is primarily used for filtered search to satisfy specific conditions and can incorporate scalar filtering to refine query results, allowing similarity searches within vectors that meet scalar criteria.

For details, refer to Analyzer Overview and Text Match.

Bitmap Index

A new scalar data index has been added to the Milvus family. The BitMap index uses an array of bits, equal in length to the number of rows, to represent the existence of values and accelerate searches.

Bitmap indexes have traditionally been effective for low-cardinality fields, which have a modest number of distinct values—for example, a column containing gender information with only two possible values: male and female.

For details, refer to Bitmap Index.

Nullable & Default Value

Milvus now supports setting nullable properties and default values for scalar fields other than the primary key field. For scalar fields marked as nullable=True, users can omit the field when inserting data; the system will treat it as a null value or default value (if set) without throwing an error.

Default values and nullable properties provide greater flexibility to Milvus. Users can utilize this feature for fields with uncertain values when creating collections. It also simplifies data migration from other database systems to Milvus, allowing for handling datasets containing null values while preserving original default value settings.

For details, refer to Nullable & Default Value.

Faiss-based HNSW SQ/PQ/PRQ

Through close collaboration with the Faiss community, the HNSW algorithm in Faiss has seen significant improvements in both functionality and performance. For considerations of stability and maintainability, Milvus 2.5 has officially migrated its support for HNSW from hnswlib to Faiss.

Based on Faiss, Milvus 2.5 supports multiple quantization methods on HNSW to meet the needs of different scenarios: SQ (Scalar Quantizers), PQ (Product Quantizer), and PRQ (Product Residual Quantizer). SQ and PQ are more common; SQ provides good query performance and build speed, while PQ offers better recall at the same compression ratio. Many vector databases commonly use binary quantization, which is a simple form of SQ quantization.

PRQ is a fusion of PQ and AQ (Additive Quantizer). Compared to PQ, it requires longer build times to deliver better recall, especially at high compression rates, saying binary compression.

Clustering Compaction (Beta)

Milvus 2.5 introduces Clustering Compaction to accelerate searches and reduce costs in large collections. By specifying a scalar field as a clustering key, data is redistributed by range to optimize storage and retrieval. Acting like a global index, this feature enables Milvus to efficiently prune data during queries based on clustering metadata, enhancing search performance when scalar filters are applied.

For details, refer to Clustering Compaction.

Other Features

Streaming Node (Beta)

Milvus 2.5 introduces a new component called the streaming node, which provides Write-Ahead Logging (WAL) services. This enables Milvus to achieve consensus before and after reading and writing channels, unlocking new features, functionalities, and optimizations. This feature is disabled by default in Milvus 2.5 and will be officially available in version 3.0.

IPv6 Support

Milvus now supports IPv6, allowing for expanded network connectivity and compatibility.

CSV Bulk Import

In addition to JSON and Parquet formats, Milvus now supports direct bulk import of data in CSV format.

Expression Templates for Query Acceleration

Milvus now supports expression templates, improving expression parsing efficiency, particularly in scenarios with complex expressions.

GroupBy Enhancements

  • Customizable Group Size: Added support for specifying the number of entries returned for each group.
  • Hybrid GroupBy Search: Supports hybrid GroupBy search based on multiple vector columns.

Iterator Enhancements

  • MVCC Support: Users can now use iterators without being affected by subsequent data changes like inserts and deletions, thanks to Multi-Version Concurrency Control (MVCC).
  • Persistent Cursor: Milvus now supports a persistent cursor for QueryIterator, enabling users to resume iteration from the last position after a Milvus restart without needing to restart the entire iteration process.

Improvements

Deletion Optimization

Improved the speed and reduced memory usage for large-scale deletions by optimizing lock usage and memory management.

Dependencies Upgrade

Upgraded to ETCD 3.5.16 and Pulsar 3.0.7 LTS, fixing existing CVEs and enhancing security. Note: The upgrade to Pulsar 3.x is not compatible with previous 2.x versions.

For users who already have a working Milvus deployment, you need to upgrade the ETCD and Pulsar components before you can use the new features and functions. For details, refer to Upgrade Pulsar from 2.x to 3.x

Local Storage V2

Introduced a new local file format in Milvus 2.5, enhancing loading and query efficiency for scalar data, reducing memory overhead, and laying the groundwork for future optimizations.

Expression Parsing Optimization

Improved expression parsing by implementing caching for repeated expressions, upgrading ANTLR, and optimizing the performance of NOT IN clauses.

Improved DDL Concurrency Performance

Optimized the concurrency performance of Data Definition Language (DDL) operations.

RESTful API Feature Alignment

Aligned the functionalities of the RESTful API with other SDKs for consistency.

Security & Configuration Updates

Supported TLS to secure inter-node communication in more complex or enterprise environments. For details, refer to Security Configuration.

Compaction Performance Enhancements

Removed maximum segment limitations in mixed compaction and now prioritizes smaller segments first, improving efficiency and speeding up queries on large or fragmented datasets.

Score-Based Channel Balancing

Introduced a policy that dynamically balances loads across channels, enhancing resource utilization and overall stability in large-scale deployments.

milvus-2.4.18

20 Dec 23:28
e82af48
Compare
Choose a tag to compare

v2.4.18

Release Date: December 20, 2024

Milvus version Python SDK version Java SDK version Node.js SDK version
2.4.18 2.4.12 2.4.9 2.4.9

Milvus 2.4.18 introduces customizable privilege groups and an enhanced Grant/Revoke API, significantly streamlining permission management. This version also brings a suite of built-in privilege groups for common operational scenarios, as well as a host of performance and stability improvements—ranging from CPU optimization to faster collection loading and more efficient indexing. In addition, critical bug fixes ensure a more resilient system, addressing issues like crash scenarios and synchronization failures. We recommend you upgrade to 2.4.18 to take advantage of these key enhancements and improved overall reliability.

Features

RBAC Enhancements:

  • Introduces customizable privilege groups, allowing users to create, drop, list, and dynamically manage privileges (add/remove) within their defined groups.
  • Includes a suite of built-in privilege groups for common operational scenarios:
    • Cluster-Level: ClusterReadOnly, ClusterReadWrite, ClusterAdmin
    • Database-Level: DatabaseReadOnly, DatabaseReadWrite, DatabaseAdmin
    • Collection-Level: CollectionReadOnly, CollectionReadWrite, CollectionAdmin
  • Provides a new version of the Grant/Revoke API, enabling the use of these new interfaces without the need to specify an ObjectType.

Improvements

  • Allow hyphens in partition names (#38474)
  • Optimize CPU usage for health check requests (#35595)
  • Support templates for expressions in RESTful APIs (#38161)
  • Remove the limit on the number of load tasks per round (#38497)
  • alterindex & altercollection now support modifying properties (#38361 #38111 #38421)
  • alterdatabase supports deleting properties (#38450)
  • Add detailed replica counts for resource groups (#38315)
  • Support score-based balancing for channel policies (#38378)
  • Add metrics to count the number of non-zero values/tokens in sparse searches (#38328)
  • Remove the RPC layer of the coordinator when running in standalone or mixed mode (#38207)
  • Add mmap file usage metrics (#38211)
  • Support database requests in RESTful API (#38188)
  • Enable rate limiting for RESTful V1 (#38190)
  • Add collection ID to search request count metrics (#38144)
  • Refine clustering compaction logs (#38102)
  • Accelerate the collection loading process (#37841)
  • Improve compaction performance by removing ParamTable lookups (#37882)
  • Support retrying searches when topk is reduced and results are insufficient (#37093)
  • Update Knowhere version (#38277)
    • Optimize sparse index and get ~10% performance improvement

Bug fixes

  • Fixed a crash caused by growing-groupby (#38553)
  • Fixed an issue where the SyncSegments RPC would always fail (#38032)
  • Fixed an issue where sync tasks remained running after the DataNode had stopped (#38441)
  • Fixed inaccurate general counts (#38525)
  • Escaped prefixes before conducting searches in inverted indexes (#38425)
  • Fixed an issue where roles could be dropped even though grants still existed (#38369)
  • Fixed empty import task results (#38317)
  • Fixed a DataNode issue where progress could stall at the writer buffer memory check (#38287)
  • Fixed an issue that prevented the permission grant on the manualcompact API (#38168)
  • Fixed inaccurate partition count metrics (#38073)
  • Accelerated flushing speed by optimizing lock usage (#37897)
  • Handled errors gracefully when the compaction queue is full (#37990)
  • Optimized loading speed by separating the pool for target observation and collection loading (#37735)
  • Fixed a crash caused by retrieving varchar data from a memory-mapped growing segment (#37995)
  • Fixed an issue where channels could be accidentally released after balancing (#37940)

milvus-2.5.0-beta

22 Nov 15:22
8fe6c1c
Compare
Choose a tag to compare

v2.5.0-beta

Release date: November 26, 2024

Milvus version Python SDK version Node.js SDK version Java SDK version
2.5.0-beta 2.5.0 2.5.0 2.5.0

Milvus 2.5.0-beta brings significant advancements to enhance usability, scalability, and performance for users dealing with vector search and large-scale data management. With this release, Milvus integrates powerful new features like term-based search, clustering compaction for optimized queries, and versatile support for sparse and dense vector search methods. Enhancements in cluster management, indexing, and data handling introduce new levels of flexibility and ease of use, making Milvus an even more robust and user-friendly vector database.

Key Features

Full Text Search

Milvus 2.5 supports full text search implemented with Sparse-BM25! This feature is an important complement to Milvus's strong semantic search capabilities, especially in scenarios involving rare words or technical terms. In previous versions, Milvus supported sparse vectors to assist with keyword search scenarios. These sparse vectors were generated outside of Milvus by neural models like SPLADEv2/BGE-M3 or statistical models such as the BM25 algorithm.

Milvus 2.5 has built-in tokenization and sparse vector extraction, extending the API from only receiving vectors as input to directly accepting text. BM25 statistical information is updated in real time as data is inserted, enhancing usability and accuracy. Additionally, sparse vectors based on approximate nearest neighbor (ANN) algorithms offer more powerful performance than standard keyword search systems.

For details, refer to Full Text Search.

Cluster Management WebUI (Beta)

To better support massive data and rich features, Milvus's sophisticated design includes various dependencies, numerous node roles, complex data structures, and more. These aspects can pose challenges for usage and maintenance.

Milvus 2.5 introduces a built-in Cluster Management WebUI, reducing system maintenance difficulty by visualizing Milvus's complex runtime environment information. This includes details of databases and collections, segments, channels, dependencies, node health status, task information, slow queries, and more.

Text Match

Milvus 2.5 leverages analyzers and indexing from Tantivy for text preprocessing and index building, supporting precise natural language matching of text data based on specific terms. This feature is primarily used for filtered search to satisfy specific conditions and can incorporate scalar filtering to refine query results, allowing similarity searches within vectors that meet scalar criteria.

For details, refer to Keyword Match.

Bitmap Index

A new scalar data index has been added to the Milvus family. The BitMap index uses an array of bits, equal in length to the number of rows, to represent the existence of values and accelerate searches.

Bitmap indexes have traditionally been effective for low-cardinality fields, which have a modest number of distinct values—for example, a column containing gender information with only two possible values: male and female.

For details, refer to Bitmap Index.

Nullable & Default Value

Milvus now supports setting nullable properties and default values for scalar fields other than the primary key field. For scalar fields marked as nullable=True, users can omit the field when inserting data; the system will treat it as a null value or default value (if set) without throwing an error.

Default values and nullable properties provide greater flexibility to Milvus. Users can utilize this feature for fields with uncertain values when creating collections. It also simplifies data migration from other database systems to Milvus, allowing for handling datasets containing null values while preserving original default value settings.

For details, refer to Default Value & Null.

Faiss-based HNSW SQ/PQ/PRQ

Through close collaboration with the Faiss community, the HNSW algorithm in Faiss has seen significant improvements in both functionality and performance. For considerations of stability and maintainability, Milvus 2.5 has officially migrated its support for HNSW from hnswlib to Faiss.

Based on Faiss, Milvus 2.5 supports multiple quantization methods on HNSW to meet the needs of different scenarios: SQ (Scalar Quantizers), PQ (Product Quantizer), and PRQ (Product Residual Quantizer). SQ and PQ are more common; SQ provides good query performance and build speed, while PQ offers better recall at the same compression ratio. Many vector databases commonly use binary quantization, which is a simple form of SQ quantization.

PRQ is a fusion of PQ and AQ (Additive Quantizer). Compared to PQ, it requires longer build times to deliver better recall, especially at high compression rates, saying binary compression.

Clustering Compaction (Beta)

Milvus 2.5 introduces Clustering Compaction to accelerate searches and reduce costs in large collections. By specifying a scalar field as a clustering key, data is redistributed by range to optimize storage and retrieval. Acting like a global index, this feature enables Milvus to efficiently prune data during queries based on clustering metadata, enhancing search performance when scalar filters are applied.

For details, refer to Clustering Compaction.

Other Features

Streaming Node (Beta)

Milvus 2.5 introduces a new component called the streaming node, which provides Write-Ahead Logging (WAL) services. This enables Milvus to achieve consensus before and after reading and writing channels, unlocking new features, functionalities, and optimizations. This feature is disabled by default in Milvus 2.5 and will be officially available in version 3.0.

IPv6 Support

Milvus now supports IPv6, allowing for expanded network connectivity and compatibility.

CSV Bulk Import

In addition to JSON and Parquet formats, Milvus now supports direct bulk import of data in CSV format.

Expression Templates for Query Acceleration

Milvus now supports expression eemplates, improving expression parsing efficiency, particularly in scenarios with complex expressions.

GroupBy Enhancements

  • Customizable Group Size: Added support for specifying the number of entries returned for each group.
  • Hybrid GroupBy Search: Supports hybrid GroupBy search based on multiple vector columns.

Iterator Enhancements

  • MVCC Support: Users can now use iterators without being affected by subsequent data changes like inserts and deletions, thanks to Multi-Version Concurrency Control (MVCC).
  • Persistent Cursor: Milvus now supports a persistent cursor for QueryIterator, enabling users to resume iteration from the last position after a Milvus restart without needing to restart the entire iteration process.

Improvements

Deletion Optimization

Improved the speed and reduced memory usage for large-scale deletions by optimizing lock usage and memory management.

Dependencies Upgrade

Upgraded to ETCD 3.5.16 and Pulsar 3.0.7 LTS, fixing existing CVEs and enhancing security. Note: The upgrade to Pulsar 3.x is not compatible with previous 2.x versions.

Local Storage V2

Introduced a new local file format in Milvus 2.5, enhancing loading and query efficiency for scalar data, reducing memory overhead, and laying the groundwork for future optimizations.

Expression Parsing Optimization

Improved expression parsing by implementing caching for repeated expressions, upgrading ANTLR, and optimizing the performance of NOT IN clauses.

Improved DDL Concurrency Performance

Optimized the concurrency performance of Data Definition Language (DDL) operations.

RESTful API Feature Alignment

Aligned the functionalities of the RESTful API with other SDKs for consistency.

milvus-2.4.17

22 Nov 15:18
fc96133
Compare
Choose a tag to compare

v2.4.17

Release Date: November 22, 2024

Milvus version Python SDK version Java SDK version Node.js SDK version
2.4.17 2.4.9 2.4.8 2.4.9

Milvus 2.4.17 was a minor release aimed at critical stability enhancements. This version addressed various bugs and implemented performance optimizations to bolster system reliability.

Improvements

  • Enhanced: [2.4] Prevented the generation of "null" search parameters (#37811).
  • Enhanced: [2.4] Removed unnecessary segment clone updates in distribution (#37797) (#37833).
  • Enhanced: [2.4] Provided secondary index criteria for filtering leaderview (#37777) (#37802).
  • Used batch processing to speed up listing collections from meta kv (#37752).
  • Removed collection queryable checks from health checks (#37731).
  • [2.4] Removed segment-level tags from monitoring metrics (#37737).
  • [2.4] Removed unnecessary cloning in setstate (#37736).
  • Added search parameters to search requests in RESTful API (#37673).
  • Made Milvus images with AddressSanitizer (ASAN) available (#37682).
  • [cp24] Tidied compaction logs (#37647).
  • [2.4] Invalidated the collection cache when releasing collections (#37628).
  • [2.4] Added CGO call metrics for load/write APIs (#37627).
  • Enabled node assignment policies in resource groups (#37588).
  • Optimized describe collection and index operations (#37605).
  • [2.4] Handled legacy proxy load fields requests (#37569).
  • [2.4] Added context tracing for query coordination queryable checks (#37534).
  • [2.4] Improved root coordination task scheduling policies (#37523).
  • Refactored createindex in the RESTful API (#37237).
  • [2.4] Used cancel labels for context-canceled storage operations (#37491).
  • [2.4] Updated the template expression proto to improve transmission efficiency (#37485).

Bug fixes

  • Supported upsert with autoid=true in the RESTful API and fixed associated bugs (#37766).
  • Ensured L0 segments were loaded to workers during channel balancing (#37758).
  • Fixed delegator stuck in unserviceable status (#37702).
  • [2.4] Stored default values when errkeynotfound was returned (#37705).
  • [cp24] Changed memory check from write lock to read lock (#37526).
  • Ensured getshardleaders retried only on retriable errors (#37687).
  • [cp24] Corrected varchar primary key size calculations (#37619).
  • Fixed channel balancing that could get stuck when increasing replica numbers (#37642).
  • Addressed issues where searches returned fewer results after query node recovery (#37610).
  • [2.4] Fixed bugs retrieving data from the wrong field for L0 segments (#37599).
  • Recovered loading collection's updateTS after query coordination restarts (#37580).
  • [2.4] Added IP address validation to paramtable (#37500).
  • Fixed search/query failures caused by segments not being loaded (#37544).
  • Resolved watch channel issues due to timer reset misuse (#37542).
  • Fixed subscription leaks (#37541).
  • Resolved issues with excessively growing segments (#37540).
  • [cp24] Corrected dropped segment metrics (#37471).
  • Fixed repeated error codes in Milvus and Segcore (#37449).
  • [cp24] Separated L0 and mixed trigger intervals (#37319).

milvus-2.4.16

20 Nov 07:14
cffde80
Compare
Choose a tag to compare

Milvus v2.4.16 is deprecated, v2.4.17 is coming soon.

milvus-2.4.15

05 Nov 13:20
b7c80f9
Compare
Choose a tag to compare

v2.4.15

Release Date: November 5, 2024

Milvus version Python SDK version Java SDK version Node.js SDK version
2.4.15 2.4.9 2.4.8 2.4.9

Milvus 2.4.15 was a critical bug-fix release focused on enhancing system stability, performance, and compatibility. This version addressed a major deadlock issue that could occur during QueryNode crashes and introduced compatibility updates for the backup tool with the database feature. Additionally, Milvus 2.4.15 improved delete performance and stability through significant optimizations in L0 handling. Upgrading to v2.4.15 was strongly recommended to benefit from these critical enhancements.

Critical bug fixes

  • Resolved a deadlock issue if the QueryNode crashed during shard client initialization (#37354).
  • Reverted the enhancement to support databases for bulk insert (#37421).

Bug fixes

  • Fixed a bug where certain expressions did not correctly parse values (#37342).
  • Enhanced the Proxy to retry getting the shard leader on unloaded collections (#37326).
  • Corrected an issue where the L0 row count metrics value was always empty (#37307).
  • Skipped marking compaction timeout for mixed and L0 compaction scenarios (#37194).
  • Rectified the containment logic of OffsetOrderedArray (#37309).
  • Added a check for resources when loading delta logs (#37263).

Improvements

  • Moved L0 logic outside of the delta lock for better performance (#37340).
  • Released compacted growing segments if present in the dropped list (#37266).
  • Introduced middleware to monitor RESTful V2 input/output RPC stats (#37224, #37440).

milvus-2.4.14

29 Oct 13:01
8370caa
Compare
Choose a tag to compare

v2.4.14

Release Date: October 31, 2024

Milvus version Python SDK version Java SDK version Node.js SDK version
2.4.14 2.4.9 2.4.7 2.4.9

Milvus 2.4.14 addresses a critical issue from version 2.4.13 that could cause collection information to be lost after snapshotKV garbage collection. It also fixed a couple of resource leaks. Additionally, this release includes numerous enhancements focused on improving stability in large-scale delete operations and compaction performance.

Features

  • Supported memory mode chunk cache (#35836)
  • Supported db for bulkinsert (#37017)

Improvements

  • Delete/Compaction Optimization
    • Enabled parallel execution of l0 compactions (#36985)
    • Batched forward delete when using direct forward (#37107)
    • Skipped loading delta data in delegater when using remoteload (#37112)
    • Directly forwarded delta excluding l0 segments (#36914)
    • Added prioritization of compaction tasks in DataCoord (#36979)
    • Tracked complex delete rates (#36958)
  • Refactored CreateCollection in RESTFul API (#36885)
  • Fused multiple 'and' and 'or' operations into a single op (#36973)
  • Made skip load work for all branches (#37161)
  • Upgraded Minio dependency to support EKS Pod Identities (#37089)
  • Tidied import options (#37078)
  • Limited maximum number of import jobs (#36892)
  • Preallocated data slice to avoid re-allocating memory (#37044)
  • Prevented DataNode from loading the bf (#37027)
  • Avoided limiting ddl operations repeatedly (#37011)
  • Made the configuration item datanode.import.maxconcurrenttasknum dynamically adjustable (#37103)
  • Used queryNode.mmap.growingMmapEnabled to control the behavior of interim index (#36391)
  • Populated the Level and StartPosition fields in segmentLoadInfo of growing segment (#36911)
  • Forced to stop buffer messages when receiving the drop collection message (#36917)
  • Added metrics for querynode delete buffer info (#37097)
  • Added collection name label for some metric (#37159)
  • Used middleware to observe RESTful v2 in/out rpc stats (#37224)
  • Changed GPU default memory pool size (#36969)
  • Updated Knowhere version to 2.3.12 (#37132)
  • Allowed deleting data when disk quota exhausted (#37139)

Bug fixes

  • Fixed collection info that could not be recovered from metakv after restart if all snapshots were garbage collected (#36950)
  • Corrected the rpc error code to avoid invalid retry in client (#37025)
  • Ignored db not found error in quota center (#36850)
  • Fixed goroutine leakage in QueryNode by using singleton delete pool (#37225)
  • Fixed collection leak in querynode (#37079)
  • Fixed leakage of clustering compaction task (#36803)
  • Prohibited renaming a collection that had an alias (#37208)
  • Made sure alias was cached (#36808)
  • Search/query could have failed during updating delegator cache (#37174)
  • Excluded l0 compaction when clustering was executing (#37142)
  • Referenced collection meta when loading l0 segment meta only (#37179)
  • Delegator might have become unserviceable after querycoord restart (#37100)
  • Dynamic release partition might have failed search/query (#37099)
  • Rectified delete buffer row count quota value (#37068)
  • Passed full field list when partial load enabled (#37063)
  • Query node panic occurred during sending rpc to worker (#36988)
  • Datacoord got stuck at stopping progress (#36961)
  • Fixed the out-of-bounds access in the growing segment when raw data was replaced by interim index (#36938)
  • Rootcoord got stuck at graceful stop progress (#36881)

milvus-2.4.13-hotfix

17 Oct 09:18
dde7979
Compare
Choose a tag to compare

v2.4.13-hotfix

Release Date: October 17, 2024

Milvus version Python SDK version Java SDK version Node.js SDK version
2.4.13-hotfix 2.4.8 2.4.5 2.4.9

Milvus v2.4.13-hotfix addresses a critical issue specific to v2.4.13, where Milvus may fail to retrieve collection information after a restart if all MetaKV snapshots were garbage-collected (#36933). Users currently running v2.4.13 are advised to upgrade to v2.4.13-hotfix at the earliest opportunity to avoid potential disruptions.

Critical fixes

  • Load original key if timestamp is MaxTimestamp (#36935)