Query Nodes Failed to Balance Data Among all Nodes Evenly

[xiaobingxia-at]

Hello, I'm using HNSW/IVF-SQ8 + MMAP + nvme local disk. I was ingesting data with 1 query node, and then later on, I stopped the ingestion, and scaled the query nodes from 1 to 4. Then I restarted the ingesting data process, but this process will delete the existing data incrementally, and then re-ingest the data with same id. Then I see that the very original query node's memory usage gradually grows to 100GB, while others query nodes are staying at the 5GB. And finally, the ingestion reported error that we can't ingest data any more.

Does any one have any insight why one query node becomes really hot and it didn't distribute the data ingestion workload to other query nodes?

Related Config:

• rootCoord:
  • dmlChannelNum: 64
• queryCoord:
  • autoBalance: true
  • autoBalanceChannel: true
  • memoryUsageMaxDifferencePercentage: 10
• queryNode:
  • enableDisk: true
  • enableSegmentPrune: true
  • scheduler:
    • maxReadConcurrentRatio: 10
  • workerPooling:
    • size: 60
  • mmap:
    • vectorField: true
    • vectorIndex: true
    • scalarField: true
    • scalarIndex: true
    • growingMmapEnabled: false
    • maxDiskUsagePercentageForMmapAlloc: 90

[xiaobingxia-at]

There are only 2 collections, each collection has 10 partitions. Each partition has around 100 segments.

[xiaobingxia-at]

Updates: because the query node crashed, and it can't emit insight to the coordinator, causing the failure to redistribute the workloads.

[liliu-z]

@xiaobingxia-at
It sounds like a known issue of deletion. All deletion msg will accumulate on the Delegator QueryNode until it get compacted. We are working on fix it

Can I ask your ingestion/deletion speed after scaling?

[xiaobingxia-at]

It's been a while, but from my memory, after recreating the cluster, everything gets back to normal.