How query coord selects query node in disk ann search

[park12sj]

According to the Milvus documentation, all requests are processed by all query nodes, and the results are then reduced. However, when I check the CPU usage of the query nodes, some nodes are active while others are not.

When a search request arrives, it is broadcast to all query nodes for a concurrent search. Each node then prunes the local segments, searches for vectors meeting the criteria, and reduces and returns the search results.

In practice, does this mean that not all query nodes are performing computations, and instead, the QueryCoord selects which query nodes will perform the operations?

[yhmo]

In your screenshot, looks like there are no many differences for each node. I don't think it is a big difference between 17.4% vs 18.4% (or 0.278% vs 0.332%).

In practice, yes, uneven cpu usage is possible. The main reason is:

1. The data rows in segments are different. Some segments might contain 10,000 rows, and some other segments might contain 12,000 rows.
2. For each collection, if the shard_number is N, there will be N query nodes act as "shard-leader": https://milvus.io/docs/replica.md#Shard-leader each shard leader manages segments of one shard, and the segments might be loaded in different normal query nodes. Shard leader will take more tasks than query nodes, including load balance and result reduce. So, shard leader CPU usage could be higher than normal query nodes.

[yhmo]

In small collections, there are only a few segments. For example, a collection with 2GB data, there could be 2 ~ 3 segments. Let's say you have 3 query nodes, the load balancer assigns them to different query nodes, each query node loads one segment. seg_A is 10000 rows, seg_B is 12000 rows, seg_C is 11000 rows, then you will easily see different cpu usage in different query nodes.

The load balancer already assigns fewer data(90% compared to other nodes) to shard leader to keep the balance for cpu usage. But technically it is not easy to make absolute balance because the runtime situation is complicated.

[yhmo]

shard_number doesn't affect search performance.
shard_number affects the number of segments.
For small dataset(millions of rows), the recommended shard_number is 1
For middle dataset(tens of millions rows), the recommended shard_number is 2
For huge dataset(billions of rows), the recommended shard_number is 4 ~ 8

[park12sj]

Thank you for your response.

So, if the number of segments is less than the number of query nodes, there will be query nodes without segments, and those nodes will not participate in the computation. is it correct?

[yhmo]

Right.
Three query nodes, two segments, then one query node is empty and idle.

[xiaofan-luan]

Thank you for your response.

So, if the number of segments is less than the number of query nodes, there will be query nodes without segments, and those nodes will not participate in the computation. is it correct?

1. When you have multiple querynodes, and the segment size is relatively small, then you can add more replicas to balance traffic on all nodes.
2. Delegator may become bottle neck -> Yes. There are two ways to split traffic on delegator. 1. Create more delegator 2. Add more replicas. In Milvus 2.6, delegator will be moved to Streamnode, so you can scale querynode and streamnode seperately .