how to make use all the nodes and works in the cluster

[vihariazure]

iam running milvus in the data bricks cluster
i want to make use of all the nodes in the cluster and worker also
below is my configuration

[yhmo]

Sorry we don't have experience to deploy milvus on databricks.

[vihariazure]

I have a VM where Milvus is running using Docker.
I am ingesting data from Databricks into Milvus (on the VM).

How can I utilize the Databricks workers to speed up the data ingestion process into Milvus?

[xiaofan-luan]

search for Spark Milvus connector and check if that can help

[xiaofan-luan]

iam running milvus in the data bricks cluster i want to make use of all the nodes in the cluster and worker also below is my configuration

we don't have too too much info about databricks.
can you explain more details about what you want to do?

[vihariazure]

we need to speed up data ingestion for that what are the ways
one way we thought of making utilizing of workers (parallel ingestion ) . how to do that

[yhmo]

Basically, the bottleneck of ingestion is not on the client side, it is on the server side.

When you insert some data into milvus by insert():

1. The data is received by proxy node, proxy node splits the data into pieces by shards_num(which is defined in collection schema, default is 1), and sends the pieces(as messages) into Pulsar/Kafka. The insert() returns success to the client once the data is persisted into Pulsar/Kafka.
2. Querynode and datanode subscribe messages asynchronously from Pulsar/Kafka, the pieces of data are consumed by querynode and datanode.
3. Querynode keeps the data in memory for search/query. Datanode persists the data to segments(on s3/minio).

The first bottleneck is the proxy node, it receives data by RPC, which is limited by network bandwidth. You can increase the number of proxies to improve throughput.
The second bottleneck is Pulsar/Kafka
The third bottleneck is querynode/datanode, if datanode consuming speed is much delay to ingestion, milvus will reject any ingestion requests.

So, sometimes even a simple multi-thread client ingestion program could hit the max throughput of a milvus server. The max throughput is affected by network bandwidth, hard disk performance, ability of proxy/querynode/datanode, etc.
You can use spark or some other frameworks to ingest streaming data, but "max ingestion speed" is not a major goal.

"Import data by files" is a recommended way to import huge amounts of data. In this approach, data file is directly parsed by datanode, Pulsar/Kafka is skipped. With splitting the data into proper size of data files, hopefully the ingestion speed is faster than normal insert requests.

[xiaofan-luan]

we need to speed up data ingestion for that what are the ways one way we thought of making utilizing of workers (parallel ingestion ) . how to do that

import/bulk insert is definitely what you need.
You can import 1B data into milvus in several others