arm服务器上部署集群Milvus，部分组件无法正常启动，求助

[LinHaiSheng580]

按照官网上的k8s集群离线Helm部署方式部署，以value.yaml模板渲染获取milvus_manifest.yaml文件安装部署，其中Milvus版本选取2.4.4-20240531-8e7f36d9-arm64。
期间发现其中的部分镜像版本不支持arm版本，将版本内容更换成其他支持arm的镜像版本，但是生成的milvus_manifest.yaml文件中还是包含了pulsar的内容（这是为啥？）
按照官网提供的方式启动kubectl apply -f milvus_manifest.yaml，但是发现etcd启动异常，日志如下：
: Unsupported system page size
: Unsupported system page size
2024/12/20 08:36:02 maxprocs: Leaving GOMAXPROCS=32: CPU quota undefined

__  _________ _   ____  ______

/ |/ / / /| | / / / / / __/
/ /|/ // // /| |/ / // /\
// /////_/__/

Welcome to use Milvus!
Version: v2.4.4
Built: Fri May 31 10:25:16 UTC 2024
GitCommit: 8e7f36d
GoVersion: go version go1.20.7 linux/arm64

TotalMem: 135841972224
UsedMem: 37158912

open pid file: /run/milvus/querycoord.pid
lock pid file: /run/milvus/querycoord.pid
[2024/12/20 08:36:02.028 +00:00] [INFO] [roles/roles.go:306] ["starting running Milvus components"]
[2024/12/20 08:36:02.028 +00:00] [INFO] [roles/roles.go:169] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so]
[2024/12/20 08:36:02.040 +00:00] [DEBUG] [config/etcd_source.go:50] ["init etcd source"] [etcdInfo="{"UseEmbed":false,"EnableAuth":false,"UserName":"","PassWord":"","UseSSL":false,"Endpoints":["m244-etcd-0.m244-etcd-headless.milvus.svc.cluster.local:2379","m244-etcd-1.m244-etcd-headless.milvus.svc.cluster.local:2379","m244-etcd-2.m244-etcd-headless.milvus.svc.cluster.local:2379"],"KeyPrefix":"by-dev","CertFile":"/path/to/etcd-client.pem","KeyFile":"/path/to/etcd-client-key.pem","CaCertFile":"/path/to/ca.pem","MinVersion":"1.3","RefreshInterval":5000000000}"]
[2024/12/20 08:36:02.041 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[m244-etcd-0.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-1.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-2.m244-etcd-headless.milvus.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/12/20 08:36:02.041 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
[2024/12/20 08:36:02.045 +00:00] [DEBUG] [config/etcd_source.go:89] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[m244-etcd-0.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-1.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-2.m244-etcd-headless.milvus.svc.cluster.local:2379]"]
[2024/12/20 08:36:02.046 +00:00] [INFO] [paramtable/component_param.go:3524] ["DeployModeEnv is not set, use default"] [default=0.5]
[2024/12/20 08:36:02.047 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=EtcdSource]
[2024/12/20 08:36:02.048 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2024/12/20 08:36:02.048 +00:00] [INFO] [roles/roles.go:255] [setupPrometheusHTTPServer]
[2024/12/20 08:36:02.048 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[m244-etcd-0.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-1.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-2.m244-etcd-headless.milvus.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/12/20 08:36:02.048 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
[2024/12/20 08:36:02.048 +00:00] [INFO] [http/server.go:112] ["management listen"] [addr=:9091]
[2024/12/20 08:36:02.053 +00:00] [DEBUG] [querycoord/service.go:218] [network] [port=19531]
[2024/12/20 08:36:02.154 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[m244-etcd-0.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-1.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-2.m244-etcd-headless.milvus.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/12/20 08:36:02.155 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2024/12/20 08:36:02.158 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2024/12/20 08:36:02.158 +00:00] [DEBUG] [querycoord/service.go:168] ["QueryCoord try to wait for RootCoord ready"]
[2024/12/20 08:36:02.160 +00:00] [DEBUG] [sessionutil/session_util.go:620] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.233.122.168:53100]
[2024/12/20 08:36:02.160 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.233.122.168:53100] [serverID=6]
[2024/12/20 08:36:02.163 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2024/12/20 08:36:02.163 +00:00] [DEBUG] [querycoord/service.go:178] ["QueryCoord report RootCoord ready"]
[2024/12/20 08:36:02.163 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2024/12/20 08:36:02.165 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2024/12/20 08:36:02.165 +00:00] [DEBUG] [querycoord/service.go:189] ["QueryCoord try to wait for DataCoord ready"]
[2024/12/20 08:36:02.166 +00:00] [DEBUG] [client/client.go:90] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2024/12/20 08:36:02.166 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:02.166 +00:00] [WARN] [grpcclient/client.go:554] ["fail to get grpc client"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:02.166 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2024/12/20 08:36:02.167 +00:00] [DEBUG] [client/client.go:90] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2024/12/20 08:36:02.167 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:02.167 +00:00] [WARN] [grpcclient/client.go:476] ["fail to get grpc client in the retry state"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:02.167 +00:00] [WARN] [grpcclient/client.go:467] ["retry func failed"] [retried=0] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2024/12/20 08:36:02.368 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2024/12/20 08:36:02.369 +00:00] [DEBUG] [client/client.go:90] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2024/12/20 08:36:02.369 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:02.369 +00:00] [WARN] [grpcclient/client.go:476] ["fail to get grpc client in the retry state"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:02.770 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2024/12/20 08:36:02.771 +00:00] [DEBUG] [client/client.go:90] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2024/12/20 08:36:02.771 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:02.771 +00:00] [WARN] [grpcclient/client.go:476] ["fail to get grpc client in the retry state"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:03.572 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2024/12/20 08:36:03.573 +00:00] [DEBUG] [client/client.go:90] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2024/12/20 08:36:03.573 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:03.573 +00:00] [WARN] [grpcclient/client.go:476] ["fail to get grpc client in the retry state"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:05.174 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2024/12/20 08:36:05.175 +00:00] [DEBUG] [client/client.go:90] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2024/12/20 08:36:05.175 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:05.175 +00:00] [WARN] [grpcclient/client.go:476] ["fail to get grpc client in the retry state"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:05.175 +00:00] [WARN] [grpcclient/client.go:467] ["retry func failed"] [retried=4] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2024/12/20 08:36:07.048 +00:00] [DEBUG] [config/refresher.go:71] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[m244-etcd-0.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-1.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-2.m244-etcd-headless.milvus.svc.cluster.local:2379]"]
[2024/12/20 08:36:08.376 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2024/12/20 08:36:08.377 +00:00] [DEBUG] [client/client.go:90] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2024/12/20 08:36:08.377 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:08.377 +00:00] [WARN] [grpcclient/client.go:476] ["fail to get grpc client in the retry state"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:12.048 +00:00] [DEBUG] [config/refresher.go:71] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[m244-etcd-0.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-1.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-2.m244-etcd-headless.milvus.svc.cluster.local:2379]"]
[2024/12/20 08:36:14.778 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2024/12/20 08:36:14.779 +00:00] [DEBUG] [client/client.go:90] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2024/12/20 08:36:14.779 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:14.779 +00:00] [WARN] [grpcclient/client.go:476] ["fail to get grpc client in the retry state"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:17.048 +00:00] [DEBUG] [config/refresher.go:71] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[m244-etcd-0.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-1.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-2.m244-etcd-headless.milvus.svc.cluster.local:2379]"]
[2024/12/20 08:36:22.048 +00:00] [DEBUG] [config/refresher.go:71] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[m244-etcd-0.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-1.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-2.m244-etcd-headless.milvus.svc.cluster.local:2379]"]
[2024/12/20 08:36:24.780 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2024/12/20 08:36:24.781 +00:00] [DEBUG] [client/client.go:90] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2024/12/20 08:36:24.781 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:24.781 +00:00] [WARN] [grpcclient/client.go:476] ["fail to get grpc client in the retry state"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2024/12/20 08:36:27.048 +00:00] [DEBUG] [config/refresher.go:71] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[m244-etcd-0.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-1.m244-etcd-headless.milvus.svc.cluster.local:2379,m244-etcd-2.m244-etcd-headless.milvus.svc.cluster.local:2379]"]

milvus组件中的datanode、proxy、querycoord都是find no available datacoord, check datacoord state这个问题无法正常启动

各位大佬求助帮忙解决解决

[yhmo]

Wait @LoveEachDay comment.

[LinHaiSheng580]

value-arm.txt
@LoveEachDay

[LinHaiSheng580]

@LoveEachDay 大佬，求助，万分感谢

[LoveEachDay]

@LinHaiSheng580 From the attached log, we can see that milvus components connect to etcd successfully, while the milvus components don't register themselves correctly. Could you first check the log of datacoord?

[LoveEachDay]

Could you also check that all the pods of message queue are running well?

[LinHaiSheng580]

问题已解决，修改namespace时导致配置文件的问题

[jhyever]

大佬，你这个arm集群是咋部署的呀，有啥想法嘛，x86的部署起来了，arm的milvus集群不知道咋搞，有好些镜像都需要拉对应的arm版本嘛

[LinHaiSheng580]

是的，需要替换arm的镜像