Trino K8S and disk caching

[dishkakrauch]

Hello there.

Does anybode implement this https://trino.io/docs/current/object-storage/file-system-cache.html in kubernetes? Looks like it's best soluiton for baremetal installation but not for the K8S with Deployemnts. There are workarounds:

• custom StafeulSet
• use emptyDir ram mount
• handly addind additional disk to K8S node
• switch to S3 storage (depercated and not gonna make cache works better)

Moved from here trinodb/trino#24395

I didn't get this recomendation in detail trinodb/trino#24395 (comment)

[nineinchnick]

As Ashhar mentioned in the original discussion, StatefulSets are not needed, because the cache does not need to be associated with any specific worker node. You can use whatever volumes you prefer, either emptyDir or other volume types.

I'm not sure what's the concern here. Are you having issues with configuring volumes?

[alexwirehead]

We are using managed k8s in a private cloud. We can only connect additional network-ssd/nvme for cache via PVC. Therefore we wanted to use StatefuleSet with VolumeClaimTemplate for automatic provisioning.

With emptyDir we can connect volume only to the root partition of the cluster node.

There are mentions in the documentation that you should avoid this configuration.

There are a few questions:

• Is it acceptable to use network-ssd/nvme for cache?
• Is it acceptable to use emptyDir with a mount to the root?

If the described options are not acceptable, then we will think how to add local disks to the workernode. In this case StatefulSet will not be necessary.

[nineinchnick]

The docs you referenced are written for bare metal installations, not containers. The recommendation not to use a root mount point is to avoid using up all the available disk space on the host.

AFAIK, regular PVCs used by Deployments can also be dynamically provisioned: https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/#using-dynamic-provisioning

If you really need to use StatefulSets, you can fork the chart, and modify it as needed. emptyDir is a memory based storage, so it might get expensive for large caches.

[alexwirehead]

Thanks for the reply!

We will set up one of the suggested options and test it.