Rook v1.11 Storage Enhancements
The Rook v1.11 release is out! v1.11 is another feature-filled release to improve storage for Kubernetes. Thanks again to the community for all the great support in this journey to deploy storage in production.
The statistics continue to show Rook community growth since the v1.10 release in August:
- 10.3K to 10.8K Github stars
- 268M to 278M Container downloads
- 5.4K to 5.8K Slack members
- 6.8K to 7.0K Twitter followers
We have a lot of new features for the Ceph storage provider that we hope you’ll be excited about with the v1.11 release!
Ceph-CSI v3.8
The v3.8 release of the Ceph-CSI driver is now the version deployed by default with Rook. The driver has a number of important updates to add more storage features available to clients.
RBD Read Affinity
RBD volumes have the ability to read data from an OSD in proximity to the client, according to OSD locations defined in the CRUSH map and topology labels on nodes. The volumes leverage the krbd map options to allow serving reads with affinity to OSDs nearby.
For example, a typical cluster will have three replicas of data spread across zones A, B, and C. If a pod is running in zone A, the pod will now read the data from the OSD in zone A instead of choosing to randomly read from the other zones. Thus, traffic across zones can be reduced and cut costs.
fscrypt Support
fscrypt support was added for both RBD and CephFS volumes, snapshots, and clones.
Shallow Volumes
Shallow volumes are now enabled by default for ROX volumes. This means CephFS subvolume snapshots can be exposed as regular read-only PVCs. No data cloning is performed and provisioning such volumes is done in constant time.
Msgr2 Protocol
Ceph msgr2 is the newer protocol by which data on the wire can be encrypted or compressed. While msgr2 has been enabled by Rook for many releases, we have still not made it the default protocol due to the requirement for newer kernels. Now we have provided an option that allows clusters to enable msgr2 exclusively. This will allow clusters to then be configured for encryption and compression when needed. We recommend for all clusters that have kernel version 5.11 or newer to enable this feature since the msgr1 protocol is no longer necessary in that case.
Clusters upgraded from previous versions of Rook can also enable msgr2. Upgraded clusters will simply require an additional step to disable msgr1. After the msgr2 setting is enabled, all mons will need to be failed over to start using the msgr2 support, and volumes will need to be re-mounted to pick up the change.
Ceph Exporter
Ceph has introduced a new daemon for scalability of collecting metrics in large clusters. The Ceph Exporter is a daemon that runs on each node to collect the performance counters reported by Ceph daemons. Prometheus then will scrape each exporter daemon for the counters. No longer is the Ceph mgr daemon a scalability bottleneck for reporting metrics.
Mirroring with Overlapping Networks
When configuring mirroring across clusters, the two clusters must be addressable across the network. While the most common solution is to use the host network, this is not always desired. Now mirroring can be configured across clusters that have overlapping CIDRs by using the Multi-Cluster Services (MCS) API. Cross-cluster connectivity is configured from pods to remote services using global IPs. We have tested this solution with Submariner Globalnet.
This solution requires using global IPs for the mons and OSDs, which will allow communication both across clusters and within the clusters for RADOS operations. While Rook v1.11 has the changes necessary to support this configuration, the feature will require a Ceph change which will be released (hopefully soon!) with Ceph v17.2.6.
OSD Encryption Key Rotation
A feature we’ve been anticipating to enhance OSD encryption is the ability to rotate the encryption keys. The key rotation ensures that even if key encryption keys are compromised, the data at rest will not be compromised since the keys are frequently rotated. The rotation is performed on the Key Encryption Keys, which means the data will not need to be re-encrypted at the time of rotation.
At the time of the v1.11 release, we are still in final review and testing of this feature, so stay tuned for a patch release soon! When the rotation is available, see the Key Management System documentation for usage. The initial feature will only support rotation for keys stored in K8s secrets. We will be adding the rotation support for KMS solutions soon as well.
Bucket Notifications and Topics
In Rook v1.8, we released the Bucket notifications and topics for object stores in experimental mode. Now in v1.11 we have declared the feature as stable. Thanks to the feedback from the community that the notifications and topics are working well in production!
Ceph Mgr Readiness
The Ceph Mgr readiness probe is an internal improvement that we would like to call out. To manage the active and standby Ceph mgr daemons, we previously had implemented a sidecar on the mgr pod to update the mgr services when the active Ceph mgr changed. Now this is simplified by using a readiness probe on the mgr so that K8s will automatically handle keeping the services updated with the active mgr. There is no more need for the side car.
Krew Plugin
The Rook Krew plugin is a tool we created last year to help troubleshoot and maintain your clusters. We hope you are finding this tool useful and look forward to your feedback! The latest addition to the Krew plugin with the v0.4 release includes a command to help recover from a disaster where multiple mons were lost and restore mon quorum from a single mon.
Stay tuned for more updates to the plugin. We are adding more troubleshooting commands and reworking some of the internals for a new release soon!
Breaking Changes
Kubernetes v1.21+
Kubernetes v1.21 is now the minimum version supported by Rook, which means we run CI tests against v1.21 and newer. If you still require running an older K8s version we haven’t done anything to prevent running Rook, we simply just do not have any test validation on older versions.
PSPs Disabled
With the deprecation of PSPs in K8s 1.21 and their removal in 1.25, we decided it was time to disable the PSPs by default in the helm chart. We also removed documentation for enabling the PSPs since we don’t expect them to be used anymore. If you still require them, just refer to the v1.10 Rook docs instead of the latest docs.
MachineDisruptionBudgets
Support is removed for machine disruption budgets. Were you using them? We didn’t think so!
What’s Next?
As we continue the journey to develop reliable storage operators for Kubernetes, we look forward to your ongoing feedback. Only with the community is it possible to continue this fantastic momentum.
There are many different ways to get involved in the Rook project, whether as a user or developer. Please join us in helping the project continue to grow on its way beyond the v1.11 milestone!
