Resizing a PersistentVolume Claimed by a StatefulSet with the Plural Operator

Suppose you're running a MySQL deployment on Kubernetes and the space that you allocated initially is not enough – you would like to increase it from 50GB to 100GB. How would you go about doing this?

Well, recall that with Kubernetes, you deploy stateless applications with Deployment and stateful applications (like MySQL) with StatefulSet. StatefulSet manages pods that are backed by a Kubernetes storage object, for example PersistentVolume.

With Kubernetes v1.11, you can now resize an existing PersistentVolume simply by editing the PersistentVolumeClaim (PVC) object. However, there is still no way to automatically resize volumes that are managed by a StatefulSet.

Here at Plural, our goal is to make it easy to deploy open source applications like Airflow, Grafana, and RabbitMQ on Kubernetes.

The Plural operator automates some manual tasks that are involved in the setup of the more complicated open source applications.

For example, in order to resize a PersistentVolume that has been claimed by a StatefulSet object, the Plural Operator controller implements the following steps:

  1. Delete the StatefulSet object without deleting the underlying pods.

    log.Info("deleting statefulset while orphaning pods", "statefulset", statefulset.Name)
        if err := r.Delete(ctx, &statefulset, client.PropagationPolicy(metav1.DeletePropagationOrphan)); err != nil {
            log.Error(err, "failed to delete", "statefulset", statefulset.Name)
            return ctrl.Result{}, err
        }
    
  2. Set allowVolumeExpansion: true on the StorageClass associated with your PVC.

    // find the storageClass associated with your PVC
    var storageClass storagev1.StorageClass
        if claim.Spec.StorageClassName == nil {
            classes := &storagev1.StorageClassList{}
            if err := r.List(ctx, classes); err != nil {
                log.Error(err, "could not list storage classes")
                return ctrl.Result{}, err
            }
    
            found := false
            for _, class := range classes.Items {
                fmt.Printf("%+v", class)
                if _, ok := class.Annotations["storageclass.kubernetes.io/is-default-class"]; ok {
                    storageClass = class
                    found = true
                    break
                }
            }
    
            if !found {
                err := fmt.Errorf("Could not find default storage class")
                log.Error(err, "could not find default storage class")
                return ctrl.Result{}, err
            }
        } else {
            if err := r.Get(ctx, types.NamespacedName{Name: *claim.Spec.StorageClassName}, &storageClass); err != nil {
                log.Error(err, "failed to get storageClass", "storageclass", *claim.Spec.StorageClassName)
                return ctrl.Result{}, err
            }
        }
    
    
    // set AllowVolumeExpansion on the PVC
    if storageClass.AllowVolumeExpansion == nil || !*storageClass.AllowVolumeExpansion {
            storageClass.AllowVolumeExpansion = resources.BoolPtr(true)
            if err := r.Update(ctx, &storageClass); err != nil {
                log.Error(err, "Failed to enable volume expansion for storage class", "storageclass", storageClass.Name)
                return ctrl.Result{}, err
            }
        }
    
  3. Modify the PVC object with the desired storage

    for i := 0; i < replicas; i++ {
        claimName := fmt.Sprintf("%s-%s-%d", claim.Name, statefulset.Name, i)
        var pvc corev1.PersistentVolumeClaim
        nsn := types.NamespacedName{
            Namespace: resize.Namespace,
            Name:      claimName,
        }
    
        if err := r.Get(ctx, nsn, &pvc); err != nil {
            log.Error(err, "failed to get pvc", "pvc", pvc.Name)
            return ctrl.Result{}, err
        }
    
        pvc.Spec.Resources.Requests["storage"] = quant
    
        if err := r.Update(ctx, &pvc); err != nil {
            log.Error(err, "failed to resize statefulset pvc", "pvc", pvc.Name)
            return ctrl.Result{}, err
        }
       }
    
  4. Recreate the StatefulSet with the new storage request

    // create a new set of VolumeClaims
    claims := statefulset.Spec.VolumeClaimTemplates
        newClaims := make([]corev1.PersistentVolumeClaim, len(claims))
        var claim corev1.PersistentVolumeClaim
        for i, cl := range claims {
            if cl.Name == resize.Spec.PersistentVolume {
                if quant.Cmp(cl.Spec.Resources.Requests["storage"]) == 0 {
                    log.Info("No change needed for storage")
                    return r.cleanup(ctx, &resize)
                }
    
                cl.Spec.Resources.Requests["storage"] = quant
                claim = cl
            }
    
            newClaims[i] = cl
        }
    
    // create a new StatefulSet and set the volumeClaimTemplates with the new claims
    newStatefulSet := &appsv1.StatefulSet{}
        newStatefulSet.Spec = statefulset.Spec
        newStatefulSet.Spec.VolumeClaimTemplates = newClaims
        newStatefulSet.Name = statefulset.Name
        newStatefulSet.Namespace = statefulset.Namespace
        newStatefulSet.Labels = statefulset.Labels
        newStatefulSet.Annotations = statefulset.Annotations
        if newStatefulSet.Spec.Template.Annotations == nil {
            newStatefulSet.Spec.Template.Annotations = map[string]string{}
        }
    
        newStatefulSet.Spec.Template.Annotations["platform.plural.sh/rotate"] = time.Now().String()
        if err := r.Create(ctx, newStatefulSet); err != nil {
            log.Error(err, "failed to recreate statefulset", "statefulset", newStatefulSet.Name)
            return ctrl.Result{}, err
        }
    

Next Steps With Plural

That was a quick overview of how storage scaling is automatically handled for your Plural-installed applications. Plural offers a number of other goodies, in particular the Admin Console which serves as a central control panel.

For more about Plural, our full docs are at docs.plural.sh. If you're a company and are interested in trying out Plural, fill out the form below and we will get back to you.

If you run into any problems or have suggestions for what else you’d like to use Plural for, please let us know in our Discord.

Yiren Lu

Yiren Lu