Get Beautiful Runbooks From Kubernetes Yaml with Plural

At Plural, we're building tools that make it easy to deploy open source applications like Prometheus, Grafana, and RabbitMQ, on Kubernetes.

One of the features we provide is programmatic, beautiful runbooks integrated with your deployment pipeline that make common manual devops tasks like scaling cpu and memory a breeze.

Plural comes with built-in runbooks for popular use cases, and you can use them alongside your Plural-deployed open source applications as is. In this blog post, we'll also walk through how, as a developer, you can write your own runbook from scratch.

Writing Your Own Plural Runbook

Let's create a runbook that presents a nice UI for scaling Airflow's Postgres instance.

  1. Make sure that you have installed Airflow and the Plural admin console. For instructions follow the getting started guide here.

  2. From the root of your Plural installation folder, navigate to the application helm folder

    cd <app-name>/helm/<app-name>/
    
  3. Create a new .xml file called db-scaling.xml and paste in the following code:

    <root gap='medium'>
    <box pad='small' gap='medium' direction='row' align='center'>
    <button label='Scale' action='scale' primary='true' headline='true' />
    <box direction='row' align='center' gap='small'>
      <box gap='small' align='center'>
        <timeseries datasource="cpu" label="CPU Usage" />
        <text size='small'>You should set a reservation to 
          roughly correspond to 30% utilization</text>
      </box>
      <box gap='small' align='center'>
        <timeseries datasource="memory" label="Memory Usage" />
        <text size='small'>You should set a reservation to 
          roughly correspond to 60% utilization</text>
      </box>
    </box>
    <box gap='small'>
      <box gap='xsmall'>
        <input placeholder="250m" label='CPU Request' name='cpu'>
          <valueFrom 
            datasource="statefulset" 
            doc="kubernetes.raw" 
            path="spec.template.spec.containers[0].resources.requests.cpu" />
        </input>
        <input placeholder="1Gi" label='Memory Request' name='memory'>
          <valueFrom 
            datasource="statefulset" 
            doc="kubernetes.raw" 
            path="spec.template.spec.containers[0].resources.requests.memory" />
        </input>
      </box>
    </box>
    </box>
    <box pad='small' gap='medium' direction='row' align='center'>
    <box direction='row' width='70%' align='center'>
      <text size='small'>You can also add more replicas to provide failover in case of outages, or optionally remove them to save cost</text>
    </box>
    <box direction='row' gap='small' width='30%' align='center'>
      <input datatype='int' placeholder="1" label='Replicas' name='replicas'>
        <valueFrom 
          datasource="statefulset" 
          doc="kubernetes.raw" 
          path="spec.replicas" />
      </input>
    </box>
    </box>
    <box width='100%' gap='small'>
    <text size='small'>Be sure to scale your rabbitmq nodes within your nodes capacities, listed here:</text>
    <table width='100%' datasource='nodes' path='nodes'>
      <tableColumn path='metadata.name' header='name' width='33%' />
      <tableColumn path='status.capacity.cpu' header='cpu' width='33%' />
      <tableColumn path='status.capacity.memory' header='memory' width='33%' />
    </table>
    </box>
    </root>
    

    This draws two boxes on a single row, each displaying a timeseries from the datasources cpu and memory respectively. Underneath, it draws another box with two input boxes, both of which is displaying a value from the datasource statefulset. It also draws a button with the scale action attached. Don't worry about these values right now, we'll come back to them below.

    Screen-Shot-2021-11-12-at-11.00.13-AM

    (For a full breakdown of the styling properties of the xml tags that have been configured by Plural, refer to the docs here.)

  4. From the root of your Plural installation repo, navigate to the templates directory in your helm directory.

    cd <app-name>/helm/<app-name>/templates
    
  5. Create a runbook.yaml file

    runbook.yaml is where you hydate your db-scaling.xml file with data sources. You can configure it a number of different ways, but in particular you'll want to:

    • Define a display field -- this is where you link your xml template
    • Define a datasources field -- this is where you put your datasources. Plural currently supports three data source types. You can view the full list here.
    • Define an actions field -- this is the update to the application's helm chart that Plural will perform.

    So for example:

    apiVersion: platform.plural.sh/v1alpha1
    kind: Runbook
    metadata:
      name: db-scaling
      labels:
        platform.plural.sh/pinned: 'true'
    {{ include "airflow.labels" . | indent 4 }}
    spec:
      name: Postgres Scaling
      description: overview of how to accurately scale airflow's postgres instance
      alerts:
      - name: AirflowPostgresCPU
      - name: AirflowPostgresMEM
      display: |-
    {{ .Files.Get "runbooks/db-scaling.xml" | indent 4 }}
      datasources:
      - name: cpu
        type: prometheus
        prometheus:
          format: cpu
          legend: $pod
          query: sum(rate(container_cpu_usage_seconds_total{namespace="{{ .Release.Namespace }}",pod=~"plural-airflow-[0-9]+"}[5m])) by (pod)
      - name: memory
        type: prometheus
        prometheus:
          format: memory
          legend: $pod
          query: sum(container_memory_working_set_bytes{namespace="{{ .Release.Namespace }}",pod=~"plural-airflow-[0-9]+"}) by (pod)
      - name: statefulset
        type: kubernetes
        kubernetes:
          resource: statefulset
          name: plural-airflow
      - name: nodes
        type: nodes
      actions:
      - name: scale
        action: config
        redirectTo: '/'
        configuration:
          updates:
          - path: 
            - airflow
            - postgres
            - resources
            - requests
            - cpu
            valueFrom: cpu
          - path:
            - airflow
            - postgres
            - resources
            - requests
            - memory
            valueFrom: memory
          - path:
            - airflow
            - postgres
            - replicas
            valueFrom: replicas
    
  6. Build && Redeploy. From the root of your Plural installation folder, run:
    plural build
    git add .
    git commit -m "Add runbook
    git push
    plural deploy

  7. Navigate to the console subdomain, and click on the runbooks tab

    Screen-Shot-2021-11-12-at-11.15.06-AM

    Click on the runbook and you'll see a dashboard with memory + cpu metrics pulled from Prometheus, with the ability to scale those metrics.

Next Steps With Plural

That was a quick overview of how to create interactive devops runbooks for your Plural-installed applications. Plural offers a number of other goodies, in particular the Admin Console which serves as a central control panel.

For more about Plural, our full docs are at docs.plural.sh. If you're a company and are interested in trying out Plural, fill out the form below and we will get back to you.

If you run into any problems or have suggestions for what else you’d like to use Plural for, please let us know in our Discord.

Yiren Lu

Yiren Lu