Automating Longhorn Recurring Jobs Assignment Using Labels

SUSE Enterprise Storage - Longhorn

As the cluster scales and the number of volumes increases, there is a need to automate the assignment of Longhorn recurring jobs to a set of volumes. To streamline this process, the approach involves using volume labels to group volumes, allowing recurring jobs to be automatically applied based on those labels.

Labels on the PVCs can be used to automate the assignment of Longhorn recurring jobs to a set of volumes. Add the labels recurring-job.longhorn.io/source=enabled and recurring-job-group.longhorn.io/<GROUP_NAME>=enabled to the PVCs. For example:

1. 1. Create a recurring job with a group name (e.g., "test-group" shown below).

      > kubectl get recurringjobs.longhorn.io -n longhorn-system -oyaml
      apiVersion: v1
      items:
      - apiVersion: longhorn.io/v1beta2
        kind: RecurringJob
        metadata:
          creationTimestamp: "2025-07-11T08:08:14Z"
          generation: 3
          name: c-mptbs9
          namespace: longhorn-system
          resourceVersion: "1247498"
          uid: 6d88a149-1ca9-489e-a358-7f2b0d326610
        spec:
          concurrency: 1
          cron: '*/2 * * * *'
          groups:
          - test-group
          labels: {}
          name: c-mptbs9
          parameters: {}
          retain: 0
          task: filesystem-trim
        status:
          executionCount: 28
          ownerID: node-xxxxx
      kind: List
      metadata:
        resourceVersion: ""

      Above example creates a recurring job for filesystem-trim on the volumes from group test-group. Alternatively, it can be created via the UI: Longhorn UI >> Recurring Job >> Create Recurring Job. 

   2. Label the PVC with the following labels:

      > kubectl -n longhorn-training get pvc --show-labels|grep test-pvc
      test-pvc              Bound    pvc-646002c0-9fc7-4940-90b9-fdeffdaba44f   1Gi        RWO            longhorn       <unset>                 97m   recurring-job-group.longhorn.io/test-group=enabled,recurring-job.longhorn.io/source=enabled

   3. Verify in the longhorn-manager pod logs that the PVC is added to the recurring job:

      kubectl -n longhorn-system logs -l app=longhorn-manager
      
      time="2025-07-11T09:25:34Z" level=info msg="Adding Volume pvc-646002c0-9fc7-4940-90b9-fdeffdaba44f recurring job label recurring-job-group.longhorn.io/test-group: enabled" func=controller.syncRecurringJobLabelsToTargetResource file="recurring_job_controller.go:604" controller=longhorn-volume node=node-xxxx

   4. Confirm that the cronjob executed fstrim from the job pod logs:

      kubectl -n longhorn-system logs -l job-name=<job_name>
      
      time="2025-07-11T09:26:01Z" level=info msg="Found 1 volumes with recurring job c-mptbs9" func=app.recurringJob file="recurring_job.go:156"
      Fri, Jul 11 2025 2:56:01 pm
      time="2025-07-11T09:26:01Z" level=info msg="Creating job" func=app.startVolumeJob file="recurring_job.go:195" concurrent=1 executionCount=23 groups=test-group job=c-mptbs9 labels="{\"RecurringJob\":\"c-mptbs9\"}" parameters="map[]" retain=0 task=filesystem-trim volume=pvc-646002c0-9fc7-4940-90b9-fdeffdaba44f
      Fri, Jul 11 2025 2:56:01 pm
      time="2025-07-11T09:26:01Z" level=info msg="job starts running" func="app.(*Job).run" file="recurring_job.go:294" executionCount=23 labels="map[RecurringJob:c-mptbs9]" namespace=longhorn-system retain=0 snapshotName=c-mptbs9-588e71a7-9e9e-4756-b0ce-34875f8d4575 task=filesystem-trim volumeName=pvc-646002c0-9fc7-4940-90b9-fdeffdaba44f
      Fri, Jul 11 2025 2:56:01 pm
      time="2025-07-11T09:26:01Z" level=info msg="Running recurring filesystem trim for volume pvc-646002c0-9fc7-4940-90b9-fdeffdaba44f" func="app.(*Job).run" file="recurring_job.go:321" executionCount=23 labels="map[RecurringJob:c-mptbs9]" namespace=longhorn-system retain=0 snapshotName=c-mptbs9-588e71a7-9e9e-4756-b0ce-34875f8d4575 task=filesystem-trim volumeName=pvc-646002c0-9fc7-4940-90b9-fdeffdaba44f

      Additional Details of recurring jobs are explained here. Once the source label on the PVC is enabled, Longhorn starts to sync the recurring-job label from PVC to Longhorn Volume, since Volume is the real resource that Longhorn manages. So, Longhorn needs to sync the label from PVCs to Volumes and further use these labels to select Volumes to do recurring jobs.