CronJobs fail to run in a Kubernetes v1.14 cluster, with more than 500 Job resources: "expected type *batchv1.JobList, got type *internalversion.List"

This document (000020208) is provided subject to the disclaimer at the end of this document.

Situation

Issue

In a Kubernetes v1.14 cluster, CronJobs fail to run when there are more than 500 Job resources in the cluster. The Kubernetes controller manager logs show errors of the format {"log":"E0818 18:25:50.081946 1 cronjob_controller.go:117] expected type *batchv1.JobList, got type *internalversion.List\n","stream":"stderr","time":"2019-08-18T18:25:50.082127727Z"}.

Pre-requisites

  • A Kubernetes cluster, running Kubernetes v1.14, from v1.14.0 - v1.14.6
  • More than 500 Job resources in the cluster

Workaround

To mitigate the issue you should ensure that there are fewer than 500 Job resources in the cluster, you can view all Jobs with kubectl get jobs --all-namespaces -o wide.

You should aim to delete completed Jobs to reduce the total number below 500. You can also check and adjust the configured job history limits for CronJobs to reduce the number of Jobs maintained for completed CronJobs per the Kubernetes documentation.

Resolution

The issue was tracked in Kubernetes GitHub Issue #77465 and a fix was released in Kubernetes v1.14.7.

A Kubernetes v1.14 patch release of v1.14.7 or above, including this fix, is available in Rancher v2.2, starting with v2.2.9 (v1.14.8), and v2.3, starting with v2.3.0 (v1.14.7). Similarly the fix is available via the RKE CLI in v0.2, starting with v0.2.9 (v1.14.8), and v0.3, starting with v0.3.0 (v1.14.7).

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020208
  • Creation Date: 06-May-2021
  • Modified Date:06-May-2021
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center
Report a Software Vulnerability Submit Tips, Tricks, and Tools Download Free Tools