'SIGSEGV: segmentation violation' in prometheus container of the prometheus-project-monitoring-0 Pod when enabling Project monitoring on the System Project

This document (000020219) is provided subject to the disclaimer at the end of this document.

Situation

Issue

Enabling project monitoring in a Rancher v2.2 cluster, in which cluster monitoring is enabled, fails with the Prometheus Pod in a CrashLoopBackOff.

The prometheus container in the prometheus-project-monitoring StatefulSet fails with an error of the following format:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x669c12]
goroutine 437 [running]:
net/http.(*Client).deadline(0x0, 0xc005381070, 0x40bb8f, 0xc0055e3600)
/usr/local/go/src/net/http/client.go:187 +0x22
net/http.(*Client).do(0x0, 0xc005cdaa00, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:527 +0xab
net/http.(*Client).Do(0x0, 0xc005cdaa00, 0x23, 0xc002802230, 0x9)
/usr/local/go/src/net/http/client.go:509 +0x35
github.com/prometheus/prometheus/scrape.(*targetScraper).scrape(0xc0060fa960, 0x1fd4a60, 0xc00010ec60, 0x1fb2760, 0xc0002eb110, 0x0, 0x0, 0x0, 0x0)
/app/scrape/scrape.go:471 +0x111
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc00616a100, 0xdf8475800, 0x2540be400, 0x0)
/app/scrape/scrape.go:813 +0x487
created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
/app/scrape/scrape.go:336 +0x45d

Pre-requisites

  • A cluster managed by Rancher v2.2
  • Cluster monitoring enabled and Project monitoring enabled on the System project

Resolution

Project monitoring is not compatible with the Rancher System project and should not be enabled in the System project. Starting with Rancher v2.3.0 monitoring of the System project is performed by cluster monitoring, when this is enabled, and the UI prevents enabling of project monitoring on the System project.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020219
  • Creation Date: 06-May-2021
  • Modified Date:06-May-2021
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center