How to recover monasca node after "disk full"

This document (7023932) is provided subject to the disclaimer at the end of this document.

Environment

SUSE OpenStack Cloud 8

Situation

After the disk went full where the log files resides, some of the services are down

Resolution

zookeper:
Move/remove the empty log file, then restart zookeeper.service

monasca storm:
Open the /etc/storm/storm.yaml file. Verify there is a "storm.zookeeper.servers:" entry with an IP address specified (if not, then storm is misconfigured and may need to be fixed).

Using that IP, run '/usr/bin/zkCli.sh -server <ip>:2181', this will start a command line interface.
Run 'ls /storm' and verify there is a list.
Should look like looks like [assignments, backpressure, blobstoremaxkeysequencenumber, credentials, nimbuses, logconfigs, leader-lock, storms, errors, supervisors, workerbeats, blobstore]

Removing with 'rmr /storm' command and run 'quit' to log out of zkCli.sh should clear out the broken references. Restart zookeeper.service,storm-nimbus.service and storm-supervisor.service to get it recreated.

Cause

Following error message in logs:

zookeper:

java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) ...
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)

Issue is that an empty logfile (size is zero) or with invalid headers exists:
/var/lib/zookeeper/data/version-2/

monasca storm:
org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader 
nimbus from seed hosts [localhost]. Did you specify a valid list of nimbus
hosts for config nimbus.seeds?

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7023932
  • Creation Date: 12-Jun-2019
  • Modified Date:03-Mar-2020
    • SUSE Open Stack Cloud

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center