Rancher pre v1.6.22 "Hosts stuck Reconnecting" Rancher server logs show 'Cursor returned more than one result'

This document (000020182) is provided subject to the disclaimer at the end of this document.

Situation

Issue

Hosts getting stuck in either state Reconnecting or Finishing-Reconnect and Rancher server logs include errors like the following: 2019-02-26 12:05:55,265 ERROR [51e1303d-21b2-409f-ba2e-7542e8de4941:9663402] [healthcheckInstanceHostMap:445975] [healthcheckinstancehostmap.remove] [] [ecutorService-3] [c.p.e.p.i.DefaultProcessInstanceImpl] Unknown exception org.jooq.exception.InvalidResultException: Cursor returned more than one result

Pre-requisites

Rancher version lower than 1.6.22

Workaround

  1. In the Rancher MySQL database, find all the duplicates by checking column 3 for entries with more than a count of 1 in the return from the following query:

    select host_id,healthcheck_instance_id,count(*) from healthcheck_instance_host_map where removed is null group by host_id,healthcheck_instance_id order by 3;
  2. For each healthcheck_instance_id in any row with more than 1 in column 3, run the following command:

    update healthcheck_instance_host_map set state='removed', removed=now(), remove_time=now() where healthcheck_instance_id='<INSERT_HEALTHCHECK_ID>';
  3. Wait and watch the hosts view. The hosts should all finish reconnecting and instances should update.

Resolution

Upgrade to 1.6.22+ or 2.x

Further reading

https://github.com/rancher/rancher/issues/15284

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020182
  • Creation Date: 13-Jul-2021
  • Modified Date:13-Jul-2021
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center