Issue

Hosts getting stuck in either state Reconnecting or Finishing-Reconnect and Rancher server logs include errors like the following: 2019-02-26 12:05:55,265 ERROR [51e1303d-21b2-409f-ba2e-7542e8de4941:9663402] [healthcheckInstanceHostMap:445975] [healthcheckinstancehostmap.remove] [] [ecutorService-3] [c.p.e.p.i.DefaultProcessInstanceImpl] Unknown exception org.jooq.exception.InvalidResultException: Cursor returned more than one result

Pre-requisites

Rancher version lower than 1.6.22

Workaround

1. In the Rancher MySQL database, find all the duplicates by checking column 3 for entries with more than a count of 1 in the return from the following query:

   select host_id,healthcheck_instance_id,count(*) from healthcheck_instance_host_map where removed is null group by host_id,healthcheck_instance_id order by 3;

2. For each healthcheck_instance_id in any row with more than 1 in column 3, run the following command:

   update healthcheck_instance_host_map set state='removed', removed=now(), remove_time=now() where healthcheck_instance_id='<INSERT_HEALTHCHECK_ID>';

3. Wait and watch the hosts view. The hosts should all finish reconnecting and instances should update.

Resolution

Upgrade to 1.6.22+ or 2.x

Further reading

https://github.com/rancher/rancher/issues/15284