Nova DB out of sync - VM not running on expected hypervisor

This document (7021199) is provided subject to the disclaimer at the end of this document.

Environment

SUSE OpenStack Cloud 7

Situation

During migration nova thinks the instance which can't be evacuated is running on a different compute node.

nova list
...
 ID                                   | Name      | Tenant ID                        | Status  | Task State | Power State | Networks                 |

+--------------------------------------+-----------+----------------------------------+---------+------------+-------------+--------------------------+

...

| aa8e645c-c737-49c5-ae41-e7ff19f75ea6 | testserver | 3ea7d60e677d4012b757284f56a159fc | ACTIVE  | -          | Running     | test_network=10.10.10.9  |

nova show <instance-id>

| OS-EXT-SRV-ATTR:hypervisor_hostname  | d00-00-00-c7-01-01.cloud.example.com

| name                                 | testserver

nova hypervisor-servers <compute-node-fqdn>

+--------------------------------------+-------------------+---------------+-------------------------------------------+

| ID                                   | Name              | Hypervisor ID | Hypervisor Hostname                       |

+--------------------------------------+-------------------+---------------+-------------------------------------------+
..
| aa8e645c-c737-49c5-ae41-e7ff19f75ea6 | instance-00000035 | 1             | d00-00-00-c7-01-01.cloud.example.com |

...

on compute node:

root@:~ # virsh list --all
Id    Name                           State
----------------------------------------------------
1     instance-00000036              running

but there is no such instance here

On other compute node where instance is running:

nova hypervisor-servers d00-00-00-c7-01-02.cloud.example.com

+----+------+---------------+---------------------+

| ID | Name | Hypervisor ID | Hypervisor Hostname |

+----+------+---------------+---------------------+

...

an empty list for nova.
but virsh sees the instance:

d00-00-00-c7-01-01:~ # virsh list
Id    Name                           State
----------------------------------------------------
187   instance-00000035              running




Resolution

Resolution would be to update nova database and reboot instance

On controller retrieve database details:

grep postgr /etc/nova/nova.conf.d/*


access to database:

psql -U nova -d nova -W -h <ip-from-config>
SELECT * FROM instances WHERE display_name = 'testserver'; 

Get the uuid from this query and update:

UPDATE instances SET host = 'd00-00-00-c7-01-01' WHERE uuid = 'UUID_OF_INSTANCE'; 

Exit the database and then run nova reboot testserver. After reboot you should check if nova list points to the correct host for that instance.

Cause


Additional Information


Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7021199
  • Creation Date: 14-Aug-2017
  • Modified Date:03-Mar-2020
    • SUSE Open Stack Cloud

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center