openstack-记一次虚机创建失败问题

d

背景

某天，突然openstack无法创建虚拟机，MQ消息大量堆积

现象

  1. nova-conductor报大量数据库超时
1
2
3
4
5
6
7
8
9
[SQL: UPDATE services SET updated_at=%(updated_at)s, report_count=%(report_count)s, last_seen_up=%(last_seen_up)s WHERE services.id = %(services_id)s]
[parameters: {'last_seen_up': datetime.datetime(2022, 8, 9, 11, 4, 54, 153870), 'services_id': 7, 'updated_at': datetime.datetime(2022, 8, 9, 11, 4, 54, 256605), 'report_count': 2340897}]
(Background on this error at: http://sqlalche.me/e/e3q8)
2022-08-09 19:05:00.293 67 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
2022-08-09 19:05:00.293 67 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 92, in _report_state
2022-08-09 19:05:00.293 67 ERROR nova.servicegroup.drivers.db     service.service_ref.save()
2022-08-09 19:05:00.293 67 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2022-08-09 19:05:00.293 67 ERROR nova.servicegroup.drivers.db     return fn(self, *args, **kwargs)
2022-08-09 19:05:00.293 67 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/service.py", line 406, in save

2.