In another question we attempted to update a table in our MySQL8 db. That didn't work out, but since then our database (even after both DB and machine restarts) seems painfully slow at the strangest times.
These things might be unrelated but, for example here is an entry from the slow query log:
--
# Query_time: 15.095238 Lock_time: 0.000127 Rows_sent: 0 Rows_examined: 0 Rows_affected: 0 Bytes_sent: 1594
SET timestamp=1685664330;
SELECT `seller_feedbacks`.* FROM `seller_feedbacks` WHERE (seller_feedbacks.tenant_id = 18251) AND `seller_feedbacks`.`order_id` = 229621225
If I do an explain on this query:
mysql> explain SELECT `seller_feedbacks`.* FROM `seller_feedbacks` WHERE (seller_feedbacks.tenant_id = 18251) AND `seller_feedbacks`.`order_id` = 229621225;
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+--------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+--------------------------------+
| 1 | SIMPLE | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | no matching row in const table |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+--------------------------------+
1 row in set, 1 warning (0.00 sec)
mysql> SELECT `seller_feedbacks`.* FROM `seller_feedbacks` WHERE (seller_feedbacks.tenant_id = 18251) AND `seller_feedbacks`.`order_id` = 229621225;
Empty set (0.00 sec)
The orders
table is large: >200 million rows
The seller_feedback
table not so much: 800,000 rows or so.
This query, with different values, appears over 10,000 times in our slow query log in the last 3 days. And the size of the slow query log has tripled.
I know this kind of vague question is difficult and frowned upon, but I do not even know where to start investigating what has happened.
We are running MySQL 8.0.23-14 Percona Server (GPL), Release 14, Revision 3558242
on a Centos
box
EDIT 1:
mysql> ANALYZE TABLE orders;
+-----------------------------+---------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+-----------------------------+---------+----------+----------+
| xxx_production.orders | analyze | status | OK |
+-----------------------------+---------+----------+----------+
1 row in set (0.76 sec)
mysql> ANALYZE TABLE seller_feedbacks;
+---------------------------------------+---------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+---------------------------------------+---------+----------+----------+
| xxx_production.seller_feedbacks | analyze | status | OK |
+---------------------------------------+---------+----------+----------+
1 row in set (0.18 sec)
MySQL > CREATE TABLE `seller_feedbacks` (
`id` int NOT NULL AUTO_INCREMENT,
`tenant_id` int DEFAULT NULL,
`order_id` int DEFAULT NULL,
`customer_id` int DEFAULT NULL,
`rating` int DEFAULT NULL,
`comment` text,
`response` text,
`arrived_on_time` tinyint(1) DEFAULT NULL,
`item_as_described` tinyint(1) DEFAULT NULL,
`feedback_date` datetime DEFAULT NULL,
`role` varchar(255) DEFAULT NULL,
`customer_service` varchar(255) DEFAULT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
`vendor_order_id` varchar(255) DEFAULT NULL,
`customer_email` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `index_seller_feedbacks_on_order_id` (`order_id`),
KEY `index_seller_feedbacks_on_customer_id` (`customer_id`),
KEY `index_seller_feedbacks_on_tenant_id_and_customer_id` (`tenant_id`,`customer_id`),
KEY `index_seller_feedbacks_on_tenant_id_and_order_id` (`tenant_id`,`order_id`),
KEY `index_seller_feedbacks_on_tenant_id_and_feedback_date` (`tenant_id`,`feedback_date`),
CONSTRAINT `fk_rails_2e1355dc58` FOREIGN KEY (`tenant_id`) REFERENCES `tenants` (`id`),
CONSTRAINT `fk_rails_3f223c66ad` FOREIGN KEY (`order_id`) REFERENCES `orders` (`id`),
CONSTRAINT `fk_rails_c10b580c0c` FOREIGN KEY (`customer_id`) REFERENCES `customers` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1481045 DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED |
This table is created by Rails migrations.
EDIT 2: This is the JSON version of the EXPLAIN.
{
"query_block": {
"select_id": 1,
"cost_info": {
"query_cost": "3.85"
},
"table": {
"table_name": "seller_feedbacks",
"access_type": "ref",
"possible_keys": [
"index_seller_feedbacks_on_tenant_id_and_customer_id",
"index_seller_feedbacks_on_tenant_id_and_order_id",
"index_seller_feedbacks_on_tenant_id_and_feedback_date"
],
"key": "index_seller_feedbacks_on_tenant_id_and_customer_id",
"used_key_parts": [
"tenant_id"
],
"key_length": "5",
"ref": [
"const"
],
"rows_examined_per_scan": 11,
"rows_produced_per_join": 11,
"filtered": "100.00",
"cost_info": {
"read_cost": "2.75",
"eval_cost": "1.10",
"prefix_cost": "3.85",
"data_read_per_join": "33K"
},
"used_columns": [
"id",
"tenant_id",
"order_id",
"customer_id",
"rating",
"comment",
"response",
"arrived_on_time",
"item_as_described",
"feedback_date",
"role",
"customer_service",
"created_at",
"updated_at",
"vendor_order_id",
"customer_email"
]
}
}
}
SHOW CREATE TABLE
and doANALYZE TABLE
.pt-osc
never completed so I don't think the table swap happened. But that is an interesting point!SELECT
? It seems not. Please provideEXPLAIN FORMAT=JSON SELECT ...
. It may give a clue of what it was thinking about. (Yeah, I know this seems to be irrelevant to pt-osc.)