APPLIES TO:
MySQL Server - Version 8.0 and later
Information in this document applies to any platform.
SYMPTOMS
On : 8.0 version, Server Crash
ERROR
-----------------------
STEPS
-----------------------
The issue can be reproduced at will with the following steps:
1. Have a high load of transactions on the server
2. The page cleaner gets overloaded
3. At least one innodb page gets truncated from the undo log pages
BUSINESS IMPACT
-----------------------
The issue has the following business impact:
Due to this issue, users cannot trust the server under load without corruption.
CAUSE
A transaction was still active on the table mentioned, however, may not have been required for some time after, or simply the data was on a different page to when the truncation occurred.
Once the truncated page was attempted to be accessed, it was recognised as corrupted and the messages shown in the error log and then the final assertion to avoid any further corruption.
There were large data reads and purging based on end-of-month processes.
Corruption shows up in the mysqld error log like:
2021-08-31T20:03:20.817267-04:00 1960210 [ERROR] [MY-013183] [InnoDB] Assertion failure: ut0ut.cc:553 thread 139840586462976
The ut0ut line shows that an assertion failure was thrown simply because it was unable to read all the required data from the page of the tablespace.
Multiple times in the log there are the following lines relating to truncating UNDO tablespace files.
This only happens in relation to the trx0trx.cc code which is for handling transactions.
2021-08-31T01:33:04.157713-04:00 0 [Note] [MY-011953] [InnoDB] Page cleaner took 20524ms to flush 2002 and evict 0 pages
2021-08-31T01:34:16.779531-04:00 0 [Note] [MY-011953] [InnoDB] Page cleaner took 23558ms to flush 2002 and evict 0 pages
2021-08-31T01:39:57.091897-04:00 0 [Note] [MY-011953] [InnoDB] Page cleaner took 5213ms to flush 2002 and evict 0 pages
2021-08-31T01:40:31.821776-04:00 0 [Note] [MY-011953] [InnoDB] Page cleaner took 15728ms to flush 2002 and evict 0 pages
2021-08-31T01:41:29.927435-04:00 0 [Note] [MY-011953] [InnoDB] Page cleaner took 9172ms to flush 2002 and evict 0 pages
2021-08-31T02:17:48.360155-04:00 0 [Note] [MY-012994] [InnoDB] Truncating UNDO tablespace innodb_undo_002
2021-08-31T02:17:48.373022-04:00 0 [Note] [MY-013000] [InnoDB] Completed truncate of undo tablespace innodb_undo_002.
2021-08-31T02:18:01.107204-04:00 0 [Note] [MY-012994] [InnoDB] Truncating UNDO tablespace undo_003
2021-08-31T02:18:01.119293-04:00 0 [Note] [MY-013000] [InnoDB] Completed truncate of undo tablespace undo_003.
2021-08-31T02:18:12.133887-04:00 0 [Note] [MY-012994] [InnoDB] Truncating UNDO tablespace undo_004
2021-08-31T02:18:12.146118-04:00 0 [Note] [MY-013000] [InnoDB] Completed truncate of undo tablespace undo_004.
2021-08-31T02:18:20.764099-04:00 0 [Note] [MY-012994] [InnoDB] Truncating UNDO tablespace innodb_undo_001
2021-08-31T02:18:20.775965-04:00 0 [Note] [MY-013000] [InnoDB] Completed truncate of undo tablespace innodb_undo_001.
This is explained in the following bug:
Bug.33162828: INNODB: Assertion failure: ut0ut.cc:552
SOLUTION
To implement the solution, please execute the following steps:
1. Download and review the release notes for the 8.0.27 version or higher.
2. Ensure that you have taken a backup of your system before upgrading.
3. Apply the upgrade in a test environment, if possible.
4. Retest the issue.
5. Migrate the solution as appropriate to other environments.