Restauración de BD MySQL OVM Manager

por Alberto Pereiro | 23 Ene 2018 | Oracle VM, portada, slider-post | 0 Comentarios

Esta mañana se produjo un fallo hardware en uno de los servidores de laboratorio

con OVM Server 3.4. En este entorno, OVM Manager está desplegado dentro del
propio clúster OVM, y la incidencia provocó su caída abrupta. Al arrancar la
máquina de manera manual desde el servidor OVM con el comando xm create, el
SO arrancó correctamente, pero no los componentes OVM. ¿El motivo?
Corrupción de datos en la BD MySQL tras la caída.

1 180123 17:53:24 mysqld_safe Starting mysqld daemon with databases from /u01/app/oracle/mysql/data
2 2018-01-23 17:53:24 0 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future
3 release.
4 2018-01-23 17:53:24 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --
5 explicit_defaults_for_timestamp server option (see documentation for more details).
6 2018-01-23 17:53:24 0 [Note] /usr/sbin/mysqld (mysqld 5.6.32-enterprise-commercial-advanced) starting as
7 process 3816 ...
8 2018-01-23 17:53:24 3816 [Note] Plugin 'FEDERATED' is disabled.
9 2018-01-23 17:53:24 3816 [Note] InnoDB: Using atomics to ref count buffer pool pages
10 2018-01-23 17:53:24 3816 [Note] InnoDB: The InnoDB memory heap is disabled
11 2018-01-23 17:53:24 3816 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
12 2018-01-23 17:53:24 3816 [Note] InnoDB: Memory barrier is not used
13 2018-01-23 17:53:24 3816 [Note] InnoDB: Compressed tables use zlib 1.2.3
14 2018-01-23 17:53:24 3816 [Note] InnoDB: Using Linux native AIO
15 2018-01-23 17:53:24 3816 [Note] InnoDB: Using CPU crc32 instructions
16 2018-01-23 17:53:24 3816 [Note] InnoDB: Initializing buffer pool, size = 1.0G
17 2018-01-23 17:53:24 3816 [Note] InnoDB: Completed initialization of buffer pool
18 2018-01-23 17:53:24 3816 [Note] InnoDB: Highest supported file format is Barracuda.
19 2018-01-23 17:53:24 3816 [Note] InnoDB: The log sequence numbers 1484386085656 and 1484386085656 in
20 ibdata files do not match the log sequence number 1511364890742 in the ib_logfiles!
21 2018-01-23 17:53:24 3816 [Note] InnoDB: Database was not shutdown normally!
22 2018-01-23 17:53:24 3816 [Note] InnoDB: Starting crash recovery.
23 2018-01-23 17:53:24 3816 [Note] InnoDB: Reading tablespace information from the .ibd files...
24 2018-01-23 17:53:26 3816 [Note] InnoDB: Restoring possible half-written data pages
25 2018-01-23 17:53:26 3816 [Note] InnoDB: from the doublewrite buffer...
26 2018-01-23 17:53:26 3816 [ERROR] InnoDB: We scanned the log up to 1511364890624. A checkpoint was at
27 1511364890742 and the maximum LSN on a database page was 0. It is possible that the database is now corrupt!
28 2018-01-23 17:53:26 7f451f3e8720 InnoDB: Error: page 5 log sequence number 1511364907354
29 InnoDB: is in the future! Current system log sequence number 1511364890742.
30 InnoDB: Your database may be corrupt or you may have copied the InnoDB
31 InnoDB: tablespace but not the InnoDB log files. See
32 InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
33 InnoDB: for more information.
34 (...)
35 InnoDB: 3 transaction(s) which must be rolled back or cleaned up
36 InnoDB: in total 0 row operations to undo
37 InnoDB: Trx id counter is 1018590976
38 InnoDB: Cleaning up trx with id 1018590873
39 2018-01-23 17:53:27 7f451f3e8720 InnoDB: Assertion failure in thread 139934853662496 in file trx0trx.cc line
 292
InnoDB: Failing assertion: trx->update_undo == NULL
InnoDB: We intentionally generate a memory trap.
40 InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
41 InnoDB: If you get repeated assertion failures or crashes, even
42 InnoDB: immediately after the mysqld startup, there may be
43 InnoDB: corruption in the InnoDB tablespace. Please refer to
44 InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
45 InnoDB: about forcing recovery.
46 16:53:27 UTC - mysqld got signal 6 ;
47 This could be because you hit a bug. It is also possible that this binary
48 or one of the libraries it was linked against is corrupt, improperly built,
49 or misconfigured. This error can also be caused by malfunctioning hardware.
50 We will try our best to scrape up some info that will hopefully help
51 diagnose the problem, but since we have already crashed,
52 something is definitely wrong and this may fail.
53
54 key_buffer_size=33554432
55 read_buffer_size=2097152
56 max_used_connections=0
57 max_threads=151
58 thread_count=0
59 connection_count=0
60 It is possible that mysqld could use up to
61 key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 653196 K bytes of memory
62 Hope that's ok; if not, decrease some variables in the equation.
63
64 Thread pointer: 0x0
65 Attempting backtrace. You can use the following information to find out
66 where mysqld died. If you see no messages after this, something went
67 terribly wrong...
68 stack_bottom = 0 thread_stack 0x40000
69 /usr/sbin/mysqld(my_print_stacktrace+0x35)[0x918445]
70 /usr/sbin/mysqld(handle_fatal_signal+0x494)[0x69bdd4]
71 /lib64/libpthread.so.0(+0xf7e0)[0x7f451efc97e0]
72 /lib64/libc.so.6(gsignal+0x35)[0x7f451da695e5]
73 /lib64/libc.so.6(abort+0x175)[0x7f451da6adc5]
74 /usr/sbin/mysqld[0xa48492]
75 /usr/sbin/mysqld[0xa41758]
76 /usr/sbin/mysqld[0x9b4488]
77 /usr/sbin/mysqld[0xa2f975]
78 /usr/sbin/mysqld[0x973e0d]
79 /usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x5df848]
80 /usr/sbin/mysqld[0x725cb1]
81 /usr/sbin/mysqld(_Z11plugin_initPiPPci+0xbb6)[0x729b16]
82 /usr/sbin/mysqld[0x5d21c8]
83 /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x3e5)[0x5d6de5]
84 /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f451da55d1d]
85 /usr/sbin/mysqld[0x5c83a5]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
180123 17:53:27 mysqld_safe mysqld from pid file /u01/app/oracle/mysql/data/mysqld.pid ended

Tiene mal aspecto, pero aunque la primera opción en la que pensamos es la de

atacar a nivel MySQL, dado que el entorno es muy estático y no ha habido
cambios de configuración en las últimas 24h, optamos por la vía rápida e integrada
en OVM, la herramienta de restauración de MySQL de OVMM. Para ello, basta
seguir la documentación bastante clara y explícita de Oracle que podemos
ver aquí. En nuestro caso, solo queremos recuperar a nivel de BD, por lo que no
necesitamos ejecutar el paso número 1.
OVMM genera diariamente y de manera automática, un backup de la BD MySQL
en la ruta /u01/app/oracle/mysql/backup (OVM 3.4). Vemos el contenido de este
directorio en nuestro caso:

1 [oracle@ovmmanager ~]$ ls -latr /u01/app/oracle/mysql/dbbackup/

2 total 2592
3 drwx------  4 oracle oinstall   4096 Apr  1  2015 3.3.1_preUpgradeBackup-20150401_114910
4 drwx------  4 oracle oinstall   4096 Mar 10  2016 3.3.2_preUpgradeBackup-20160310_145455
5 drwx------  4 oracle oinstall   4096 Mar 28  2016 3.3.3_preUpgradeBackup-20160328_133313
6 drwx------  4 oracle oinstall   4096 Dec 29  2016 3.4.1_preUpgradeBackup-20161229_195435
7 drwx------  4 oracle oinstall   4096 Sep 10 19:24 AutoFullBackup-20170910_192222
8 drwx------  4 oracle oinstall   4096 Sep 11 19:25 AutoFullBackup-20170911_192432
9 drwx------  4 oracle oinstall   4096 Sep 12 19:27 AutoFullBackup-20170912_192620
10 drwx------  4 oracle oinstall   4096 Sep 13 19:28 AutoFullBackup-20170913_192733
11 drwxr-xr-x  5 oracle dba        4096 Jan  5 12:18 ..
12 drwx------  4 oracle oinstall   4096 Jan  6 12:38 AutoFullBackup-20180106_123729
13 drwx------  4 oracle oinstall   4096 Jan  7 12:39 AutoFullBackup-20180107_123822
14 drwx------  4 oracle oinstall   4096 Jan  8 12:40 AutoFullBackup-20180108_123931
15 drwx------  4 oracle oinstall   4096 Jan  9 12:41 AutoFullBackup-20180109_124037
16 drwx------  4 oracle oinstall   4096 Jan 10 12:42 AutoFullBackup-20180110_124144
17 drwx------  4 oracle oinstall   4096 Jan 11 16:52 AutoFullBackup-20180111_165134
18 drwx------  4 oracle oinstall   4096 Jan 12 16:53 AutoFullBackup-20180112_165229
19 drwx------  4 oracle oinstall   4096 Jan 13 16:54 AutoFullBackup-20180113_165327
20 -rw-r-----  1 oracle oinstall 251602 Jan 13 17:21 OVMModelExport-20180113_172134.xml
21 drwx------  4 oracle oinstall   4096 Jan 14 16:55 AutoFullBackup-20180114_165436
22 -rw-r-----  1 oracle oinstall 254527 Jan 14 17:21 OVMModelExport-20180114_172136.xml
23 drwx------  4 oracle oinstall   4096 Jan 15 16:56 AutoFullBackup-20180115_165545
24 -rw-r-----  1 oracle oinstall 257452 Jan 15 17:21 OVMModelExport-20180115_172138.xml
25 drwx------  4 oracle oinstall   4096 Jan 16 16:57 AutoFullBackup-20180116_165655
26 -rw-r-----  1 oracle oinstall 260363 Jan 16 17:21 OVMModelExport-20180116_172141.xml
27 drwx------  4 oracle oinstall   4096 Jan 17 16:59 AutoFullBackup-20180117_165804
28 -rw-r-----  1 oracle oinstall 263280 Jan 17 17:21 OVMModelExport-20180117_172144.xml
29 drwx------  4 oracle oinstall   4096 Jan 18 17:00 AutoFullBackup-20180118_165918
30 -rw-r-----  1 oracle oinstall 241287 Jan 18 17:21 OVMModelExport-20180118_172147.xml
31 drwx------  4 oracle oinstall   4096 Jan 19 17:01 AutoFullBackup-20180119_170019
32 -rw-r-----  1 oracle oinstall 244200 Jan 19 17:21 OVMModelExport-20180119_172149.xml
33 drwx------  4 oracle oinstall   4096 Jan 20 17:02 AutoFullBackup-20180120_170140
34 -rw-r-----  1 oracle oinstall 247125 Jan 20 17:21 OVMModelExport-20180120_172151.xml
35 drwx------  4 oracle oinstall   4096 Jan 21 17:03 AutoFullBackup-20180121_170254
36 -rw-r-----  1 oracle oinstall 250050 Jan 21 17:21 OVMModelExport-20180121_172154.xml
37 drwx------  4 oracle oinstall   4096 Jan 22 17:05 AutoFullBackup-20180122_170402
38 -rw-r-----  1 oracle oinstall 250050 Jan 22 17:21 OVMModelExport-20180122_172156.xml
39 drwxr-xr-x 27 oracle dba        4096 Jan 22 17:21 .

La última copia, de ayer, es AutoFullBackup-20180122_170402. Con esta

información, en primer lugar paramos todo OVMM (Weblogic OVM Server, Oracle
VM Command Line Interface y MySQL database):
 1 [root@ovmmanager ~]# service ovmm stop
2 Stopping Oracle VM Manager                                 [  OK  ]
3
4 [root@ovmmanager ~]# service ovmcli stop
5 Stopping Oracle VM Manager CLI                             [  OK  ]
6
7 [root@ovmmanager ~]# service ovmm_mysql stop
8 Shutting down OVMM MySQL.. SUCCESS!

Ahora, como usuario oracle, lanzamos el shell

script RestoreDatabase.sh especificando el nombre del backup desde el que
queremos hacer la recuperación.

[oracle@ovmmanager ~]$ /u01/app/oracle/ovm-manager-3/ovm_tools/bin/RestoreDatabase.sh AutoFullBackup-

1 20180122_170402
2
3 Before the database can be restored, the following database directories/files must be deleted:
4 appfw ibdata1 ib_logfile0 ib_logfile1 mysql ovs performance_schema
5
6 Are you sure it is safe to delete these directories/files now? [y,n] y
7 Deleting /u01/app/oracle/mysql/data/appfw
8 Deleting /u01/app/oracle/mysql/data/ibdata1
9 Deleting /u01/app/oracle/mysql/data/ib_logfile0
10 Deleting /u01/app/oracle/mysql/data/ib_logfile1
11 Deleting /u01/app/oracle/mysql/data/mysql
12 Deleting /u01/app/oracle/mysql/data/ovs
13 Deleting /u01/app/oracle/mysql/data/performance_schema
14 INFO: Expanding the backup image...
15 INFO: Applying logs to the backup snapshot...
16 INFO: Restoring the backup...
17 INFO: Restoring OVM keystores and certificates
18 INFO: Success - Done!
19 INFO: Log of operations performed is available at: /u01/app/oracle/mysql/dbbackup/AutoFullBackup-
20 20180122_170402/Restore.log
21
22
23 IMPORTANT:
24
25       As 'root', please start the OVM Manager database and application using:
26             service ovmm_mysql start; service ovmm start; service ovmcli start
27
28
[oracle@ovmmanager ~]$
Arrancamos de nuevo los servicios:

1 [root@ovmmanager ~]# service ovmm_mysql start;

2 Starting OVMM MySQL...... SUCCESS!
3
4 [root@ovmmanager ~]# service ovmm start
5 Starting Oracle VM Manager                                 [  OK  ]
6
7 [root@ovmmanager ~]# service ovmcli start
8 Starting Oracle VM Manager CLI                             [  OK  ]

Todo arranca OK, incluída la BD. ¡Hemos recuperado la normalidad!

1 180123 18:03:50 mysqld_safe Starting mysqld daemon with databases from /u01/app/oracle/mysql/data
2 2018-01-23 18:03:50 0 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future
3 release.
4 2018-01-23 18:03:50 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --
5 explicit_defaults_for_timestamp server option (see documentation for more details).
6 2018-01-23 18:03:50 0 [Note] /usr/sbin/mysqld (mysqld 5.6.32-enterprise-commercial-advanced) starting as
7 process 4526 ...
8 2018-01-23 18:03:50 4526 [Note] Plugin 'FEDERATED' is disabled.
9 2018-01-23 18:03:51 4526 [Note] InnoDB: Using atomics to ref count buffer pool pages
10 2018-01-23 18:03:51 4526 [Note] InnoDB: The InnoDB memory heap is disabled
11 2018-01-23 18:03:51 4526 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
12 2018-01-23 18:03:51 4526 [Note] InnoDB: Memory barrier is not used
13 2018-01-23 18:03:51 4526 [Note] InnoDB: Compressed tables use zlib 1.2.3
14 2018-01-23 18:03:51 4526 [Note] InnoDB: Using Linux native AIO
15 2018-01-23 18:03:51 4526 [Note] InnoDB: Using CPU crc32 instructions
16 2018-01-23 18:03:51 4526 [Note] InnoDB: Initializing buffer pool, size = 1.0G
17 2018-01-23 18:03:51 4526 [Note] InnoDB: Completed initialization of buffer pool
18 2018-01-23 18:03:51 4526 [Note] InnoDB: Highest supported file format is Barracuda.
19 2018-01-23 18:03:51 4526 [Note] InnoDB: The log file was created by mysqlbackup --apply-log at ibbackup
20 180123 17:59:17. The following crash recovery is part of a normal restore.
21 2018-01-23 18:03:51 4526 [Note] InnoDB: The log sequence numbers 1484386085656 and 1484386085656 in
22 ibdata files do not match the log sequence number 1511045530636 in the ib_logfiles!
23 2018-01-23 18:03:51 4526 [Note] InnoDB: Database was not shutdown normally!
24 2018-01-23 18:03:51 4526 [Note] InnoDB: Starting crash recovery.
25 2018-01-23 18:03:51 4526 [Note] InnoDB: Reading tablespace information from the .ibd files...
26 2018-01-23 18:03:54 4526 [Note] InnoDB: Restoring possible half-written data pages
27 2018-01-23 18:03:54 4526 [Note] InnoDB: from the doublewrite buffer...
28 2018-01-23 18:03:55 4526 [Note] InnoDB: 128 rollback segment(s) are active.
29 2018-01-23 18:03:55 4526 [Note] InnoDB: Waiting for purge to start
30 2018-01-23 18:03:55 4526 [Note] InnoDB: 5.6.32 started; log sequence number 1511045530636
31 2018-01-23 18:03:55 4526 [Note] RSA private key file not found: /u01/app/oracle/mysql/data//private_key.pem.
32 Some authentication plugins will not work.
 2018-01-23 18:03:55 4526 [Note] RSA public key file not found: /u01/app/oracle/mysql/data//public_key.pem.
Some authentication plugins will not work.
2018-01-23 18:03:55 4526 [Note] Server hostname (bind-address): '*'; port: 49500
2018-01-23 18:03:55 4526 [Note] IPv6 is available.
33
2018-01-23 18:03:55 4526 [Note]   - '::' resolves to '::';
34
2018-01-23 18:03:55 4526 [Note] Server socket created on IP: '::'.
2018-01-23 18:03:55 4526 [Note] Event Scheduler: Loaded 0 events
2018-01-23 18:03:55 4526 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.32-enterprise-commercial-advanced'  socket: '/u01/app/oracle/mysql/data/mysqld.sock'  port: