aLIGO LHO Logbook

H1 CDS

patrick.thomas@LIGO.ORG - posted 21:31, Wednesday 04 February 2015 - last comment - 08:56, Thursday 05 February 2015(16477)

h1conlog3 reporting disk errors

This is the conlog replica.

It appears to have started around 5:00 PM and repeated each hour after. I had to restart the database replication, it appears that the database has been crashing in response. It may not be a coincidence that the search for frequently changing channels also runs every hour, so I have disabled it.

cdsadmin@h1conlog3:/var/log$ grep error syslog
Feb  4 17:04:08 h1conlog3 kernel: [2335398.323116]          res 51/40:0a:16:e8:88/00:00:00:00:00/0b Emask 0x9 (media error)
Feb  4 17:04:08 h1conlog3 kernel: [2335398.323407] ata3.00: error: { UNC }
Feb  4 17:04:08 h1conlog3 kernel: [2335398.344358] Add. Sense: Unrecovered read error - auto reallocate failed
Feb  4 17:04:08 h1conlog3 kernel: [2335398.344367] end_request: I/O error, dev sda, sector 193521686
Feb  4 17:04:11 h1conlog3 kernel: [2335401.397976]          res 51/40:02:16:e8:88/00:00:00:00:00/0b Emask 0x9 (media error)
Feb  4 17:04:11 h1conlog3 kernel: [2335401.398264] ata3.00: error: { UNC }
Feb  4 17:04:11 h1conlog3 kernel: [2335401.420344] Add. Sense: Unrecovered read error - auto reallocate failed
Feb  4 17:04:11 h1conlog3 kernel: [2335401.420352] end_request: I/O error, dev sda, sector 193521686
...

Comments related to this report

patrick.thomas@LIGO.ORG - 21:36, Wednesday 04 February 2015 (16478)

Link

I attempted a simple search and it crashed with the same syslog error. I am going to try to disable the webpage. Please do not try to use conlog until this is resolved.

patrick.thomas@LIGO.ORG - 21:42, Wednesday 04 February 2015 (16479)

Link

To clarify, the acquisition of data appears to be ok, but the machine with the copy of the data that is used for user searches is not.

patrick.thomas@LIGO.ORG - 08:56, Thursday 05 February 2015 (16484)

Link

From the MySQL error log:

InnoDB: Error: tried to read 16384 bytes at offset 7 692060160.
InnoDB: Was only able to read 8192.
150204 17:04:11  InnoDB: Operating system error number 0 in a file operation.
InnoDB: Error number 0 means 'Success'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html
InnoDB: File operation call: 'read'.
InnoDB: Cannot continue operation.
150204 17:04:13 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150204 17:04:13 [Note] Plugin 'FEDERATED' is disabled.
150204 17:04:13 InnoDB: The InnoDB memory heap is disabled
150204 17:04:13 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150204 17:04:13 InnoDB: Compressed tables use zlib 1.2.8
150204 17:04:13 InnoDB: Using Linux native AIO
150204 17:04:13 InnoDB: Initializing buffer pool, size = 1.0G
150204 17:04:13 InnoDB: Completed initialization of buffer pool
150204 17:04:13 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 47921973351
150204 17:04:13  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 47922305837
150204 17:04:13  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 
88 89 90 91 92 93 94 95 96 97 98 99 
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 4307, file name /var/log/mysql/mysql-bin.000221
150204 17:04:14  InnoDB: Waiting for the background threads to start
150204 17:04:15 InnoDB: 5.5.38 started; log sequence number 47922305837
150204 17:04:15 [Note] Recovering after a crash using /var/log/mysql/mysql-bin
150204 17:04:15 [Note] Starting crash recovery...
150204 17:04:15 [Note] Crash recovery finished.
150204 17:04:16 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
150204 17:04:16 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
150204 17:04:16 [Note] Server socket created on IP: '0.0.0.0'.
150204 17:04:17 [Note] Event Scheduler: Loaded 0 events
150204 17:04:17 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.38-0ubuntu0.14.04.1-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)