TITLE: 06/25 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC STATE of H1: Preventive Maintenance INCOMING OPERATOR: Corey SHIFT SUMMARY: Full day of maintenance due to vacuum work at mid Y (WP 8250). Maintenance continues at close of shift. LOG: 15:09 UTC Robert and students to LVEA to set up vibrometer and pull viewport covers (WP 8253) 15:19 UTC Vanessa to end X to clean 15:21 UTC Weed control company through gate 15:22 UTC Fire department through gate for hydrant testing 15:26 UTC Gerardo to mid Y (WP 8250) 15:33 UTC Karen to end Y to clean 15:35 UTC Marc to end stations to swap baluns. Starting at end X. (WP 8255) 15:36 UTC Jeff B. done forklifting (WP 8257) 15:44 UTC Jeff B. to end stations and corner station mechanical room to check HEPI pumps and dust monitor vacuum 15:47 UTC Filiberto to LVEA to swap laser power supply fans (WP 8249) 15:48 UTC Lock loss 15:55 UTC Hugh taking corner station HEPI platforms to OFFLINE (WP 8256) 15:56 UTC Jason adjusting PMC alignment from control room 15:57 UTC Rahul starting charge measurements at end Y 16:00 UTC Hugh checking/charging corner station HEPI fluid accumulators (WP 8256) 16:10 UTC Paradise water through gate Peter transitioning LVEA to laser safe (WP 8260) 16:15 UTC Seal coat solutions through gate 16:22 UTC Jason done adjusting PMC alignment 16:28 UTC Norco LN2 delivery through gate 16:31 UTC Unifirst through gate Jeff K. leading tour to LVEA (WP 8251) 16:36 UTC Peter done transitioning LVEA to laser safe 16:37 UTC Karen leaving end Y 16:39 UTC Jeff B. done 16:45 UTC Rahul starting charge measurements at end X 16:47 UTC Marc done swapping baluns at end stations 16:56 UTC Dick G. to CER 17:08 UTC Ray through gate 17:08 UTC Dave to end X 17:11 UTC Peter to optics lab 17:17 UTC Hugh bringing corner station HEPI pump back online 17:19 UTC Karen to LVEA 17:21 UTC Tour done 17:22 UTC Dave starting work at end X (WP 8247) 17:22 UTC Chris to LVEA to start checks on forklifts, etc. 17:24 UTC Marc and Ray to vault (WP 8248) 17:29 UTC end Y charge measurement done 17:45 UTC Rick and company to end X, PCAL 17:56 UTC Filiberto done swapping fan on ALS power supply (WP 8249) 18:02 UTC end X transitioned to laser hazard 18:07 UTC Dave turning off DAQ switch at end X (WP 8247) 18:37 UTC Chris out of LVEA 18:41 UTC Cheryl transitioning LVEA to laser hazard 18:41 UTC Hugh taking end Y HEPI platforms to OFFLINE (WP 8256) Hugh to end Y 18:55 UTC Jeff B. opening OSB receiving door 18:56 UTC Cheryl done transitioning LVEA to laser hazard 19:03 UTC Rick and company leaving end X to get equipment and break for lunch 19:04 UTC Dave back from end X 19:07 UTC Marc and Ray back from vault. Ray left the site. 19:19 UTC Filiberto and Peter transitioning end Y to laser hazard 19:22 UTC Hugh, Filiberto and Peter leaving end Y 19:35 UTC Dave to end X, adding admin cable to new network switch 19:36 UTC Hugh back from end Y, bringing HEPI back online 19:55 UTC TJ restarting ISC_LOCK guardian 19:57 UTC Filiberto done at end Y 20:06 UTC Chris bringing new dishwasher to OSB with forklift 20:12 UTC Christina driving down arms to get receipts from Norco 20:15 UTC Rick and company back to end X 20:41 UTC TJ taking equipment to LVEA 20:43 UTC Karen to end Y to drop off supplies 20:44 UTC Dave back to end X 20:47 UTC TJ back 20:59 UTC Jeff B. moving equipment into cleaning area 21:00 UTC Gerardo to LVEA to make measurement of HAM5 door 21:10 UTC Peter to optics lab 21:38 UTC Dave done working on switch 21:47 UTC Chris to LVEA to unplug equipment 21:51 UTC Hugh to LVEA to take measurements of HAM chambers for A+ work 22:02 UTC Chris done 22:04 UTC Hugh done 22:18 UTC Marc and Filiberto to LVEA to look at racks 22:21 UTC Jeff B. done 22:36 UTC Cheryl done running transfer functions on IMs 22:55 UTC Marc to LVEA to swap balun (WP 8261)
We are testing a new version of the TwinCAT EPICS Ioc in the corner.
To revert: remote desktop into h1ecatc1. Close the IOC (running in a poweshell window). Start a new one using the "Restart EPICS database" button in the "LIGO TwinCAT Target Configuration" app.
To restart: Make sure the old IOC is gone. Go into C:\SlowControls\Target\H1ECATC1, right click on "start_tcioc_2_0.ps1" and select "Run with PowerShell". ("start_tcioc.ps1" is for the old code)
Version 2.0
Added features:
Bug Fixes:
TITLE: 06/25 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: Patrick
CURRENT ENVIRONMENT:
Wind: 11mph Gusts, 7mph 5min avg
Primary useism: 0.08 μm/s
Secondary useism: 0.09 μm/s
QUICK SUMMARY:
Y-arm is still blocked (MY GV10 & GV11 are CLOSED), so Long Maintenance Day continues.
Michael Ross attempted to upgrade the BRS code at EndX (WP 8258) but it would hang up after a few tens of seconds. Michael was unable to trouble shoot and there seemed to be trouble with the Twim Cat and a file it was looking for. There was also network slowdowns or sothing and even difficulties running the old code. After rebooting the old code worked again and has been now for a couple hours. Something inherent weird happening though as the BRSY hung up as well and we never logged into that machine. Same behavior as in even the old code would hang a few 10 of seconds after restarting. Once the computer was restarted, EndY has now run fine for a couple hours.
Krishna & Michael will be back in a couple weeks and we'll maybe have the time for more thorough TS.
WP 8249
The power supply for the ISCT6 laser was uninstalled to replace failing fans inside the unit. Status of the unit before uninstalling:
1. Unit not screwed to its mounting brackets.
2. Unit sitting on some rubber pads for isolation.
3. Laser Diode temp (set) 20.22
4. Laser Diode curr (actual) 1.856 5. Laser Crystal (both set & actual) 31.83
Both fans inside the unit were replaced. Unit was reinstalled as found without the mounting brackets attached and left on the rubber pads. Here are the setting after install. I was careful not to move the knobs for the temp/current setting:
1. Laser Diode temp (set) 20.23
2. Laser Diode curr (actual) 1.872
3. Laser Crystal (both set & actual) 31.83
WP 8252 Change to system manager checked into svn and reloaded. Screenshot of initial svn update prior to work attached.
Today's OPLEV charge measurement did not go through successfully since the ISI tripped at both the ends (roughly after taking 45 mins of data). I am not sure about who was working at the end station, although there were a few stuff going on. I will look for another window today since the detector will down for 8 hours of maintenance.
This seems to have been caused by balun replacement for CPS at the end stations this morning.
https://services.ligo-la.caltech.edu/LHO/workpermits/view.php?permit_id=8255
Bimonthly check of the dust monitor vacuum pumps, revealed no problems or issues. Made a slight tweak to the vacuum pressure in the corner station. Both end station pressures were normal, and no excessive temperatures were found.
Check the HEPI fluid levels at both end stations (Hugh had already checked the CS). End-Y was unchanged at 9 0/16. End-X was down by 1/16 to 8 8/16. Did not observe any apparent leaks. If fluid level drops again on the next check, will inspect the system for leaks.
The LVEA has transitioned to LASER SAFE.
This is related to work permit #8260.
The PMC reflected power has been creeping up the last couple of weeks, from 9.8W to 10.4W, so this morning I tweaked the beam alignment into the PMC (remote from the control room). This was done with the ISS OFF. Before I started, the PMC was transmitting 53.8W and reflecting 10.4W. Once complete the PMC was transmittng 54.2W and reflecting 10.0W. With the extra power out of the PMC the ISS RefSignal needed to be adjusted to maintain the diffracted power % between 2% and 3%; the ISS RefSignal is now -2.06 V and the diffraction % is ~2.4%; this has been accepted in SDF. With the ISS back ON, the PMC is transmitting 54.5 W and reflecting 10.0 W.
Both PSL power watchdogs were reset at 16:18 UTC (9:18 PDT). This completes FAMIS 10716.
Edit to add: I accepted the ISS RefSignal change for both the SAFE and OBSERVE SDF files. When transitioning back to the SAFE file, SDF was showing yet another high-precision 10-16 difference for the ISS RefSignal (it does this almost everytime I switch between SDF files). I accepted this for the SAFE file, but did not go back to the OBSERVE file to check. So note to the on-shift operator repsonsible for recovering after today's extended maintenance: when going to OBSERVE for the first time you may see a 10-16 SDF difference for the ISS RefSignal; accepting this difference is OK.
Tue Jun 25 13:26:36 PDT 2019
For channel(s):
H0:FMC-CS_FIRE_PUMP_1
H0:FMC-CS_FIRE_PUMP_2
TITLE: 06/25 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Patrick
SHIFT SUMMARY:
LOG:
Last Lock: 25:43:33
10:05 Begin IA
10:30 Begin locking
12:01 H1 back to Observing
14:46 PEM Weekly script running - Intention bit Commissioning
14:50 Chris and Jeff B moving lift trucks
14:51 Chandra and Kyle heading out to MY - gate valves
14:57 Sensor correction turned off and H1 in Preventive maintenance mode
6.3 & 6.6Mag quakes from Japan reported incoming. Attempted to switch to Large Earthquake mode. This most likely broke the lock. However P & S waves already spikiing BLRMS
I was looking at riding out the smaller aforementioned quake when a verbal came in with a 5.8 from somewhere I wasn't familiar with. As a precautionary measure I made the switch. Jim's new SC output lights said it was ok. ;-)
5.8 Heard Island and McDonald Islands quake was a dud. After further investigation, It seems that this was the 6:01UTC quake listed on USGS as "Southeast Indian Ridge". I think something may be wrong with Verbal Alarms as this quake was about an hour old already and when I scrolled the Verbal terminal it disappeared (weird). Going to restart Verbal. Going to keep H1 in EQ mode as BLRMS are still dancing around a little in the .1-.2um/sec range which is ~ an order of magnitude higher than recently nominal.
7:49 Restarted Verbal Alarms
8:06 Switched SEI_CONF back to WINDY
Verbal logs the seismon information anytime that it announces an EQ. This second announcement at 0722 UTC of the 5.8 in Heart Island and McDonald Islands was definitely a repeat of the same earthquake notifcation at 0622 UTC. Trending the seismon EQ_TIME_GPS channels, you can see that seismon removed this earthquake for a few minutes, then brought it back. Verbal sees this as a new EQ so it will announce it if the arrival time is still positive and in a band on the seiplot.
While Verbal didn't do anything wrong, I will try to figure a way around this. Trending these channels revealed to me that this happens more often than I orginally thought.
Operators: In the meantime, please always confirm the earthquakes when you get a notification (similar to what we do with GRBs).
See screenshot for error midway up the log. All I did was reload INIT_ALIGN guardian and it seemed to move on fine. Maybe needs more investigation.
This is a case of too many managers trying to both use ALIGN_IFO. I have commented out the unstall_nodes decorator in ISC_LOCK's inital_alignment state. The INIT_ALIGN node is already unstalling what it needs to, and INIT_ALIGN shouldn't need to be unstalled.
Sorry about that. I added that unstall nodes because we had seen the IMC guardian become stalled during Init align. Does Align IFO manage the IMC and unstall it when it looses lock?
Ah, no it doesn't. I'll change ISC_LOCK to check on that, and I'll triple check the other nodes.
Screen shot of the new channels. The CB_QUEUE channels are monitoring the usage of the ring buffers which are used by EPICS to process reads (_0) and writes (_2).