Closes FAMIS 19656
HAMs
HAM7_CPSINF_V2s seems to have gotten less noisy a high frequency
HAM5_CPSINF_V3 looks a little noisier at high frequency
BSCs
ITMY_ST2_CPSINF_V3 is higher
Everything else looks just as it did during the last check
FAMIS 19961
pH of PSL chiller water was measured to be between 10.0 and 10.5 according to the color of the test strip.
I also checked the water lines in the chiller room; all full of water with no accumulated air or bubbles to be seen.
On a few occasions EX_ISC has had 3 SDF diffs pop up and take us out of Observing, they only last for a few seconds so I haven't been able to catch all of them yet but I did catch this one channel this morning and unmonitored it, I also unmonitored its companion in EY_ISC.
FAMIS 19950
PMC Trans continues to fall, but Jason and I plan to take a look at this tomorrow in the enclosure. RefCav transmission is also dropping.
Closes 23804
Seems like there was a spike/glitch ~ 3 days ago for both mid vibrometers and a couple spikes for MR_FAN5_170_2 but spikes are well within threshold (and have since dropped to 0).
PSL Status FAMIS Report for this week:
Laser Status:
NPRO output power is 1.829W (nominal ~2W)
AMP1 output power is 68.54W (nominal ~70W)
AMP2 output power is 142.4W (nominal ~140W)
NPRO watchdog is GREEN
AMP1 watchdog is GREEN
AMP2 watchdog is GREEN
PMC:
It has been locked 10 days, 23 hr 1 minutes
Reflected power = 20.59W
Transmitted power = 104.1W
PowerSum = 124.7W
FSS:
It has been locked for 0 days 9 hr and 23 min
TPD[V] = 0.7962V
ISS:
The diffracted power is around 2.2%
Last saturation event was 0 days 9 hours and 24 minutes ago
Possible Issues:
PMC reflected power is high
TITLE: 05/01 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 134Mpc
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 9mph Gusts, 6mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.28 μm/s
QUICK SUMMARY:
I can confirm that running the top plot on opslogin0 indeed shows the CS seismometer line dropped out about 18 hours, but running the lower plot the CS signal is correct. Could be a GDS monitor issue?
diaggui was updated to 3.1.5~dev12 in a further attempt to stabilize the program when reading GDS channels.
The DARM plot on nuc30 was restarted and is once again displaying H1:GDS-CALIB_STRAIN.
Following the CS front end crash and recovery effort by team CDS, I have recovered all suspensions and remaining seismic systems. I noticed the alignment on the AS AIR camera looked poor, so I decided to restore all optics to their alignments at 14:17 UTC on 4/29 (the last time we locked ALS). As I'm not available to babysit H1 this evening, I've told it to relock itself and will leave it be. This should be a useful test of the automation as the TRX/Y signals are floating around 0.2; not amazing flashes to start with.
I glanced at the control room screenshots around 6:45, and saw that the IFO was stuck because it could not find the COMM IR.
I helped it out, but after looking at the ALS Camera error signals (and the poor DRMI flashes) we need an initial alignment, which I will start now.
Initial alignment mostly worked by itself. The BS was enough out of alignment (which, actually could have been due to me poking the sliders when I was trying to lock before aligning) that it couldn't lock MICH_BRIGHT. It was doing a nice job of going to DOWN in ALIGN_IFO though. While it was running the DOWN of ALIGN_IFO, I poked the BS sliders to get better flashes on the AS_air camera, and then it caught MICH and kept going.
After that, initial alignment went on all by itself and completed successfully.
The flashes look better. I'm going to let the automation take it from here (we just started to try DRMI), and may check back in a while. Others are free to take a look though, if it looks like it needs more help.
Once in the last few hours we made it to Engage_ASC_for_full_IFO, after I poked the SRM and SR2 to convince DRMI to lock (after a few PRMI aligns). But, the buildups looked very ratty and poor.
Just now, I noticed a notification that SEI_CS was not at its nominal configuration. The notification had a helpful note to re-request the state, so I clicked Init on SEI_CS, and it seems to have re-executed whatever it needed, to get back to Config_windy. The notification is now gone.
I wonder if, when the computers came back up, the guardian didn't know that it should do something to reset the seismic config - if that's what happened, we should think about what we can do to improve this in the future, as we work toward more automation.
I'll check back in on the IFO if I wake up in the middle of the night, but I'm hopeful that things will be more smooth now.
SDFs accepted, with some names in bold for folks to have a look and double check me:
* SUSETMY, 1st attachment. Same as accepted by AustinC in alog 69176. I'm unsure why they needed accepting, since it looks like RyanC and Austin had both recently accepted them. But, it's probably becasue this is one of the models that previously was linked between the safe and observe, and now is not. Hopefully this won't come up agian.
* SUSPROC, 2nd attachemnt. These are filters for violin mode monitoring (not damping), so I've reverted them to match how they were on Friday (alog 69129). Rahul, please have a look, to see which way they should actually be, and check their safe.snap so that they come up after reboot with the filters that are correct.
* CALINJ, 3rd attachment. It looks like there was a calibration switch that was off, but had been on for the last 2 weeks. Before turning it back on (since it's the kind of thing that will send real siganl to the IFO), I went to NLN_CAL_MEAS so that any ramping would be taken care of nicely. I then reverted the SDF, and requested Nom_Low_Noise. After that, I have an error in the CAL_AWG_LINES guardian (4th attachment). I hit the Load button, and it was happy. I have not looked at the data, but this may mean that I reverted and turned that switch on while some calibration lines were still going. Seems to not have caused an issue with the lock though. Jeff, please have a look at the SDF, as well as this new guardian.
*CALEY. 5th attachment. This didn't have a diff when I started, but I think it got a diff when I cycled through the CALINJ system, as noted above. Reverted to the lower value, but Jeff, please have a look to see why this came back to the wrong value.
* SEIPROC, 6th attachment. The HAM1 FF switch being off was probably me, looking at things with Elenna on Friday. It is normally on, I believe, but Elenna, please have a look and make sure all the settings are as you want them to be.
After having done these, I put the IFO into Observing.
We have lost several corner stations front end models in what may be a Dolphin glitch. Caused the lock loss at 16:12PDT.
h1omc0 was synchronized at the wrong time, see the attached GSD TP screen. Duotone value is negative. T0 shows timing was lagging behind by 30 ms.
Output from timing card looked normal.
Full set of IOP GDS-TP screens
Jonathan, Erik, Dave, TJ:
We did not see anything useful in the front end logs.
Erik restarted the models on h1omc0, h1susb123, h1sush2a, h1sush34, h1sush56 and h1oaf0.
Note, we did not need to fence front ends from Dolphin and reboot, just model restarts were sufficient.
I cleared all the SWWD watchdogs, no SEI WDs had tripped.
We called TJ who will continue the recovery of the interferometer.
Sun30Apr2023
LOC TIME HOSTNAME MODEL/REBOOT
16:59:56 h1omc0 h1iopomc0
17:00:10 h1omc0 h1omc
17:00:24 h1omc0 h1omcpi
17:02:29 h1susb123 h1iopsusb123
17:02:34 h1sush2a h1iopsush2a
17:02:43 h1susb123 h1susitmy
17:02:48 h1sush2a h1susmc1
17:02:57 h1susb123 h1susbs
17:03:02 h1sush2a h1susmc3
17:03:11 h1susb123 h1susitmx
17:03:11 h1sush34 h1iopsush34
17:03:16 h1sush2a h1susprm
17:03:25 h1susb123 h1susitmpi
17:03:25 h1sush34 h1susmc2
17:03:30 h1sush2a h1suspr3
17:03:35 h1sush56 h1iopsush56
17:03:39 h1sush34 h1suspr2
17:03:53 h1sush34 h1sussr2
17:03:55 h1sush56 h1sussrm
17:04:01 h1oaf0 h1iopoaf0
17:04:09 h1sush56 h1sussr3
17:04:15 h1oaf0 h1pemcs
17:04:23 h1sush56 h1susifoout
17:04:29 h1oaf0 h1tcscs
17:04:37 h1sush56 h1sussqzout
17:04:43 h1oaf0 h1susprocpi
17:04:57 h1oaf0 h1seiproc
17:05:11 h1oaf0 h1oaf
17:05:25 h1oaf0 h1calcs
17:05:39 h1oaf0 h1susproc
17:05:54 h1oaf0 h1calinj
17:06:08 h1oaf0 h1bos
Following up on LHO aLOG 69061, we modified a local checkout of the pydarm_H1.ini file to be consistent with front end FOTON filter bank values for the inverse sensing gain, exported the inverse sensing filter that has a new cavity pole frequency, and the actuation gain values (N/A or N/V^2) so that the pyDARM model output would give the correct answer. Note that pyDARM assumes that gain values given in the ini file match what is installed in the FOTON filter banks. If not, then inconsistencies can arise. In addition we verified that the CALCS actuation output matrix (all are -1 for the LHO X-arm) and the pyDARM values are consistent. Here, the pydarm_H1.ini file had +1 for all values, but it had been giving values that didn't make sense when installed. We changed them back to -1 in the ini file and got more consistent results. We've recently had some discussions about the GDS filters giving incorrect answers because of minus sign problems (see G2300832; the fact that pyDARM gives a much better answer using values that match the front end, this might mean (maybe) the problem lies not within the model but within how the GDS filters are generated or applied. This needs additional attention. Below are the updated EPICS record values for the PCAL DELTAL corrections: 'CAL-CS_TDEP_PCAL_LINE1_DELTAL_PCAL_CORR_REAL': 0.0033852702240058613, 'CAL-CS_TDEP_PCAL_LINE1_DELTAL_PCAL_CORR_IMAG': 0.0003688309923906657, 'CAL-CS_TDEP_PCAL_LINE2_DELTAL_PCAL_CORR_REAL': 5.918160347836006e-06, 'CAL-CS_TDEP_PCAL_LINE2_DELTAL_PCAL_CORR_IMAG': -3.8606119335684666e-07, 'CAL-CS_TDEP_PCAL_LINE3_DELTAL_PCAL_CORR_REAL': 8.389172516042244e-07, 'CAL-CS_TDEP_PCAL_LINE3_DELTAL_PCAL_CORR_IMAG': -1.5076809464241352e-07, 'CAL-CS_TDEP_PCAL_LINE4_DELTAL_PCAL_CORR_REAL': 0.00165323521544998, 'CAL-CS_TDEP_PCAL_LINE4_DELTAL_PCAL_CORR_IMAG': 0.00011505749801666585, 'CAL-CS_TDEP_PCAL_X_COMPARE_DELTAL_PCAL_CORR_REAL': -5.9211839406814295e-06, 'CAL-CS_TDEP_PCAL_X_COMPARE_DELTAL_PCAL_CORR_IMAG': 3.8609892061668445e-07, 'CAL-CS_TDEP_PCAL_Y_COMPARE_DELTAL_PCAL_CORR_REAL': 5.918160347836006e-06, 'CAL-CS_TDEP_PCAL_Y_COMPARE_DELTAL_PCAL_CORR_IMAG': -3.8606119335684666e-07, 'CAL-CS_TDEP_PCAL_LINE5_DELTAL_PCAL_CORR_REAL': -0.0008859219262577612, 'CAL-CS_TDEP_PCAL_LINE5_DELTAL_PCAL_CORR_IMAG': -3.9867889605789786e-05, 'CAL-CS_TDEP_PCAL_LINE6_DELTAL_PCAL_CORR_REAL': -0.00034094924519537996, 'CAL-CS_TDEP_PCAL_LINE6_DELTAL_PCAL_CORR_IMAG': -8.284228371577319e-06, 'CAL-CS_TDEP_PCAL_LINE7_DELTAL_PCAL_CORR_REAL': -0.00016249959919441566, 'CAL-CS_TDEP_PCAL_LINE7_DELTAL_PCAL_CORR_IMAG': -3.251411986550448e-06, 'CAL-CS_TDEP_PCAL_LINE8_DELTAL_PCAL_CORR_REAL': -9.46064626745444e-05, 'CAL-CS_TDEP_PCAL_LINE8_DELTAL_PCAL_CORR_IMAG': -1.5341758740447534e-06, 'CAL-CS_TDEP_PCAL_LINE9_DELTAL_PCAL_CORR_REAL': -1.2512538215094095e-05, 'CAL-CS_TDEP_PCAL_LINE9_DELTAL_PCAL_CORR_IMAG': 5.893565987708546e-07, 'CAL-CS_TDEP_PCAL_LINE10_DELTAL_PCAL_CORR_REAL': -5.9211839406814295e-06, 'CAL-CS_TDEP_PCAL_LINE10_DELTAL_PCAL_CORR_IMAG': 3.8609892061668445e-07, These were installed and accepted in the SDF
To generate the records, we used the pyDARM configuration file pydarm_H1.ini (f1939173bac56a7fff7bde578558bd9da9b71ff4) and the pyDARM method pydarm.calcs.CALCSModel.compute_epics_records(): >>> model = pydarm.calcs.CALCSModel('pydarm_H1.ini') >>> model.compute_epics_records(gds_pcal_endstation=False, gds_sus_endstation=False, gds_darm_omc=False)
