[Jenne, Jason, Daniel]
Since Daniel has seen so much coherence with REFL port RIN (alog 70611), we thought we'd try increasing the ISS gain in case it's due to our being gain limited at high frequencies.
We didn't have much phase margin at all in the second loop (Measured last week in alog 70684), but we did in the first loop (last measured Feb 2022, alog 61826).
So, we increased the first loop by 6 dB (first attachment), now UGF is 46kHz with 39 deg phase margin. The gain slider is now at 11 dB. We also had to adjust the output offset from 5 to 4.3. We can turn on / off the first loop at input power of 2W just fine.
We increased the second loop by 4 dB (second attachment), now UGF measured at the -20dB line is 37 kHz with 32 deg phase margin. We had tried increasing the second loop by a further 2 dB (third attachment), but we decided that 41 kHz UGF wasn't worth having only 23 deg phase margin. These measurements were both taken with the input power at our current nominal 60W, but PRM and both ITMs misaligned. For the current lock (as well as for all these measurements) we just changed the gain after the ISS second loop was closed and DC coupled.
Still to do: At low power / DOWN in IMC_LOCK, the ISS second loop gain slider should be -2 dB (that's where all the offsets were most recently tuned). Then, in CLOSE_ISS, once the ISS is closed, the gain slider should be increased to +2 dB.
Also to do - make sure we know / remember how to get the SR785 connected to the wifi so we can download the data to the control room workstations.
The IMC_LOCK guardian's DOWN state already looks at and uses the lscparams.ISS_acquisition_gain setting of -2.
I have added some logic to the OPEN_ISS state of IMC_LOCK to also reset the second loop gain slider to -2 (to prepare for re-closing it).
I have modified the logic in CLOSE_ISS to increase the second loop gain. The final gain value for H1:PSL-ISS_SECONDLOOP_GAIN when we get to Observing should be +2.
Since we're in Observing, I have not loaded the IMC_LOCK guardian. I'll leave a sticky not for the operator to do that when we're next out of Observe.
If, when we get to Observing, the H1:PSL-ISS_SECONDLOOP_GAIN is not at +2, it is okay (for tonight, until I debug / fix why guardian didn't do it right) to change the slider in units of 1 (which is the default slider click value) until it is +2.
If there are any troubles with this this evening, please give me a call.
TITLE: 06/27 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 142Mpc
SHIFT SUMMARY: Maintenance day. Recovery was straight forward and hands off, aside from some automation testing (DRMI/PRMI less waiting, even shorter power up times). We've been in and out of observing breifly for a SQZ issue and then a timing port difference. Locked for 2+ hours
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 14:19 | COMM | Craig | Remote | N | Moddepth up-down test | 14:21 |
| 14:26 | CAL | Camilla | CR | N | BB Calibration Measurement | 14:30 |
| 14:31 | PEM | Camilla | CR | N | Measurementts | 14:53 |
| 14:32 | FAC | Bubba | LVEA | N | Plugging in | 14:48 |
| 14:56 | FAC | Karen, Kim | LVEA | N | Technical Cleaning | 16:24 |
| 15:16 | Property | Christina, Nicole | LVEA | n | Property hunt | 17:37 |
| 15:17 | FAC | Tyler | LVEA | n | Grab a part | 15:27 |
| 15:27 | FAC/VAC | Tyler, Bubba | MX | - | Retrieve Hepta pump | 17:14 |
| 15:38 | CDS/SEI | Fil | EX | n | Cabling for HEPI controller | 17:43 |
| 16:21 | SEI | Jim | EX | n | Check on power cord status | 18:21 |
| 16:22 | - | Jeff, Elenna, Amber | LVEA | n | LVEA tours | 17:57 |
| 16:24 | FAC | Karen, Kim | FCES | n | Tech clean | 17:24 |
| 16:33 | VAC | Janos | LVEA | n | Looking for parts | 16:53 |
| 16:34 | ISC | Daniel | CR | n | OMC single bounce meas | 19:34 |
| 16:45 | CDS | Marc, Fernando | CER | n | ITMY baffle pd chassis inspection | 17:56 |
| 16:48 | VAC | Travis | EX, EY | n | Install fittings on Hepta | 18:53 |
| 16:53 | VAC | Janos | Outbuildings | n | Inventory and labels | 18:53 |
| 17:04 | Contractor | Pepsi | OSB | n | Removing vending machine | 19:04 |
| 17:15 | FAC/CDS | Patrick | remote | n | FMCS channel changes | 18:15 |
| 17:32 | VAC | Norco | CS | n | LN2 fill | 19:32 |
| 17:44 | CDS | Fil | EY | n | Clean up cables, temperature sensor | 18:29 |
| 18:25 | SAF | Betsy | LVEA | YES | Laser hazard transition | 18:32 |
| 18:25 | SQZ | Daniel | LVEA | YES | SQZT0 pictures | 18:42 |
| 18:26 | ISC | Jenne, Jason | CR | n | ISS measurement, higher input power | 18:32 |
| 18:32 | CDS | Fil, Sheila, Betsy | LVEA | YES | OMC camera swap | 19:02 |
| 18:33 | VAC | Gerardo, Jordan | LVEA, EX | - | Grab wrench, then go to EX | 19:18 |
| 19:38 | ISC | Jenne, Jason | LVEA | n | ISS measurement at rack | 19:58 |
TITLE: 06/27 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 140Mpc
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 23mph Gusts, 20mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.06 μm/s
QUICK SUMMARY:
Inheriting H1 while it's locked for 2 hours in NOMINAL_LOW_NOISE & OBSERVING.
WP 11139
The temporary setup for synchronizing the Beckhoff system with the NTP server was removed. Part of WP 11139. The following terminals were powered down and removed from MSR Rack 8.
1. EK1101 coupler
2. EL3004
3. EL6688
The EK1101 coupler was connected to the spare Beckhoff computer. The EL6688 terminal was connected to the NTP server. Spare computer was powered off.
There was a timing configuration change that dropped us out of observing briefly. Dave, Erik, Fil figured out this came from removing a spare Beckhoff unit.
Time out of Observing: 2237-2244 UTC
Sheila, Naoki
As shown in the attached figure, SQZ ASC ran away and was turned off since the SQZ ASC trigger was below the threshold. We removed squeezing and reset the AS42 offset and push the graceful clear history. After we brought back the squeezing, everything seems working now.
Even with new input matrix, SQZ ASC ran away today. I reset the AS42 offset and pushed the graceful clear history. The SQZ ASC came back, but I am not sure why this happened.
Naoki, Vicky - We added checkers to both SQZ_MANAGER and SQZ_FC guardians, to give notifications if ASC is not on.
SQZ_MANAGER now checks for ASC on if SQZ-ASC_ANG_P/Y_INMON = 0; same for filter cavity asc, which checks if SQZ-FC_ASC_ANG_P/Y_INMON = 0. Earlier today, looks like the ASC_WFS switch was ON but the alignment ran away, so the signal fell below trigger threshold, and AS42 ASC did not engage. Hope these notifications make it easier to catch when ASC isn't on.
These guardian edits for SQZ_MANAGER and SQZ_FC were the only SVN diffs; the changes are now committed to SVN version 25945.
T. Guidry During today's testing of the OSB fire suppression system all air handlers that serve the OSB were taken down. This is a normal and necessary function of our fire alarm system during an alarm. All AHU's are back online and functioning normally. The duration of the interruption was 37 minutes (10:08-10:45 local time)
Here are a couple plots of oplevs and OSEM signals during our last lock, focusing on the change related to turning on the OM2 TSAMS. I plotted test mass top mass OSEMs and oplevs (in case they told different stories), and the OSEMs for SRM, SR2, OM3 and OMC.
Conclusion: there doesn't appear to be an alignment change in the arms. We hypothesized that a change in the signal to WFS 45 could change the DHARD and MICH alignment loops. That doesn't appear to be the case. However, we do see a pretty clear change in the OM3 and OMC alignment. There might be some OMC ASC alignment offsets we could try in order to mimic the changes we are seeing from the OM2 change.
From squeezer ASC trends as OM2 heats up, we also see an alignment shift, most strongly related to AS_A_RF45_Pitch (corresponds to SQZ-ASC_ANG_P_OUTPUT). Not implausible that the output alignment is what's changing; this would also be consistent with having to reset SQZ ASC offsets 70890 with a hot OM2.
At 76W, Camilla and I tried walking OMC ASC QPD offsets and didn't find anything better (69307). Before at 60W, Koji had success increasing the optical gain by walking the OMC ASC QPD offsets, but that was in mid-March before great TCS tuning, 67994. I thought this helped to improve and stabilize the range then. He also light heated OM2 after the new OMC ASC offsets 68048 and saw a similar ~2% reduction in optical gain, consistent with now 70859.
So, I also think it makes sense to try walking the OMC alignment, and generally the IFO's output alignment.
Fast Channels Removed:
> H1:LSC-POPAIR_B_RF18_I_ERR_16k_DQ 4 16384
> H1:LSC-POPAIR_B_RF90_I_ERR_16k_DQ 4 16384
Slow Channels Removed:
> H1:VID-CAM15_AUTO 4 16
> H1:VID-CAM15_XY 4 16
Implemented my code to speed up lockloss down time in the ACQUIRE_DRMI_1F and ACQUIRE_PRMI checks with a successful run!
Changes made to my code since previous post (70765):
In ACQUIRE_DRMI_1F, switched to check POPAIR_B_RF18_I_ERR_DQ data instead of POPAIR_B_RF90_I_ERR_DQ and lowered max threshold to 130 from 200. Changes made due to better correlation between RF18 and DRMI.
Code is easy to update if better thresholds are found.
WP11274 Replace HAM6 camera, h1cam15
Fernando, Fil, Betsy, Patrick, Erik, Dave:
h1cam15 (HAM6) camera was replaced (see Fil's alog for details). Patrick install a new server for this camera on h1digivideo3 (was running on h1digivideo1). Erik added the new camera to the DHCP server, I added the new camera to H1EPICS_DIGVIDEO.ini. DAQ restart was required.
WP11276 Add FCES channels to FMCS EPICS
Patrick, Dave:
Patrick added the FCES FMCS channels to his EPICS IOC (see Patrick's alog for details). I added the new channels to H0EPICS_FMCS.ini. DAQ restart was required.
WP11285 CW hardware injection code, use gpstime, discontinue using tconvert
Joe B, Keith R, Mike Thomas, Erik, Dave:
Following LLO, we upgraded the psinject (CW hardware injection) code on h1hwinj1 to no longer use tconvert to get GPS time, and use gpstime instead.
For gpstime to run, the version of python3 needed to be updated from 3.4 to 3.6. Mike did this and reinstalled gpstime.
I applied Joe's changes by "svn up" to get the latest code and verifying the changes.
When we tried to run the new code it would not start. We tracked this down to an install issue, lalapps were removed by the upgrade. Mike re-installed lalapps and the new code is now running.
The first run only ran for about a minute, exiting with an awg-stream error. On the second restart awg-stream connected and the exitation is stable.
WP11278 h1lsc add filtermodules, remove two 16kHz DQ channels
Dave:
I installed the latest h1lsc model (two additional filter modules) and changed the DAQ_Channels block to no longer send the H1:LSC-POPAIR_B_RF[18,90]_I_ERR_16k_DQ channels to the frame. DAQ restart was required.
DAQ Restart
Dave:
The DAQ was restarted twice, first around 10:45 in conjuction with h1lsc model and EDC restarts to apply LSC changes and add the FCES FMCS channels.
A second restart, along with a EDC restart, around 12:46 to add the new h1cam15 channels.
Tue27Jun2023
LOC TIME HOSTNAME MODEL/REBOOT
10:42:56 h1lsc0 h1lsc <<< add 2 FM, remove 2 DQ channels
10:43:54 h1daqdc0 [DAQ] <<< 0-leg
10:44:07 h1daqfw0 [DAQ]
10:44:07 h1daqnds0 [DAQ]
10:44:07 h1daqtw0 [DAQ]
10:44:15 h1daqgds0 [DAQ]
10:44:50 h1susauxb123 h1edc[DAQ] <<< EDC add FCES FMCS channels
10:45:36 h1daqgds0 [DAQ] <<< 2nd gds0 restart
10:48:03 h1daqdc1 [DAQ] <<< 1-leg
10:48:12 h1daqfw1 [DAQ]
10:48:12 h1daqtw1 [DAQ]
10:48:13 h1daqnds1 [DAQ]
10:48:21 h1daqgds1 [DAQ]
10:49:06 h1daqgds1 [DAQ] <<< 2nd gds1 restart
12:46:33 h1daqdc0 [DAQ] <<< 0-leg
12:46:45 h1daqfw0 [DAQ]
12:46:45 h1daqtw0 [DAQ]
12:46:46 h1daqnds0 [DAQ]
12:46:54 h1daqgds0 [DAQ]
12:47:20 h1susauxb123 h1edc[DAQ] <<< Add h1cam15 new chans
12:48:00 h1daqgds0 [DAQ] <<< 2nd gds0 restart
12:50:47 h1daqdc1 [DAQ] <<< 1-leg restart
12:50:57 h1daqfw1 [DAQ]
12:50:57 h1daqtw1 [DAQ]
12:50:59 h1daqnds1 [DAQ]
12:51:07 h1daqgds1 [DAQ]
12:51:41 h1daqgds1 [DAQ] <<< 2nd gds1 restart
The camera has been removed from the IOC and the server has been stopped and removed from monit on h1digivideo1. The camera has been added to puppet control on h1digivideo3. The H1VID_DIGITAL_OVERVIEW.adl medm screen has been updated. There is a video image, but the IFO is not locked, so it is black.
There is now a beam image.
Closes WP 11276. List of added channels: H0:FMC-FCES_AHU1_COLD_COIL_TEMP_DEGF H0:FMC-FCES_AHU1_FAN_SPEED H0:FMC-FCES_AHU1_HEPA_PRESS H0:FMC-FCES_AHU1_MIXED_TEMP_DEGF H0:FMC-FCES_AHU1_PRESS H0:FMC-FCES_AHU1_RELIEF_FAN_AIRFLOW_CFM H0:FMC-FCES_AHU1_RETURN_TEMP_DEGF H0:FMC-FCES_AHU1_SUP_FAN_AIRFLOW_CFM H0:FMC-FCES_AHU1_SUP_TEMP_DEGF H0:FMC-FCES_AUX_OSA_TEMP_DEGF H0:FMC-FCES_AUX_OUTSIDE_HUMIDITY H0:FMC-FCES_AUX_SPACE_HUMIDITY H0:FMC-FCES_AUX_SPACE_PRESS H0:FMC-FCES_AUX_SPACE_TEMP_DEGF H0:FMC-FCES_AUX_SUP_TEMP_DEGF H0:FMC-FCES_H2O_TANK_LEVEL H0:FMC-FCES_VEA_OSA_TEMP_DEGF H0:FMC-FCES_VEA_OUTSIDE_HUMIDITY H0:FMC-FCES_VEA_SPACE_HUMIDITY H0:FMC-FCES_VEA_SPACE_PRESS H0:FMC-FCES_VEA_SPACE_TEMP_DEGF H0:FMC-FCES_VEA_SUP_TEMP_DEGF H0:FMC-FCES_AHU1_RELIEF_FAN_STATUS H0:FMC-FCES_AHU1_SUP_FAN_STATUS H0:FMC-FCES_AUX_MAIN_FAN_STATUS H0:FMC-FCES_H2O_TANK_LOW_LEVEL_ALARM H0:FMC-FCES_VEA_MAIN_FAN_STATUS H0:FMC-FCES_AHU1_COLD_COIL_TEMP_DEGC H0:FMC-FCES_AHU1_MIXED_TEMP_DEGC H0:FMC-FCES_AHU1_RETURN_TEMP_DEGC H0:FMC-FCES_AHU1_SUP_TEMP_DEGC H0:FMC-FCES_AUX_OSA_TEMP_DEGC H0:FMC-FCES_AUX_SPACE_TEMP_DEGC H0:FMC-FCES_AUX_SUP_TEMP_DEGC H0:FMC-FCES_VEA_OSA_TEMP_DEGC H0:FMC-FCES_VEA_SPACE_TEMP_DEGC H0:FMC-FCES_VEA_SUP_TEMP_DEGC
Dave restored the alarm fields using the snapshot file from /ligo/cds/lho/h0/burt/2022/01/31/00:00/h0fmcs.snap. He wrote a script to parse out the values and caput them.
Per WP11286 in response to ALOG70818
We installed the spare S1302233 Baffle PD Amplifier directly below the installed Baffle PD Amplifiers into SUS R5 in the Biergarten. ITMX is S1400066 and ITMY was S1400065 but we swapped functionality to the spare we just mounted below it (this rack is mostly empty).
If there are issues with the spare we can immediately swap functality back to the old unit. If the spare fixes the issues, we will recover the old unit and perform a post mortem. As there was no laser light during install we could not verify functionality while on the floor.
M. Pirello, F. Mera
We seem to see some signal on this pd now, unlike before. Interestingly, the pd seems a fair amount of light during the ISC_LOCK state of Move_Spots. Clearly we are moving through a place with high scatter, best to avoid.
I pulled the ITMY Baffle PD Amplifer S1400065 and moved the replacement S1302232 up to occupy positions 11 & 12 to the right of the ITMX Baffle PD Amplifier. Post mortem on S1400065 showed that one of the internal cables was loose, this has been fixed and all cables pull tested. S1400065 has been included in the spares inventory.
This completes WP11286
Since we are back to a lower and more stable operating power, I reduced the power step up time in the MAX_POWER guardian state. Each 5W step up from 25W took 60 seconds, now that time is down to 45 seconds. Tomorrow, I'd like to try a 30 second step time, which I think should be fine since the ASC is much more stable up to 60W. This has already reduced the relocking time by almost 2 minutes, and will reduce it more.
Today I reduced this timer to 30 seconds. We just had a successful power up! This is now guardianized. Should shave 3.5 minutes off the locking process.
Note: the ASC loops are notoriously unreliable. I encourage operators to pay attention to the power up process and if there are problems with ASC instabilities during the final power up, increasing this timer will likely be a good solution. Tagging ops.
Relock was fully auto after one lock loss while finding IR. There was a missing comma that brough ISC_LOCK into error in LOWNOISE_LENGTH_CONTROL, easy fix.
There were a few SDF diffs that look like they need to be accepted based on alog70648. Accepted with screenshots attached.
I turned on the CAL_AWG_LINES Guardian at request of Jeff. I had to change this node's nominal state to LINES_ON for it to be OK.
I'm thinking that this lower range is related to the squeezer. I've attached a screenshot fo the FDS DARM FOM where the live trace is above the refernece in the same frequencies that DARM seems to be higher than normal. I followed the instructions on the Troubleshooting SQZ wiki to adjust the sqeeze angle, but I wasn't able to make anything better, only worse.
I adjusted the sqz angle from 0630-0640 UTC.
Our range is staying low at ~125Mpc, there seems to be extra noise from 20-60Hz. Investigating.
There's a lot more coherence of PRCL and SRCL in the ongoing lock than the previous one. The thermalization cal lines are also very high in DARM and CHARD - maybe they weren't turned on until this new lock though.
Tagging CAL regarding the turn on of CAL_AWG_LINES for this 60W lock stretch -- thanks TJ!
Tagging DetChar as well -- to note that we have 8 extra calibrations on during this nominal low noise stretch that started at June 22 2023 05:04 UTC
These are in because we want to characterize the thermalization of the detector's DARM loop sensing and response functions now that we're operating at 60W rather than 75/76W. I hope to get a few more of these lock acquisitions with these extra lines on, and then we'll turn them off as we had done for the start of the engineering run.
If you'd like to create a data quality flag, you can find the status of these lines "in one go" by looking at the CAL_AWG_LINES guardian state channel,
H1:GRD-CAL_AWG_LINES_STATE_N
The numerical value of the channel is 10.0 when the extra calibration lines are ON (the state is called LINES_ON), and 2.0 when the lines are OFF (the state is called IDLE). See CAL_AWG_LINES_StateGraph for the flow of the state graph.
A retrospect on the lower range: The solution was a lack of capturing the SRCLFF1 gain in SDF / Guardian during yesterday's power reduction from 75W to 60W LHO:70648. See LHO:70712 and LHO:70710 where Tony recovered the correct gain of 2.1. @DetChar -- it might be worth creating a data quality flag for this: Observation segment start (with SRCLFF1 Gain at 1.0): 2023-06-22 05:04:38 UTC 22:04:38 PDT 1371445496 GPS Observation segment stop (with SRCLFF1 Gain at 1.0): 2023-06-22 13:53:47 UTC 06:53:47 PDT 1371477245 GPS
RyanC just added the SRCLFF1 gain of 2.1 to lscparams, saved, and reloaded the ISC_LOCK guardian. So, if we need to relock it'll come back on with the correct gain. Note though, that we expect to update this yet again later today.
Here's that SDF screenshot that I said I was going to attach. Turns out my tired brain had flipped setpoint and epics value in the tables, Doh! My fault.
Adding a quick comment to Jeff's note about lines, with a bit of relevant info from ER15.
Abby Wang and Athena Baches recently analyzed lines in May 2023 data, grouping those that evolve similarly in time. They found a cluster of lines corresponding to the awg lines, but not including any other entries. This is good news; it implies that there are *not* strong narrow artifacts with very similar histories-- i.e. these lines aren't causing unexpected strong lines elsewhere, which ought to have shown up in the same cluster. (Note: it's still possible that there are weak artifacts which aren't caught by this analysis.)
The attached plots show what the time evolution looks: each row is a line (corresponding to 11.475, 11.575, 15.175, 15.275, 24.4, and 24.5 Hz) , yellow = above threshold and blue = below threshold.