11:26UTC I'm still increasing power manually. PI mode 26 seems pretty buried at this point.
11:27 OMC DCPD Saturation - lockloss; SUS OMC and HAM6 ISI WD tripped.
A few other things tonight:
A little bit of ASC work:
Cleared ST1 and ST2 BS accumulators.
Thanks Ed. Couple things. Not sure where St1 and St2 come in to play for HEPI, that sounds like ISI. Plus, with the upgrade done to HEPI yesterday, no seismic system should need the Watchdog saturations clearing anymore. I'll remove the FAMIS task associated here for HEPI.
Possibly after some changes made durring maintence, the lockloss tool stopped working. I can use lockloss plot, but not select.
sheila.dwyer@opsws4:~/Desktop/Locklosses$ lockloss -c channels_to_look_at_TR_CARM.txt select
After noticing LLO alog 28710 I tried to svn up the lockloss script (we're at rev 14462 now), but I still get the same error when I try to use select
The problem at LLO is totally different and unrelated.
The exception you're seeing is from an NDS access failure. The daq NDS1 server is saying that it can't find the data that is being requested. This can happen if you request data too recent in the past. It also used to happen because of gaps in the NDS1 lookback buffer, but those should have been mostly fixed.
Right now the lockloss tool is looking for all lock loss events from 36 hours ago to 1 second ago. The 1 second ago part could be the problem, if that's too soon for the NDS1 server to handle. But my testing has indicated that 1 second in the past should be sufficient for the data to be available. In other words I've not been able to recreate the problem.
In any event, I changed the look-back to go up to only two seconds in the past. Hopefully that fixes the issue.
Trouble locking DRMI improved after turning TCSY back on. Handing over to Ed and Sheila. 23:15 UTC Cheryl started initial alignment. Taking over. 23:35 UTC Initial alignment done. 00:13 UTC Changed phase to damp PI mode 26 00:15 UTC NLN, RF looks good 00:17 UTC Sheila running A2L, Kiwamu to LVEA to turn on TCSY laser 00:23 UTC Kiwamu back 00:45 UTC Lock loss. TCSY laser turned off. Kiwamu to LVEA to turn it back on. 01:01 UTC Kiwamu back 01:26 UTC NLN 01:46 UTC Jeff K. starting calibration measurements 02:24 UTC Jeff K. done measurements 03:40 UTC Sheila to LVEA to check ISC racks 03:43 UTC Sheila back 03:45 UTC Sheila starting frequency noise coupling measurement 06:08 UTC Lock loss (Sheila) 07:17 UTC Sheila and Ed to mezzanine to check TCS chiller. TCSY laser has been off since around the beginning of the last lock. 07:28 UTC Sheila and Ed back. TCSY laser back on.
Ed, Sheila and Patrick
The "Gate" indicator LED was off on the TCSY LASER controller. This condition was reset by Sheila by pushing the red button on the front panel of the controller. I added 250ml of water to the Y-chiller to bring the level back to the top mark. This was most likely the reason that PRMI wouldn't stay locked.
J. Kissel, D. Tuyenbayev,
We turned on two calibration lines using ETMY coil drivers on stages L1 and L2. The SNRs of these lines are roughtly 1/3 of the regular calibration lines (regular cal. lines have SNR of ~100 with 10s FFT).
_FREQ (Hz) _CLKGAIN (ct)
H1:SUS-ETMY_L1_CAL_LINE 33.7 60.0 O2-scheme synched oscillator for kappa_U
H1:SUS-ETMY_L2_CAL_LINE 34.7 27.0 O2-scheme synched oscillator for kappa_P
These lines will be used to better quantify calibration of the PUM and UIM actuators.
A related report: LHO alog 29291.
κU, κP and κT using these additional lines at ~6am on Wed., Oct. 12 are:
κU = 0.9828 - 0.0637i
κP = 0.9499 - 0.0380i
κT = 0.9784 - 0.0440i
These are preliminary values calculated manually for transfer functions taken at a selected time (no trends or averaging except for 10avg. in DTT), and the DARM model for ER9 (H1params_2016-07-01.conf). We will use SLM tool data to look at the parameter trends.
Greg M, Darkhan T,
We calculated kappa values using SLM data (10s FFTs) generated over 10 days between Oct 10 and Oct 21 with ER9 parameter file
'${CalSVN}/Runs/PreER9/H1/params/H1params_2016-07-01.conf'
The plots show raw (unaveraged) values. SNRs of the L1 and L2 lines (used for calculation of κU and κP respectively) were set to give approximately 1/3 SNR compared to the L3 line (with the ER9 noise floor).
The data was taken from an additional SLM tool instance which was setup by Greg to calculate 33.7 Hz, 34.7 Hz, 35.9 Hz and 36.7 Hz line FFTs.
J. Kissel for S. Karki I've changed the > 1 kHz PCALX calibration line frequency, a.k.a. the "long duration sweep" line, to 2501.3 Hz. Recall this started moving again on Oct 6th (see LHO aLOG 30269) I report the progress towards completing the sweep below. Frequency Planned Amplitude Planned Duration Actual Amplitude Start Time Stop Time Achieved Duration (Hz) (ct) (hh:mm) (ct) (UTC) (UTC) (hh:mm) --------------------------------------------------------------------------------------------------------------------------------------------------------- 1001.3 35k 02:00 1501.3 35k 02:00 2001.3 35k 02:00 2501.3 35k 05:00 39322.0 Oct 12 2016 03:20:41 UTC 3001.3 35k 05:00 39322.0 Oct 06 2016 18:39:26 UTC Oct 12 2016 03:20:41 UTC days 3501.3 35k 05:00 39322.0 Jul 06 2016 18:56:13 UTC Oct 06 2016 18:39:26 UTC months 4001.3 40k 10:00 4301.3 40k 10:00 4501.3 40k 10:00 4801.3 40k 10:00 5001.3 40k 10:00
J. Kissel, D. Tuyenbayev We've grabbed another set of calibration measurements of the IFO's sensing function in order to build confidence in our newly commissioned front-end tracking of time dependent model parameters, and to confirm the discrepancy seen yesterday (LHO aLOG 30391) was not something funny about that lock stretch. Indeed the discrepancy remains. We've found bugs in our code that compares our DARM model against these measurements, so we can't yet confirm exact numbers like optical gain, DARM coupled cavity pole frequency, or SRC detuning spring frequency to offer a proposed change to the CAL-CS front-end parameters to fix it. We'll hopefully have something by tomorrow. Attached are screenshots of the measurements and the DARM loop parameters during the measurement. Timeline: (in UTC) 2016-10-12 01:30:00 -- IFO in stable Nominal Low Noise (PCALY calibration lines are ON); IFO undisturbed 01:46:00 -- IFO sufficiently thermalized to start measurements; PCALY lines turned OFF 01:47:48 -- PCAL2DARM measurement begins 02:05:48 -- DARM OLGTF measurement begins 02:24:00 -- measurements done; PCALY lines turned back ON; IFO undisturbed 02:40:00 -- Official Undisturbed period ends (but no one has done anything with the IFO since, up to 03:06 UTC) Data files (which are exported and committed to the SVN): /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER10/H1/Measurements/ DARMOLGTFs/2016-10-12_H1_DARM_OLGTF_4to1200Hz.xml PCAL/2016-10-12_H1_PCAL2DARMTF_4to1200Hz.xml
J. Kissel, D. Tuyenbayev,
These are kappas calculated in the front-end after the TF measurements have been completed.
The optical gain is still ~15% lower w.r.t. the value in the DARM model parameters and the coupled-cavity pole frequency is ~ 340-350 Hz, which is higher compared to the yesterday's value (LHO alog 30391).
There's still more noise in the calculated values of fC. The SNR of the 331.9 Hz line is affected by a slightly increased noise in the [0.2, 1.0] kHz band.
Doing the same thing as was done in LHO aLOG 30475, I've grabbed similar undisturbed data from the times quoted in the original entry, and replotted them in matlab after doing some rudimentary math. The 15 minute (900 sec) average value (starting at Oct 12 2016 02:39:00 UTC) for all of the time dependent parameters just after the sweeps are: Param Units Mean Std kappa_{C} [ ] 0.868 pm 0.01 f_{cc} [Hz] 325 pm 5.5 Re: kappa_{PU} [ ] 1.01 pm 0.0089 Im: kappa_{PU} [ ] 0.0007 pm 0.015 Re: kappa_{TST} [ ] 0.998 pm 0.011 Im: kappa_{TST} [ ] -0.010 pm 0.0091 Input Power [W] 51.8 pm 0.029 The script to generate these values lives in /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER10/H1/Scripts/CAL_EPICS/get_fe_tdep_params.m
J. Kissel I've used Kiwamu's templates (LHO aLOG 29860) to gather new DARM OLG and PCAL2DARM transfer functions, to help increase the number of data points for darm coupled cavity pole and SRC detuning spring frequencies. In doing so, I've found that PCAL suggests that CAL-DELTAL_EXTERNAL is wrong low by ~20% above 50 Hz, in a frequency dependent fashion (confirmed by the phase and ridiculously high coherence). We will analyze the data in-full, and report what the measured pole and spring frequencies are in the morning. We'll also show trends of calibration line heights, such that eventually we can confirm the frequencies over time. This frequency dependent error in CAL-DELTAL_EXTERNAL may be because the DARM coupled cavity pole frequency programmed into the CAL CS fron end is 342 Hz, and that's just not correct for this lock stretch. We have to turn OFF the PCALY calibration lines during the measurement due to range issues, but the front end calculation of the DARM cavity pole, after the measurement, suggests anywhere for 315 Hz to 340 Hz. Unclear whether we can trust the online calculation just yet, so we're leaving the IFO alone for ~30 minutes after the measurement to be able to confirm offline later. Recall that we're still not controlling the cavity waist angle/translation of the SRC entirely: the SRC1 loop (nominally controlling SRM alone) is OFF, but the SRC2 loop (controlling SRM and SR2) is ON. Undisturbed time post-sweeps: Start Oct 11 2016 02:36:00 UTC Oct 10 2016 19:36:00 PDT 1160188577 Stop Oct 11 2016 03:07:00 UTC Oct 10 2016 20:07:00 PDT 1160190437 The new measurements have not yet been exported, but have been committed to the CalSVN here: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER10/H1/Measurements/ DARMOLGTFs/2016-10-10_H1_DARM_OLGTF_4to1200Hz.xml PCAL/2016-10-10_H1_PCAL2DARMTF_4to1200Hz.xml EDIT: I've added two screenshots showing the digital settings relevant to the DARM loop parameters. Unfortunately, the production conlog is down for maintenance/upgrades, but thankfully Patrick setup a test bed a few nights ago (goto http://conlog-test-replica/), I was able to pull the list of channels I needed. FAlso for the record, some of those channels have changed since O1, so I quote the new list here for easier copy and paste in the future: H1:LSC-DARM1_SWSTAT H1:LSC-DARM1_GAIN H1:LSC-DARM2_SWSTAT H1:LSC-DARM2_GAIN H1:SUS-ETMY_L1_LOCK_L_SWSTAT H1:SUS-ETMY_L1_LOCK_L_GAIN H1:SUS-ETMY_L1_DRIVEALIGN_L2L_SWSTAT H1:SUS-ETMY_L1_DRIVEALIGN_L2L_GAIN H1:SUS-ETMY_L2_LOCK_L_SWSTAT H1:SUS-ETMY_L2_LOCK_L_GAIN H1:SUS-ETMY_L2_DRIVEALIGN_L2L_SWSTAT H1:SUS-ETMY_L2_DRIVEALIGN_L2L_GAIN H1:SUS-ETMY_L3_ISCINF_L_SWSTAT H1:SUS-ETMY_L3_ISCINF_L_GAIN H1:SUS-ETMY_L3_LOCK_L_SWSTAT H1:SUS-ETMY_L3_LOCK_L_GAIN H1:SUS-ETMY_L3_DRIVEALIGN_L2L_SWSTAT H1:SUS-ETMY_L3_DRIVEALIGN_L2L_GAIN H1:SUS-ETMY_L3_LOCK_INBIAS H1:SUS-ETMY_L3_ESDOUTF_LIN_BYPASS_SW H1:SUS-ETMY_L3_ESDOUTF_LIN_FORCE_COEFF H1:SUS-ETMY_BIO_M0_STATEREQ H1:SUS-ETMY_BIO_L1_STATEREQ H1:SUS-ETMY_BIO_L2_UL_STATEREQ H1:SUS-ETMY_BIO_L2_LL_STATEREQ H1:SUS-ETMY_BIO_L2_UR_STATEREQ H1:SUS-ETMY_BIO_L2_LR_STATEREQ H1:SUS-ETMY_BIO_L3_UL_STATEREQ H1:SUS-ETMY_BIO_L3_UR_STATEREQ H1:SUS-ETMY_BIO_L3_LL_STATEREQ H1:SUS-ETMY_BIO_L3_LR_STATEREQ
J. Kissel, D. Tuyenbayev,
Kappas calculated in the front-end suggest that an optical gain was lower compared to the reference time (optical gain in the DARM model) by ~15% and the coupled-cavity pole frequency was ~325 Hz at the TF measurement time.
Also it seems that the SNR of 331.9 Hz line is not sufficient - higher statistical noise in the calculated κC and fC.
A little bit more quantitative assessment if the time dependent correction factors as computed by the front end (or "kappas"): I've grabbed similar undisturbed data from the times quoted in the original entry, and replotted them in matlab after doing some rudimentary math. The 15 minute (900 sec) average value (starting at Oct 11 2016 02:51:00 UTC) for all of the time dependent parameters just after the sweeps are: Param Units Mean Std kappa_{C} [ ] 0.866 pm 0.011 f_{cc} [Hz] 323 pm 4.5 Re: kappa_{PU} [ ] 1.01 pm 0.011 Im: kappa_{PU} [ ] -0.007 pm 0.012 Re: kappa_{TST} [ ] 0.998 pm 0.0085 Im: kappa_{TST} [ ] 0.0095 pm 0.0065 Input Power [W] 50.5 pm 0.031 The script to generate these values lives in /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER10/H1/Scripts/CAL_EPICS/get_fe_tdep_params.m And for ease of copy and paste for someone in the future, I'm gathering the following channels (which correspond to the parameters in the same order as the table above) H1:CAL-CS_TDEP_KAPPA_C_OUTPUT H1:CAL-CS_TDEP_F_C_OUTPUT H1:CAL-CS_TDEP_KAPPA_PU_REAL_OUTPUT H1:CAL-CS_TDEP_KAPPA_PU_IMAG_OUTPUT H1:CAL-CS_TDEP_KAPPA_TST_REAL_OUTPUT H1:CAL-CS_TDEP_KAPPA_TST_IMAG_OUTPUT H1:IMC-PWR_IN_OUTPUT
WP6232 Upgrade frame writers
Jonathan, Jim B, Dave:
h1fw2 was running with close to the new code, it was upgraded to add the md5 filename change only. Quickly following h1fw0 and h1fw1 were upgraded to the latest code. I am updating the DAQ MEDM to show the new diagnostic md5 check sum data.
WP6217 ISI frontend code upgrade
Hugh, Jim B
All ISI models were restarted with new code
WP6227 HEPI frontend code upgrade
Hugh, Jim B
All HEPI models were restarted with new code.
WP6235 PSL-DBB jitter feed-forward to LSC
Daniel, Gabrielle, Dave:
New code on the h1psldbb exports the jitter signal as a 64kHz Dolphin channel. The h1lsc model uses this feed-forward data. Change added a 16kHz LSC DQ channel to the commissioning frame.
WP6242 Migration of virtual machines to proxmox
Carlos
Old vmware virtual hosts were migrated to the new Proxmox system.
WP6236 SUS ITMX.ETMX model changes for HWWD install
Fil, Richard, Jim B, Dave
sister wp to 6234. The ITMX and ETMX hardware watchdogs were install. The h1susitmx, h1susetmx models were modified to read the HWWD status via the binary input feed from these chassis. Since we are not controlling these units, I did not put in the binary output logic.
WP6229 h1tcscs model change for SIM parts
Kiwamu, Dave
A new h1tcscs model was installed. We verified that the TCS CO2 chillers behaved correctly on loss of temperature control voltage.
WP6231 GDS-DMT upgrade
team-GDS
Code was upgraded, no impact was seen from the control room
WP6239 PSL-ISS model change
Daniel, Dave
New h1psliss model was installed. This was done at the same time as the h1psldbb change.
WP6240 Beckhoff Slow Controls change
Daniel, Dave
Daniel installed new PLC code. I updated the DAQ INI files, the target autoBurt.req files and the Beckhoff SDF monitor files.
WP6238 CAL-CS model change
Darkhan, Jeff K, Dave
a new h1calcs model was installed.
DAQ restarts
Dave:
The DAQ was restarted several times today to support the above work.
(note we did not get to removing h1ldasgw2 WP#6237, will defer this to later in the week)
Regarding the following:
WP6217 ISI frontend code upgrade
Hugh, Jim B
All ISI models were restarted with new code
Only the HAM ISIs models were updated. I might have implied to DaveB that I was going to do something to the BSC ISIs but in the end I do no changes there.
LP_v_H4 shows the laser output power and the head 4 flow rate. At this point in time the flow rate trip 0.2 lpm. The drop in laser power seems to coincide with the increase in flow rate. XC_v_H4 shows the output of the crystal chiller and the head 4 flow rate. The chiller clearly switches off after the spike in the head 4 flow rate. InletP_v_H4 shows the inlet pressure versus head 4 flow rate. The oscillations in pressure take place after the spike in the flow rate. The same is true for the outlet pressure. The sudden increase in the flow rate perhaps suggests that something in the flow sensor may have been dislodged but I (PK) would expect that the flow rate would not drop to zero but to some other non-zero value. Unless there was a following object that promptly jammed things up. But things started up okay for the subsequent startup. The various filters in the system are clean from a visual inspection. Jason/Peter
The trip Peter discusses above took place at around 6:17am PDT on Sunday, 10/9/2016.
The laser did trip off on Saturday (first reported by Patrick here) at 21:06:02 PDT (04:06:02 UTC, Sunday 10/9/2016). As noted by Kiwamu the reason for the trip was due to a trip of the Laser Head 1-4 Flow interlock.
PSL_Heads_Flow_2016-10-8.png shows the 4 laser head flow sensors around the time of the trip; head 4 is once again the most likely culprit. As seen in the plots posted above by Peter for the early Sunday morning trip, head 4 sees a spike in flow rate before falling off to zero. Interestingly, when compared to the actual interlock (PSL_Head4_v_HeadIL_2016-10-8.png) the interlock trips just prior to the spike in flow rate; from the plot it can be seen that the interlock starts to trip just before the spike in head 4 flow begins. The trip threshold at this time was set to 0.2 lpm, which none of the flow sensors were below at the time the interlock tripped; I can't explain why the interlock tripped.
PSL_Head4_v_XChil_2016-10-8.png shows the crystal chiller turning off several seconds after the interlock trips (5 seconds after in this case). I've noted this before (here).
PSL_Head4_v_InletP_2016-10-8.png shows the head 4 flow rate versus the inlet pressure in the PSL water manifold. The oscillation in the pressure occurs after the interlock tripped and has been seen before (noted above by Peter and here).
details in alog 30410
Update as of 23:05UTC (16:05PT)
State of H1: in Initial Alignment
Activities:
Peter, Kiwamu,
We have taken out the PD A photodiode out of the ISS inner loop box and placed it at a location where the PWR_EOM photodiode was. We confirmed that we can close the loop with the moved PD A.
[New setup]
We opened up the inner loop box and removed the base plate on which the two photodiodes sat. Then we took out only PD A from the base plate while leaving PD B mounted on the base plate. We put the base plate back into the inner loop box. A black glass beam dump was placed at the location where PD A used to be. Careful: because we could not find a good screw hole, we could not clamp down the black glass beam dump which is currently just sitting on the base plate.
We placed PD A to the location where PWR EOM (Thoralbs PDA55) had been. We rerouted all the associated cables for PD A (a 5 pin lemo, 2 normal lemos) to the new location. We measured the beam power at this location to be 8 mW. Because this is too much for PD A (which we think typically runs with 1 mW), we installed an ND filter on the front panel of PD A (see the picture).
A few more pictures are available at ResourceSpace.
[ISS loop closed with PD A]
Here shows a screenshot of the setup for closing the inner loop with PD A. The gain slider needed to be a 12 dB higher (which still smells like a factor of two calibration error...) than the usual. The ref signal for PD A needed to be -0.46.
Here are some plots of the open loop transfer function with PD A used as the in-loop sensor. A gain of more than 20-30 dB in the gain slider actually does not do the job -- needs more investigation. The ugf is about 46 kHz with the gain slider set to 30 dB. Note that when PD B was in use, the ugf had been 54 kHz (29942).
ECR E1600042 WP 6217; ECR E1500325 WP 6227
To Remove DACKILL elements from HAM ISIs and HEPIs:
HAM ISI
.) Removed DACKILL Widgets on WD Block--Attachment 1.
.) Removed DACKILL link from WD Block to the ERRMON Block--Attachment 2.
.) Routed WD_STATE from WD Block to ERRMON--Attachment 2.
.) Added WD_STATE Logic in ERRMON Block for Rogue Excitation Evaluation--Attachment 3.
HEPI
.) Removed DACKILL elements from HEPI (Top) Level--Attachment 4.
.) Removed DACKILL parts on WD Block--like ISI Attachment 1.
To update HEPIs to Bleedoff-Accumulated-Saturations feature running on ISIs for months:
.) Add WINDOWTIME inputs on HEPI WD Block--Attachment 5.
.) Removed Test Point pickoffs WD Flag channels--also on Attachment 5.
.) Replace model saturation counter on WD/IPS, L4C & ACT Blocks with WD_SATCOUNT_vb.c counter & bleedoff Function--Attachment 6 becomes Attachment 7, e.g.
JimB restarted all these models after the build and install. Reisolation was not a problem. HAM2 did have one trip on CPS during 1st isolation attempt but it worked fine on the 2nd...
SVN COMMITS:
/opt/rtcds/userapps/release/isi/common/models/
hugh.radkins@opsws1:models 0$ svn commit -m "Removed DACKILL remenants & wired WD_STATE to ERRMON"
Sending models/isihammaster.mdl
Transmitting file data .
Committed revision 14424.
/opt/rtcds/userapps/release/hpi/common/models
hugh.radkins@opsws1:models 0$ svn commit -m "Added sensor/actuator WD Saturation Bleedoff ala ISI & removed Test Point WD Flag channels"
Sending models/hepitemplate.mdl
Transmitting file data .
Committed revision 14425.
Still have some medm and SDF cleanup to do. In progress.
Yes I do know how to spell remnants!
I've edited the HEPI and ISI Watchdog medms getting rid of the DACKILL widgets that do not connect to anything now. Also added the IOP DACKILL to the HEPI screen which were absent. Also removed is the RESET ALL button as all it did was push the DACKILL and the RSET buttons. Only 1 WD reset button remains, RESET WD. CLEAR SATURATIONS ONLY button is on the ISIs, pretty self explanitory there. On the HEPIs, I elected to not add the Clear Saturations Only infrastructure as the RESET WD button will do the trick anyway and won't harm the platform if it is not tripped. The RESETs for the IOPDACKILL and the Rogue Exc WD on the ISIs still need to be done in addition to the RESET WD, if, the medms suggest (red) they need it. There is no Rogue Excitation Watchdog on the HEPIs.
I will commit these medms to the SVN when I confirm that LLO does not need the RESET ALL button.
I have updated the TCS corner station model (h1tcscs.mdl). I confirmed that the model comiples without an error. The model is ready for installation tomorrow. The model change includes the followings.
The first item is described in T1600451 in detail. The second point is something we newly implemented today in order to get rid of two paricular ezcaread blocks which had been referencing channels that are in the same front end model. To improve this self-referencing situation, we modified TCS_MASTER.mdl so that the TCS block returns the CO2 power as an output. See the attached screenshot for how we routing the CO2 powers to the SIM block.
This is now installed. See 30429. The filters for the new channels were set by running the code attached to T1600451-v2. The SDF is also updated.