Similar to what I did yesterday for HEPI re 21309 I made OBSERVE.snap files for all the ISIs.
However, I came upon an issue for the BSC ISIs where toggling the MONITOR SELECT choice button to ALL left the SDF with differences.
These come from the following: When an ISI or HEPI Isolates, it observes and averages the free hanging position for 5 seconds and puts this mean into the SETPOINT_NOW. This makes the residual error of the loop zero when it is turned on. After the loop is completely on, the Guardian puts the Reference Location (e.g.: H1:ISI-BS_ST1_CPS_X_TARGET) into the SETPOINT_NOW record causing the Isolation loop to servo to that same position from platform trip to platform trip. That is for some but not all platforms/dofs. All HEPIs and all HAM ISIs are resorting the TARGET. No BSC ISI is restoring the Target location.
So even though I just took a new OBSERVE.snap, many of the BSCs showed diffs out at e-10 to -15 in the SDF. I'm not sure why that was variable but it brought the realization forward that each time a BSC ISI tripped, the new SETPOINT_NOW would be different (as it would for all the HEPIs and HAM ISI,) and would not be replaced by the TARGET Location at the end of Isolation (as it would be for the HEPIs and HAM ISIs.)
So for the BSC ISIs, after getting the OBSERVE.snap loaded into SDF as done for the HEPIs and HAM ISIs, the NOT MONITORED channels were all set to monitor and then the SETPOINT_NOW channels were set back to NOT MONITORED. Given this, the BSC ISIs will not be pink and the MONITOR SELECT button will be set to MASK, rather than ALL like the HEPIs and HAM ISIs. It may be that I will do the same for all the HEPIs and HAM ISIs if that is best for consistency, TBD.
It seems like shifting the pole from 120Hz to 1.2Hz (alog 21300) reduced the beam jitter greatly only at 60 and 180 Hz. The broad jitter noise floor might have been somewhat reduced between 10 and 100Hz, but I see no change at 300Hz. In the attached, references are before, current traces are after.
Robert reported that the coupling to the chiller water flow dominates the periscope motion at around 300Hz (alog 21273).
Laser Status:
SysStat is good
Front End power is 33.02W (should be around 30 W)
Frontend Watch is GREEN
HPO Watch is RED
PMC:
It has been locked 8.0 days, 1.0 hr 19.0 minutes (should be days/weeks)
Reflected power is 2.097Watts and PowerSum = 26.62Watts.
FSS:
It has been locked for 0.0 days 1.0 h and 10.0 min (should be days/weeks)
TPD[V] = 1.498V (min 0.9V)
ISS:
The diffracted power is around 7.045% (should be 5-9%)
Last saturation event was 0.0 days 1.0 hours and 10.0 minutes ago (should be days/weeks)
In prep for turning ~all settings configuration control over to SDF, I have turned off the Set Point Monitoring (SPM) feature on the SUS guardians that were in false alarm (yellow). We will not be using the SPM feature as a requirement for O1, although any ones that go into alarm which are stll enabled may be used as a good troubleshooting tool. (Most that are enabled have been green and running for many months now, so can't hurt to stay on.)
Running pump cart in parallel near BSC9 until ion pump reaches operating temperature
(TJ writing as Travis)
Reminder to Operators: We are NOT adding a second stage of whitening anymore. We are calibrrated for only one stage, and a rare case of zero.
The SSD RAID attached to h1tw0 (and h1nds0) has failed. Until the problem can be diagnosed and fixed, minute trend data won't be available from h1nds0.
late entry for yesterday's CDS maintenance
DAQ frame writer install
Dave, Jim:
The second fast frame writer was connected to the LDAS Sataboy-raid system. All frame writers were then renamed, details in Jim's alog
Adding Duotone Channels to the Science Frame
Keita, Dave:
The h1omc, h1calex and h1caley models were modified to all DUOTONE timing signals from all three buildings to the science frame at 16kHz. These signals are being readout by a filtermodule, to capture the raw signal before any filtering/gain/offset could be applied, we are using the IN1 channel.
H1CALEX.ini:[H1:CAL-PCALX_FPGA_DTONE_IN1_DQ]
H1CALEY.ini:[H1:CAL-PCALY_FPGA_DTONE_IN1_DQ]
H1OMC.ini:[H1:OMC-FPGA_DTONE_IN1_DQ]
Loading all the filters
Sheila, Dave:
SUS PRM and SRM had partially loaded filters, we performed a full load of these systems.
Restarted External Alerting System
Dave:
I restarted the external alert system which had stopped between 6pm and 7pm PDT Monday. I'm looking into a mechanism to restart this system.
ER8 Day 23. Maintenance day. No unexpected restarts. DAQ completion of frame and trend writers. OMC and CAL changes to add duotone channel to science frame.
model restarts logged for Tue 08/Sep/2015
2015_09_08 09:21 h1tw0
2015_09_08 09:57 h1tw1
2015_09_08 12:12 h1omc
2015_09_08 12:15 h1calex
2015_09_08 12:17 h1caley
2015_09_08 13:10 h1broadcast0
2015_09_08 13:10 h1dc0
2015_09_08 13:10 h1nds0
2015_09_08 13:11 h1broadcast0
2015_09_08 13:11 h1nds0
2015_09_08 13:12 h1nds1
2015_09_08 13:12 h1tw0
2015_09_08 13:12 h1tw1
2015_09_08 13:18 h1dc0
2015_09_08 13:25 h1dc0
2015_09_08 13:26 h1nds0
2015_09_08 13:26 h1nds1
2015_09_08 13:28 h1broadcast0
2015_09_08 13:28 h1tw0
2015_09_08 13:28 h1tw1
2015_09_08 13:58 h1nds1
ER8 Day 22. No restarts reported
8:34 Noticing that LLO was down, and determining that the OMC whitening hadn't been engaged, I turned on the whitening. This took the IFO out of Observing mode for 2 minutes, but gained us a few MPCs.
9:18 Big glitch, ETMy saturation
10:42 lockloss, ITMx saturation. I reloaded ISC_DRMI guardian per Sheila's instructions.
11:25 lockloss @ CARM_ON_TR, PRM saturation. I also fixed a typo in the ISC_DRMI guardian that started throwing errors.
11:36 lockloss @ PREP_TR_CARM. PRM, SRM, ETMx, BS, SR2, and MC2 saturation.
11:49 began initial alignment after seeing things looked a little wonky.
12:05 started locking
12:45-12:52 SUS OMC SW watch dog tripped 8 times
13:47 lockloss REDUCE_CARM_OFFSET
14:02 lockloss @ PREP_TR_CARM. PRM, SRM, ETMx, BS, SR2, and MC2 saturation.
14:10 lockloss REDUCE_CARM_OFFSET
14:16 lockloss REDUCE_CARM_OFFSET
14:26 lockloss @ PREP_TR_CARM. PRM, SRM, ETMx, BS, SR2, and MC2 saturation.
14:34 lockloss @ DRMI_LOCKED, PRM, SRM saturation.
14:56 lockloss @ REDUCE_CARM_OFFSET.
Pretty tough relocking this morning for no obvious reason; wind and seismic are all calm. The couple times I tried locking PRMI, it would always lock on split modes which required touching up both BS and PRM. Afterwards, it would lock DRMI OK, but the next time it required touch up again. Only once was I able to make the PRMI->DRMI transition trick work.
Yesterday the grout forms were placed and secured around the HEPI piers at HAM 1 with the intention of pouring the grout on the next maintenance day 09/15. I also checked the nitrogen flow at all of the LTS containers. I was also able to crane the TCS chiller that I craned over the tube last week for Jodi back over the tube for Jeff B.
As Jeff reported yesterday (alog 21280), we have completed the assessment of the suspension scaling factors. In order to finish up the suspension side of the calibration, all we needed to do is to update the CAL CS filters that should accurately simulate the latest suspension transfer functions. So I implemented and tested the new filters today. It seems good so far -- no strange specral shape or drastic change in the DARM strain curve were seen as expected. The filters are ready to go.
The screen shot attached below is a quick comparison of the calibration -- the blue curve is with the old ER7 suspension filters engaged and the red curve with the new ER8 suspension filters engaged. As shown in the plot, there is no big difference as expected. Unfortunately since the low frequency components below several 10 Hz seemed to be sufficiently non-stationary that it does not allow us to make accurate comprasion between the old and new filters in this way.
(Overview and implementation process)
The implementation of the suspension filters has been done in the order of
autoquack to implement the discretized models into CAL CS.Step 1, 2 and 3 are done by a single matlab script which can be found at:
aligocalibration/trunk/Runs/ER8/H1/Scripts/CALCS/quack_eyresponse_into_calcs.m
The script calles H1DARMOLGTFmodel_ER8.m at first in order to collectively obtain all the up-to-date parameters. Subsequently it extracts the necessary suspension responses and convert them into discretized filters. The way we handle the suspension filters are described in the next section. Once the filters are in the discrete format, then the script runs autoquack to directly edit the foton file of CAL CS.
Step 4 is done by a different script;
aligocalibration/trunk/Runs/ER8/H1/Scripts/CALCS/copySus2Cal.py
It simply copies the relevant parts of H1SUSETMY.txt to H1CALCS.txt. There is one exception that is FM1 of ETMY_L1_LOCK_L which caused a trouble before (alog 20700 and alog 20725). The script is now coded such that it modifies this particular filter and gets rid of the integrator. I followed our latest approach that is to replace the filter module with zpk([0.3], [0.01], 30, "n") as described in alog 20725. Optionally, one can run a python script which turns on the necessary switches and filters in CAL CS. The script can be found at
aligocalibration/trunk/Runs/ER8/H1/Scripts/CALCS/setCalcsEtmy.py
Right now, both python scripts edit the ITMY part of CAL CS instead of the ETMY part so that we can make a comparison between them -- the ITMY signal paths represent the new suspension filters while the ETMY paths still uses the old ER7 filters.
(Accuracy of the suspension responses)
One of the tricky parts in implementing the suspension filters into a front end model is that we have to pay attention to how accurately the front end can mimic the suspension responses. There are practical limitations associated with this issue. For example, a single filter module can not handle more than 10 SOS segments. Also, the distortion of the filter is inevitable (see for example G1501013 ) because of the finite sampling rate. So it is extremely important to check if the installed filters are accurate.
In order to asses the accuracy, I made a comparison between the discrete suspension responses and the state-space full suspension model. The plot shown right below is a plot of the transfer functions only in magnitude. The dashed lines are the full state space models that we are trying to mimic and the solid lines are the resultant direcrete models. As one can see, the TST stage looks accurate as the dashed and solid curves are almost completely on top of each other. On the other hand the PUM and UIM stages have a big difference as they go to high frequencies. This is due to compromize for the extremely high-q violin modes and I will explain this in the next section. Apart from the violin modes, UIM had the largest decrepancy at around the mechanical resonances, in 0.3 - 1 Hz. By the way the y-axis is meant to be m/N.
If we take ratio of (discrete transfer function) / (full state space model), it is going to look like this:
As mentioned earlier, the TST stage is very accurate across the entire frequency band. On the other hand, the PUM and UIM deviate from the full state space models as they go to high frequencies. Even though they both deviate from the full ss models by 2.4 % at 100 Hz and 0.2 % at 30 Hz in magnitude, this is already good enough. PUM is the leading-actuator in a band of 1-30 Hz and that is the band where the PUM is accurate at a sub-pecent level. UIM takes care frequencies below 1 Hz and does not really matter above 1 Hz, although I made a slight hand tweak as described in a subesequency section in this alog. Overall, this result is satisfying.
(Violine modes' Q are intentionally lowered)
The current full ss suspension model assumes Q of the violine modes to be on the order of 1e9. Regardless of whether it is accurate or not, it is not a good idea to directly implement such filters in a front end. If they were implemeted with such a high Q, their impulse respones would last forever. Also, the resonance is so tall that any kind of numerical precision error would ring it up. Anyway, I did not like the high-Q filters and therfore artificially lowered their Q's to 1e3 such that they damp on a time scale of a few seconds. At the beggining, I tried completely getting rid of the violin modes, but this turned to be a bad idea because they produced a large descrepancy in PUM by 1 or 2 % at 30 Hz where the accuracy matters. So instead, I decided to keep the vilone modes but with a much lower Qs.
I edited quack_eyresponse_into_calcs.m such that it automatically detects high-Q components and lowers the Q to a user-specified value. Also, the code still internally keeps the high-Q version of the filters and therefore switching back to high-Q is trivial (if necessary in future). This lower-Q modification is applied in PUM and UIM.
(A compromise factor in UIM)
There was another trick. Since UIM needed more poles and zeros than TST and PUM do, it was difficult to simulate the UIM mechanical response in a single filter module. I could have split it into multiple filter banks, but as UIM does not really impact on the calibration, I decided to go with a single filter module. I decreased the number of zeros and poles by running matlab's minreal with a high tolerance and it resulted in frequency-dependent inaccuracy as shown in the above plots. In addition to the frequency dependent components, the response in a band of 1-30 Hz became lower than that of the full ss by roughly 2%. Even though this does not matter, I decided to raise the discrete response by multiplying an extra scale factor such that the UIM is accurate in 1-30 Hz band for completeness.
(A coarse comparison between ER7 and ER8 suspension filters)
This is a screenshot of foton comparing the suspension filters from ER7 and the ones for ER8. The black solid, blue solid and green solid lines are the new UIM, PUM and TST filters in meters/counts respectively. The dashed lines are the old ER7 suspension filters. There are some remarks:
(Implemented filters)
Here is a screen shot of the filter modules for the new suspension responses in CAL CS. Currently they are installed in the ITMY filter modules for convenience. The suspension filters are split into multiple modules as described in the followings.
I calculated the time varying calibration parameters kappa_tst, kappa_pu, kappa_A, kappa_C and Cavity pole for the lock stretch beginning from September 1st using the new ER8 DARM model. All the data used for this analysis had guardian state vector greater than 600 (NOMINAL LOW NOISE). The plots are attached.
Details:
The equation used in this calculation are obtained from the T1500377 document which describes the time varying parameters in details.
For the displacement produced due to pcal, I used the following script to advance the pcal signal by 21 us (LHO alog 21320) , take out the AA filter and and dewhitened the signal.
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/S7/Common/MatlabTools/xpcalcorrectionER8.m.
I was not able to get kappa_tst closer to 1, which is the expected value of the all kappa's. However, the variation in Kappa_tst is within 1% which is a good news. I am not sure where the problem is. I will look into it further. For the rest of the plots I assumed kappa_tst to be 1. This assumption only effects kappa_pu because it is the only factor that depends on kappa_tst.
Plots:
The first plot shows the real part of kappa_tst, kappa_pu and kappa_A. Kappa_pu and kappa_A are close to 1 and the variation of these parameters during this time is less than a 1%. kappa_tst under investigation.
The second plot is the imaginary part of the parameters in plot 1. All these values are expected to be 0 or closer to zero. Again, kappa-pu and kappa_A are indeed very close to zero but kappa_tst is not.
The third plot includes kappa_C (change in optical gain) and cavity pole. Optical gain within a lock stretch seems to be very stable but it varies by few percent between locks. One particular stretch it varied by almost 20% (not sure why, could be commissioning activities). Also, there is evident of some transient at the beginning and end of the locks. Will try to sort the data by intent bit for future analysis. The cavity pole shows some trend and variation between lock stretches but nothing crazy.
The script used to analyze this is committed to the svn:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Projects/PhotonCalibrator/drafts_tests/ER8_Data_Analysis/Scripts/plotCalparameterFromSLMData.m
For the record, this analysis was performed on all data above guardian lock states above "600," (which I think is DC readout) i.e. more than just lock stretches that have the observation intent bit on.
Collected OBSERVE.snap files for use in full observing configuration monitoring.
With the SDF happy (green) and Guardian Nominal (Robust Isolated) and green, I made an OBSERVE.snap file with SDF_SAVE screen
choosing EPICS DB TO FILE & SAVE AS == OBSERVE
This saves the current values of all switches and maintains the monitor/not monitor bit into the target area, e.g. /opt/rtcds/lho/h1/target/h1hpietmy/h1hpietmyepics/burt as OBSERVE.snap
This file is then copied to the svn: /opt/rtcds/userapps/release/hpi/h1/burtfiles as h1hpietmy_OBSERVE.snap and added/commited as needed.
The OBSERVE.snap in the target area is now deleted and a soft link is created for OBSERVE.snap to point to the svn copy:
ln -s /opt/rtcds/userapps/release/hpi/h1/burtfiles/h1hpietmy_OBSERVE.snap OBSERVE.snap
Finally, the SDF_RESTORE screen is used to select the OBSERVE.snap softlink and loaded with the LOAD TABLE button.
Now, for the HEPIs for example, the not monitored channels dealt with by the guardian will be a different value from the safe.snap but, the not monitored channels are still not monitored so the SDF remains green and happy. And if the HEPI platform trips, it will still be happy and green because, the not monitored channels are still not monitored.
What's the use of all this you say? Okay, I say, go to the SDF_TABLE screen and switch the MONITOR SELECT choice to ALL (vs MASK.) Now, the not monitored channel bit flag is ignored and all records are monitored and differences (ISO filters when the platform is tripped for example) will show in the DIFF list until guardian has the platform back to nominal.
Notice too that the SDF_OVERVIEW has the pink light indicating monitor ALL is set. This should stay this way unless Guardian is having trouble reisolating the platform and then the operator may want to reenable the bit mask to make more evident any switches that guardian isn't touching more apparent.
But rather than rely on selecting ALL in the SDF_MON_ALL selection, I would suggest you actual set the monitor bit to True for all channels in the OBSERVE.snap. That way we don't have to do a two-step select process to activate it, and we can indicate if there are special channels that we don't monitor, for whatever reason.
Yes Jameson. That is why I selectied the ALL button allowing all channels to be monitored.
Hugh, I think the OBSERVE snaps should have the montor bit set for all channels. In some sense that's the whole point of having separate OBSERVE files to be used in this way, that we use them to define setpoints against which every channel should be moniotred.
Evan, Sheila, Jeff B Ed
Earlier today we were knocked out of lock by a 5.2 in New Zealand, the kind of thing that we would like to be able to ride out. The lockloss plot is attached, we saturated M3 of the SRM before the lockloss and PRM M3 was also close to saturating.
While LLO was down, we spent a little time on the offloading, basically the changes described in alog 21084. This offloading scheme worked fine in full lock for PRM, however we ran into trouble using it durring the acquisiton sequence. Twice we lost lock on the transition to DRMI, and twice we lost lock when the PRM coil state swtiched in DRMI on POP. Hpwever, we can acquire lock with the new filter in the top stage of PRM and SRM, but the old low gain (-0.02). We've been able to turn the gain up by a factor of 2 in full lock twice, so I've left the guardian so that it will turn of the gain in M2 (before the intergrator) in the noise tunings step.
If anyone decides they need to undo this change overnight, they can comment out lines 344-347 and 2508-2514 of the ISC_LOCK guardian.
Before we started this, the PRM top mass damping was using 10000 cnts rms at frequencies above a few hundred Hz, because of problems in the OSEMS (alog 21060 ). Evan put some low passes at 200 Hz in RT and SD OSEMINF which reduces this to 2000 cnts rms. Jeff B accepted this change in SDF.
The second attached screenshot shows the PRM drives, the references are in the minutes before the earthquake dropped us out of lock. The red and blue curves show the current drives, with the high frequency reduction in M1 due to Evan's low pass, and the new offloading on. The last attached screenshot shows SRM drives with the new offloading.
I don't think the 5.2 EQ is the cause of the lockloss.
According to your plot, the lockloss happened on Sep 08 at 00:31:07 UTC. The 5.2 EQ happened on Sep 07 at 20:24:56.84 UTC and hit the site at 20:38:21 UTC according to Seismon (so 4 hours earlier). The BLRMS plot confirm that statement (see attachment).
Around loss time, the ground seems as quiet as usual.
I was mistaken in identifying the earthquake, but the ground motion did increase slightly, which seems to be what caused the lockloss.
None of these things changed the DARM noise. (just the calibration)
As a follow-up to point #4: I redid the bias reduction test, this time by reducing the voltage from 380 V to 190 V.
As before, there was no obvious change in the DARM noise. [See attachment.]