A core switch in the OSB hung up over night. A reboot of the offending switch returned service to wired network, wireless, and phones at approximately 08:35AM PT.
Sheila, Elli, Evan, Ryan, Lisa Apparently today was a particularly bad day for the whole locking business. To get the DRMI locking with the arms off-resonant, we had to retune the locking gains as previous settings were not working, meaning we could not get a single lock of DRMI + off resonant arms in ~ 20 min. After carefully aligning the DRMI (with no arms), we changed the locking settings as follows: DRMI ---> DRMI with off-resonant arms MICH 3 ---> 3 PRCL 15 ---> 8 (it was 11) SRCL -50 ---> -40 (it was -45) It would be extremely helpful to test these new settings again several times, and do some systematic lock acquisition studies from DRMI to DRMI with off-resonant arms in the morning. It still takes too long to acquire lock. We then went through the locking sequence, and stop at some point to give Evan and Elli a chance to measure the opto-mecahnical TF of the DHARD alignment DOF, so we could design a good filter to increase the bandwidth of DHARD, if necessary. We broke the lock in the process, and Sheila struggled to get ALS DIFF working right after that. While investigating the problem, Ryan found out that we could not get any actuation at all from the ETX L1 stage. Please investigate/fix this in the morning, a manual reboot at the end station might be required. On a positive side: * Evan and Elli got a good measurement of the DHARD plant and designed a compensation filter for that, so we should be ready to test it; * ETMY L1 has now a working L2P filter, so we could eventually try DARM actuation to both end masses, if necessary.
J. Kissel, H. Radkins, Digging deep on the HEPI Pump Servo noise, we measured the voltage amplitude spectral densities of both pressure sensor signals right as they enter the Pump Servo box with a breakout board, some clip leads, and an SR785. The results confirm our suspicion -- the pressure sensor voltage is WELL below what is reported by EPICs. So, it's either the Athena II's ADC noise (1e-2 [V/rtHz] ?? -- seems too high even for a crappy ADC), or some nastiness in turning these channels into EPICs channels. See attachment. In order to get the pressure sensor signal voltage above the readout noise floor without saturating the ADC, one would need either need to AC couple the signal with a ~ 10 [mHz] high-pass and gain up the signal by an additional factor of 1000, or add a whitening filter; something like two zeros at 1 [mHz] and two poles at 50 [mHz]. I don't know anything about electronics, but I think I know enough that such filters would be difficult to construct at best. These pressure sensors clearly were not meant to be anything more than DC sensors, so if we intend to keep using them as is, we should lower the unity gain frequency of the pump servo to WELL below 10 [mHz] (along the lines of what Hugh started to do in LHO aLOG 16466). Up to this point, we've been simply injecting terrible ADC noise into the pumps. This explains why the *very* low frequency response and DC values add up, but anything to do with the spectral densities are just injecting this bad ADC noise into the pumps. We still have to make sure the PID parameters are being employed in the units we expected them to be by the CALC record, but tomorrow's another day. Note -- I'm not yet sure how Rich took his data, but from what I see, both the supply and return sensors are above my measurement noise floor, but the return pressure is significantly (as much as a factor of 5 @ 10 [mHz]) below Rich's eye-ball fit to the noise he measured in SEI aLOG 688. -------- Details: Measurement -- With the pump servo in manual mode (i.e. feedback PID loop OPEN), we inserted a DB9 breakout board in-line with the pressure sensor inputs to the HEPI pump servo (channels 1 [supply] and 2 [return], successively). As directed by the pinout in D0901559, independent clip-doodles were connected to pins 4 & 6, the positive and negative legs of the differential signal (with the shield / black clips clipped to pins 7 & 8, which are the pressure preamp's GND returns). These two legs were then fed into the A and B ports of Channel 1 of the SR785, which was set to A-B mode. Other relevant parameters: Freq: SPAN 3.125 [Hz], LINES 800, LINEWIDTH 3.9 [mHz] Input: INPUT CONFIG A-B, DC Coupled, INPUT RANGE 20 dBVpk Disp Step: UNITS -- db units OFF, Pk units RMS, PSD units ON This measurement was done with the supply line sensor, the return line sensor, with one of the 100 [ohm] resistor plugs from , and finally the SR785 noise floor was determined by sticking 50 [ohm] plugs on the A and B inputs (using the same input range for all measurements -- though I did confirm that the return ASD did not change, and was still above the SR785 noise floor when the input range of 2 dBVpk). Calibration -- The signal chain goes as follows: Pressure Sensor Pressure Preamp Diff 2 S.E. ADC EPICs Calc Record 100e-3 / 300 [V_diff/PSI] 151 [V_diff / V_diff] 2 [V_se / V_Diff] 2^16 / 20 [ct/ V_se] 0.0032 ["PSI"/ct] The SR785 voltage was measured just before the Diff 2 S.E., instrumentation-amplifier, factor-of-2 in the HEPI pump servo. So - to calibrate the 100 [ohm], input terminated signal ASD as reported by EPICs into single-ended ADC input voltage, one needs EPICS ["PSI"] ( * (1/0.0032) [ct/"PSI"] * 20 / 2^16 [V_se / ct] = 0.0477 [V_se / "PSI"] ). - to calibrate the differential SR785 voltage into single-ended ADC input voltage, one needs SR785 Voltage [V_diff] ( * 2 [V_se / V_diff] = 2 [V_se / V_diff] ). - to calibrate the differential SR785 back to PSI at the sensor, SR785 Voltage [V_diff] ( * (1/151) [V_diff / V_diff] * 300 / 100e-3 [PSI / V_diff] = 19.8675 [PSI / V_diff] ) What did else did I learn today? - The EPICs calc record defines the sampling frequency to be 10 [Hz]. EPICs data is normaliy stored at 16 [Hz]. That means the features seen in the EPICs spectra (the ~1.6 [Hz] and harmonics notches) could just be Fourier reflections NOT features in the plant. I don't know what the native rate of the Athena II ADC is, I have an email out to Ben. But there is certainly some sort of nasty aliasing / imaging nonsense going on. - The .ini files that determines which channels are stored in the frames for these pump servos lives here: /opt/rtcds/lho/h1/chans/daq/ H1EDCU_HEPIPUMPL0.ini H1EDCU_HEPIPUMPEX.ini H1EDCU_HEPIPUMPEY.ini - The CALC record that calibrates the EPICs channels from [ct] to [PSI], and also defines the sampling rate which the records are picked off of the ADC lives in /opt/rtcds/lho/h1/target/h1hpipumpl0/h1hpipumpl0epics/db/ a.12.db - An SR785's high-pass for its AC coupling mode is at 160 [mHz], so for these low frequency measurements, I sure-as-heck need to be DC coupled, and that limits my choice of input range, since *actual* DC signal is clearly what dominates these pressure sensors.
The epics record is actually:
/opt/rtcds/userapps/release/hpi/h1/epics/CSPumpServoGeneric.db, not:
- The CALC record that calibrates the EPICs channels from [ct] to [PSI], and also defines the sampling rate which the records are picked off of the ADC lives in /opt/rtcds/lho/h1/target/h1hpipumpl0/h1hpipumpl0epics/db/ a.12.db I'll talk with CDS about getting all the correct files in the correct place.
PID Control, Epics R13.2
from page 153 of the EPICS Record Reference Manual,
delM(n) = KP * [(En-En-1) + En*dTn*KI + KD{(En-En-1)/dTn - (En-1 - En-2)/dTn-1}]
where delM is the change in the output, that is, the change in the voltage to the VFD driving the pump motor,
the Ks are the PID parameters, E is the Process Variable error: setpoint - current value (VAL - CVAL), and,
dT is time difference from n-1 to n.
EPICS SCAN RATE
Further info: the 0.1 second scan rate for this system is the fastest allowed by EPICS: 0.1, 0.2, 0.5, 1, 2, 5, & 10 seconds are the allowed scan rates.
Summary:
There was some suspicion that the fast shutter was firing prematurely during CARM transition at 2W, breaking the lock, but it doesn't seem likely.
Details:
With a single bounce beam with 5W into IMC, H1:SYS-MOTION_C_SHUTTER_G_TRIGGER_VOLTS outputs about 0.02 volts (after subtracting dark offset of about 2mV).
With this input, the IR power coming to HAM6 should be roughly 5W*0.03(PRM)*0.25(BS)*0.2 0.37(surrogate SRM) = 14mW.
[Voltage]/[HAM6 power] of that signal is 20mV/14mW = 1.4 Volt per Watt on HAM6.
By changing threshold (H1:SYS-MOTION_C_SHUTTER_G_THRESHOLD) I confirmed that the shutter triggers when the trigger voltage crosses the threshold, which is usually set at 2V.
So the shutter should trigger when HAM6 power is 2/1.4=1.4W, which sounds OK to me.
I don't know where exactly the H1:SYS-MOTION_C_SHUTTER_G_TRIGGER_VOLTS signal is monitored, and there's something odd about how the threshold voltage receiver has a gain of 0.2 while the trigger voltage receiver has a gain of 1 in D1102312, but anyway just from observing how the signals behave at which power, things don't look crazy.
If people still think that the shutter is triggered prematurely, we need to hook up a fast ADC channel to the trigger voltage. That voltage comes from the summation output of the AS_C interface board and recorded in EPICS only, AS_C_SUM is a test point, so we cannot do any serious trigger analysis except in real time for the moment.
I was just checking that the fast channels coming out of PEM EX,EY look the same before and after the CAL-PEM merge on Tuesday. The PEM channels look good, but I noticed PCAL-EY differences. This was tracked to a lack of a burt restore for the EY oscillator settings. At 17:10PST today I burt restored from the h1caleyepics.snap file from 00:00 Sun 01 Feb 2015 and the PCAL photodiode now shows a varying signal. Rick and Sudarshan are going to check the system looks OK.
Here is a block diagram of the signals that go into the summing node board, the IFO common mode board and the IMC common mode board during the CARM handoff. It can be used as a reference when following the recent commissioning alogs.
7:30? Karen, Cris - LVEA
8:15 Elli - To EX and EY
8:20 Corey - To Squeezer Bay and around LVEA
8:32 Mitch - To W. Bay
8:41 Sudarshan, Thomas - To EY
8:46 P. King - To H2 Enclosure
8:47 Mitch - Back
9:10 R. McCarthy - To EY
9:17 Greg - To LVEA
9:27 Greg - Back
9:34 Corey - Back
9:58 R. McCarthy - Back
~10 J. Kissel, Hugh - To Pump Room setting the pump servo to manual
10:59 J. Kissel, Hugh - Back for meetings, pump servo still on manual.
12:23 R. McCarthy - To EX and EY
12:48 J. Kissel, Hugh - To HEPI Pump
13:02 Alastair, Elli - To LVEA to work on TCSY table
14:05 Alastair, Elli - Back
14:05 J. Kissel, Hugh - Back
15:40 Jodi - To MY
15:44 P. King - Back from H2 Enclosure
Earlier today, with Ryan's encouragement, I turned on more of the I/ETMY isolation loops. Currently these 2 ISI's are running a configuration like LLO's, with all DOFs engaged on St1 and X,Y,Z,RZ running on St2. The St1 RZ loop is running a high (750mhz) blend with no T240. St2 is running the same 250mhz blend on Z and RZ as it was previously running on X and Y. The Stage Guardians for these two chambers have also been modified to turn on these loops, in case we have some traumatic event.
FF is now running on all test mass chambers as well with X&Y running on ETMY, Y on ITMY and X on E/ITMX.
Turning on the extra loops seems to mostly reduce the suspension point motion, although vertical motion looks to be a bit worse at .1-.3 hz. Attached plots are ETMX and ETMY suspension point motions, respectively. ETMX was running the normal LHO configuration, with no RZ loops on, only X&Y on stage 2, LLO blends plus ST0-1 FF running on the Y degree of freedom. ETMY was running a mostly LLO configuration, with all loops on St1, LLO blends plus the Start blend on RZ, St2 with all loops running except RX & RY, with St0-1 FF running on X & Y.
Jim, Is that "start-blend" still using just the L-4C as the inertial sensor? If so, it is going to be noisier than you want down below 1 Hz. Let's look for a similar high-blend which uses the T-240s.
Plots of the two high blend filters we have. CPS Signal is common to both, blue is inertial (L4C only, no T240 in that blend) part of the Start filter, green and brown are inertial parts of the T750 filter.
Alastair, Elli, Jamie
We're working on Guardian today, and wanted to install a beam dump at the output of the Y-arm table so that we can play with that laser without injecting power to the CP. While doing this we discovered the laser was running only at 1/2 power. The history to this is that a few months ago the same laser was running at half power, and after swapping in a different driver, went back to full power. The "not working" driver was sent to Caltech where it was found to be working fine. So this new episode was actually quite useful in finding that a problem still exists.
We went out to the LVEA and found that the power connector at the patch panel is badly connected. We couldn't effect a permanent solution in the 1hr window we had, but were able to get the laser back to full power again. These connectors will likely need replaced long term, and we'll start working on a solution to that.
The ouput of the table was found to be dumped to a beam dump already (it has been like this since installation of the table was completed).
I've added a Bug 1009 that describes this.
Minor changes to the drift monitor threshold updater script:
The log files are now recorded to: "/opt/rtcds/userapps/release/sus/common/medm/sus_driftmon_logs/", and have the following file name structure: SUSDRIFT_{OPTIC NAME}_{START_TIME}-{DURATION}.log (e.g. SUSDRIFT_ALL_1107199664-15.log, SUSDRIFT_MC1_1107199664-15.log)
Jeff informed me that the userapps repo is not a good place for log files, so I move them back to /tmp/ until a better place can be determined. The same filename structure applies.
no restarts reported.
Kiwamu, TJ, Elli
This morning while EY plugging HWS cables into the ISCTEY feedthrough panel, I touched the fiber bringing the red PLL reference signal to EY. This changed the fiber wrong polarization (see chanel H1:ALS-Y_FIBR_LOCK_FIBER_POLARIZATIONPERCENT) from 1% to 32%, which is above the maximum allowable value of 30%. This fiber is also connected to ISCTEY feedthrough panel, and then some of the fiber is sitting on a coil on top of ISCTEY. All I did to change the polarization was to gently move the coil on top of the table. When Kiwamu and TJ noticed the polarization had changed, I nudged the fiber coil on top of the table around a bit untill the fiber polarization returned to 8%. It seems it is very easy to bump the fiber and change the polarization.
One can use the fiber polarization controller in the MSR to bring it back.
We had another lock of about 40 seconds, durring which all the signals seemed more stable than last night. It was knocked out by an oscillation that showed up in boh the MICH WFS and MICH length loops. We added an extra boost (2 poles at 0.1 and 2 zeros at 2Hz) to the CARM path, which seems to have made things more stable. The lockloss time was 11:56:32 UTC Feb 5th.
Today we saw that our alingment was drifting much more quickly than we had seen in the last few weeks. Without ASC we could only lock DRMI for a few minutes before MICH would get misaligned. The attached screenshot shows the slow drift of ITMX pitch, the red trace, which is about 1.5urad over the last 3 hours. We are not sure if this is real or just OpLev drift. Wspent some time on the MICH ASC loops. We phased ASB 36 so that the BS shows up in Q, and saw that this is a better signal for the WFS throughout the CARM offset reduction, which is what Ryan says they saw at LLO as well. They come on with a bandwidth of about 1 Hz, and hold the DRMI buildup is stable.
We also ran into difficulty tonight with the PSL rotation stage, it stopped responding to the command button for several hours. We need some code that checks if the input power is really the requested power, or we could try running through the whole sequence at the same power.
Here is a StripTool of the lock acquiistion. My DTT trend is taking too long to run, and I am tired...
I have attached the temperature plot during the lock time (time at lock loss - 2 hrs). To see if the ITMX pitch drift was real I attached the vertical dof of the top stage as well...
The LVEA average temperature is taken from many sensors. Here is a map of the individual temperature sensors - which are available in dataviewer, I believe.
For example the temperature sensor nearest ITM X might be Zone 3B sensor 3A.
This lock event shows interesting things. An oscillation in MICH is what eventually unlocks, but (I think) only because at that point the sideband power in the recycling cavity had significantly dropped. As soon as REFL 9I is engaged,there is an oscillation ~ 0.45 Hz showing up in the AS RF45 and 36, and it is clearly visible in the ASAIR RF90 power. The DARM length correction signal shows awful "bursts" at that point.
This is the lock trend I meant to post last night.
This is the conlog replica.
It appears to have started around 5:00 PM and repeated each hour after. I had to restart the database replication, it appears that the database has been crashing in response. It may not be a coincidence that the search for frequently changing channels also runs every hour, so I have disabled it.
cdsadmin@h1conlog3:/var/log$ grep error syslog
Feb 4 17:04:08 h1conlog3 kernel: [2335398.323116] res 51/40:0a:16:e8:88/00:00:00:00:00/0b Emask 0x9 (media error)
Feb 4 17:04:08 h1conlog3 kernel: [2335398.323407] ata3.00: error: { UNC }
Feb 4 17:04:08 h1conlog3 kernel: [2335398.344358] Add. Sense: Unrecovered read error - auto reallocate failed
Feb 4 17:04:08 h1conlog3 kernel: [2335398.344367] end_request: I/O error, dev sda, sector 193521686
Feb 4 17:04:11 h1conlog3 kernel: [2335401.397976] res 51/40:02:16:e8:88/00:00:00:00:00/0b Emask 0x9 (media error)
Feb 4 17:04:11 h1conlog3 kernel: [2335401.398264] ata3.00: error: { UNC }
Feb 4 17:04:11 h1conlog3 kernel: [2335401.420344] Add. Sense: Unrecovered read error - auto reallocate failed
Feb 4 17:04:11 h1conlog3 kernel: [2335401.420352] end_request: I/O error, dev sda, sector 193521686
...
I attempted a simple search and it crashed with the same syslog error. I am going to try to disable the webpage. Please do not try to use conlog until this is resolved.
To clarify, the acquisition of data appears to be ok, but the machine with the copy of the data that is used for user searches is not.
From the MySQL error log: InnoDB: Error: tried to read 16384 bytes at offset 7 692060160. InnoDB: Was only able to read 8192. 150204 17:04:11 InnoDB: Operating system error number 0 in a file operation. InnoDB: Error number 0 means 'Success'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html InnoDB: File operation call: 'read'. InnoDB: Cannot continue operation. 150204 17:04:13 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. 150204 17:04:13 [Note] Plugin 'FEDERATED' is disabled. 150204 17:04:13 InnoDB: The InnoDB memory heap is disabled 150204 17:04:13 InnoDB: Mutexes and rw_locks use GCC atomic builtins 150204 17:04:13 InnoDB: Compressed tables use zlib 1.2.8 150204 17:04:13 InnoDB: Using Linux native AIO 150204 17:04:13 InnoDB: Initializing buffer pool, size = 1.0G 150204 17:04:13 InnoDB: Completed initialization of buffer pool 150204 17:04:13 InnoDB: highest supported file format is Barracuda. InnoDB: Log scan progressed past the checkpoint lsn 47921973351 150204 17:04:13 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 47922305837 150204 17:04:13 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 4307, file name /var/log/mysql/mysql-bin.000221 150204 17:04:14 InnoDB: Waiting for the background threads to start 150204 17:04:15 InnoDB: 5.5.38 started; log sequence number 47922305837 150204 17:04:15 [Note] Recovering after a crash using /var/log/mysql/mysql-bin 150204 17:04:15 [Note] Starting crash recovery... 150204 17:04:15 [Note] Crash recovery finished. 150204 17:04:16 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306 150204 17:04:16 [Note] - '0.0.0.0' resolves to '0.0.0.0'; 150204 17:04:16 [Note] Server socket created on IP: '0.0.0.0'. 150204 17:04:17 [Note] Event Scheduler: Loaded 0 events 150204 17:04:17 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.38-0ubuntu0.14.04.1-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
(Peter K, Richard M, , Filiberto C, Daniel S)
We noticed that the fast channels corresponding to H1:IMC-PWR_IN and H1:IMC-PWR_EOM were never hooked up. This required installing the PD interface box in the PSL enclosure, running the DAQ cable into the PSL and installing a tee in the photodetector readback. (These channels have previously been hooked up to the EtherCAT system, so they that they are available to the rotation stage.) The EOM channel is currently railed and needs an adjustment of the PD gain.
EPICS updates:
Another update:
I have put a filter in IMC-PWR_IN in order to calibrate the signal into watts. I used the slow readback (i.e. H1:PSL-PERISCOPE_A_DC_POWERMON) as a reference for the calibration such that IMC-PWR_IN matches to the slow one. So now the filter bank looks like this:
What new ETMY L2P are you referring to?? We found no filters engaged and a gain of 1.0, and the output OFF.
The tidal feedback goes to L1 state. Locking the Y-arm on green and engaging the slow servo caused a really bad L2P with the above configuration. We did a burt restore to yesterday morning (L1 L2P FM8 z0.5p10 gain -1, output ON), and everything is fine now.
The EY L1 L2P state that I found this morning is not what Ryan designed last night ( FM9 (zpk([],[0.5],1,"n")gain(0.001)), Gain 1 and output ON). However, this does not work with the slow servo on.
The EX L1 L2P FM9 was installed zpk([],[0.5],1,"n")gain(0.001). The overall filter gain remained 5.2. Not much invesitgation has been done, but based on the green transmission dips, it seems that the old FM10(-60dB) setting is better.
Elli, Evan
We have taken a TF for the DHARD pitch plant. Now filters are installed in ETMX/Y L2 LOCK P which invert this plant.
With the DHARD loop open (and oplev damping engaged), we drove band-limited white noise (up to ≈4 Hz) into DHARD EXC, which drives the ETM PUMs differentially in pitch. We then measured the transfer function DHARD EXC → DHARD IN1. This gives the plant transfer function for the DHARD loop, which we then fit by eye in Matlab to a zpk model.
The fit is not so great in phase above 0.5 Hz, but maybe as a first try it is good enough to allow us to bump up the DHARD bandwidth anyway.
The filter modules are FM6 and FM7.