Ran new cables at EX for CPS sync fanout install. Cables are run from SUS-R1 rack to CPS units on chamber. Fanout chassis and RF cable (71MHz) still need to be installed/pulled. Filiberto C. & Ed M.
New lab dust monitors
Jeff B, Jim, Dave [WP5697]
As part of the install of the new OSB lab dust monitors:
generated new autoBurt.req file. Generated new DAQ EDCU INI file. Recanned conlog against autoBurt.req file.
New GDS version
Jim [WP 5696]:
completed this morning.
Clearing partially loaded filter modules
Was able to do full loads on susbs, susitmx, susitmy, susprm, sussrm. Still have partial loads on lsc, omc, suspr2 and sussr2 which are waiting for the system experts to clear.
DAQ reconfigure
DAQ EDCU was reconfigured for the missing h1tw0 and new DUST channels. Was applied during today's two DAQ restarts.
DAQ Restarts
In both DAQ restarts the dataconcentrator, tw1 and both framewriters started with no issues. In the first restart the broadcaster started by both NDS daqd process were no longer controlled by monit. A simple monit restart of the daqd processes cleared the error. On the second restart the broadcaster started, stopped and later restarted. Again both NDS processes did not start, and when they were restarted they failed. The error was that high dcuid nodes for the Beckhoff SDF system has been mistakenly added to the testpoint.par file, and only the NDS machines objected to this. The file was repaired and monit was able to restart the NDS daqd processes.
TCS model changes
Aidan and Dave [WP5688]:
Installed new h1tcscs model onto h1oaf0. DAQ was restarted.
New Beckhoff SDF code release
Jonathan [WP5695]:
Jonathan tested the new code on EY Beckhoff. He found an issue with enumerated binaries and backed this version out.
RACCESS monitoring/control of h1hwinj1
Jonathan, Jim, Dave:
Jonathan added the hardware injection machine h1hwinj1 to his RACCESS control system. MEDMs were updated accordingly.
H1OAF change to drive audio in control room
Richard, Evan, Dave [WP5704]:
h1oaf and h1pemcs models were changed to permit oaf to drive the 8th DAC channel of PEM's 18-bit DAC card. This analog signal is then routed from the CER to the control room using the audio channel in the digital video fiber link between the rooms.
New operator table install in the control room
Richard, Carlos, Jim, Dave:
The new powered, adjustable-height operator table was installed in the control room at the operator station. The access control PC, gate cam and alarm machines were shuffled to the left and a new workstation was installed on the new table.
HPI and ISI safe.snap work
Hugh, Dave:
Hugh worked on the safe.snap files for HPI and ISI. I was able to offer some help by creating a new script to convert sub-systems to safe SDF mode and converted all HPI and ISI to safe. Later I converted them back to OBSERVE mode.
New ECAT EPICS channels for corner station
Daniel, Dave:
Daniel made Beckhoff changes in the corner station. I updated the ECAT DAQ INI files and the target autoBurt.req files.
Server patching and rebooting
Carlos, Dave:
Carlos applied security patches and rebooted cdsssh, cdslogin and cdsadminctrl. We restarted the vacuum alarms system on cdslogin.
Installed Whitening Board (D1500389) to ETM Low Voltage Driver Chassis (D1500129) per ECR E1500417. ETMY LV Driver S1500066 - Whitening board installed S1600032 ETMX LV Driver S1500073 - Whitening board installed S1600035 High voltage was disabled prior to disconnecting of any cables. Verified outputs were not railing when units were reinstalled. One thing worth noting, is that the wiring for the binary cables is not consistent at both end stations. For now, we left cables in the same configuration we found them this morning. Filiberto C. & Ed M.
Day Shift: 16:00-23:59UTC (08:00-16:00PT)
State of H1: down all day, down more than 12 hours, currently JimW at EY to fix ESD driver
Shift Summary:
Activities logged in alogs 25155, 25163, and 25171
Other activities:
Two items:
I started updating the slow controls SDF models to the latest version in trunk. This was to fix a bug Daniel saw after the power outage. Unfortunately there were problems with the channel type detection code so we fell back to running the old code again. I have left the work permit open, I will debug, test, and deploy later.
Crashed earlier in the day, I'm guessing with the Beckhoff restart: Jan 26 13:07:00 h1conlog1-master conlog: ../conlog.cpp: 301: process_cac_messages: MySQL Exception: Error: Out of range value for column 'value' at row 1: Error code: 1264: SQLState: 22003: Exiting. Dave updated the channel list to remove the dust monitor channels.
Richard, Dave, Evan
We added infrastructure in the OAF model to take LSC error signals, filter them, and output them to an 18-bit DAC channel in the PEM frontend. This signal is then piped into the control room and can be played on a speaker.
MEDM screeens still need to be made.
1 h1psl PSL.team HPO optic exam x113 00:02:19
2 ISC/IO/ALS Richard add sensors to all tables 202 00:01:19
3 TCSY Alistair startup TCSY laser 202 00:01:19
4 LVEA crane Bubba shim rails - requires lift 202 00:00:48
5 TCS Aidan/Alastair install/modify 202 00:01:21
6 optics lab Betsy work 212 00:02:19
7 ESD drives filiberto/ed EX and EY 212 00:02:38
8 mid-y john/bubba check cryo pumps 202 00:02:43
9 EY Kyle check pumps 202 00:02:48
10 OAF Dave+Evan new oaf and pem, audio testing 202 00:03:44
List of reservation system - some times are longer than the task is expected to take 1 h1psl PSL.team HPO optic exam x113 00:03:40 2 ISC/IO/ALS Richard add sensors to all tables 202 00:02:40 3 TCSY Alistair startup TCSY laser 202 00:02:40 4 LVEA crane Bubba shim rails - requires lift 202 00:02:09 5 LN2 company? delivery on site 202 00:01:03 6 TCS Aidan/Alastair install/modify 202 00:02:42 7 optics lab Betsy work 212 00:03:40 8 ESD drives filiberto/ed EX and EY 212 00:03:59
Before bringing the SEI platforms back up. Checked the diffs with the safe.snaps in SDF. There were a few chages related to GS13 gains and FF ON/OFFs. All updated and committed to svn.
Brought all the SEI Platforms down and spun down the HEPI Pumps to check the Accumulator charge. Most accumulators showed a few PSI drop--likely from pressure gauge loss. No charge dropped to action level. It was mid July last time this was done. This suggests none of the accumulators are leaking. I'll check again in a couple months--checking the pressure sometimes loosens the valve and it can start leaking.
In anticipation of going to Observe last night (which didn't happen due to a lockloss sometime just before 1am), I had accepted some ISI SDF diffs while JimW was still here to approve them. I'm sure he'll look at his data more, and make a final decision on whether we should keep the sensor correction changes or not, but for now the Guardian will probably put the EY ISI back in its old nominal state, so we'll see a bunch of SDF diffs when we get to NomLowNoise. Anyhow, I attach a screenshot of what I accepted, and we can re-accept the old values later today, until Team SEI decides to change the Guardian.
Updated the corner TwinCAT system to the most recent svn. This includes the changes needed for adding the ASC-AS_A/B.RF90 channels.
The new AS_B.RF90 demod channels won't be active until the corner 4 chassis is upgraded. The AS_A.RF90 demod channels have been taken over from the old (now deleted) AS_D. Whitening channels for both AS_A and AS_B are working.
Added whitening screens for AS_A/B_RF90 to ISC-CUST_WHITENING_OVERVIEW.adl
Added related screen buttons for AS_A/B_RF90 to ASC_OVERVIEW.adl
WP 5696, ECR 1500402 The version of GDS software for the control room has been updated to branch gds-2.17.1.3. This release addresses several Bugzilla bugs regarding data access, especially NDS2 data access. A new NDS2 input section has been added which will allow the user to specify an epoch start and end time, which will limit the channel list to those that existed during the epoch. For those channels whose data rate was changed without changing the name, it will be possible to specify an epoch where only a single data rate exists. (This assumes the user knows the time that the data rate changed.) If multiple channel rates for a channel exist during the chosen epoch, those rates will be displayed when channels are selected. Because of the way NDS2 returns data, it is still possible to get incorrect data if you choose a data rate for a time when the data really doesn't exist, so it's best to narrow the epoch to display only a single data rate for the channel. Diaggui now warns you if you don't have a kerberos ticket when attempting to use the NDS2 input. A release note document has been generated for the GDS tools, see T1600007.
Current activities:
1 h1psl PSL.team HPO optic exam x113 00:04:57
2 HAM4/TCS Filiberto/Ed cables 4 hartman 202 00:03:14
3 CPS/PI Richard cabling for CPS, status PI 202 00:03:17
4 HEPI Hugh CS HEPI check - all down 202 00:03:24
5 Sprague Joe check all buildings 202 00:03:26
6 dust monitors JeffB modify pumps 202 00:03:32
7 ISC/IO/ALS Richard add sensors to all tables 202 00:03:33
8 TCSY Alistair startup TCSY laser 202 00:03:33
9 LVEA crane Bubba shim rails - requires lift 202 00:03:54
Alarm Test:
There are small glitches very close to each second boundary in the ETMY drive signal and the ETMY SUS ad SEI rack magnetometers. In order to investigate the half-Hz combs in DARM (see alog 20790), I took an hour of data and folded it with a four-second period. If there is a repeated glitch at any multiple of this period, it should become far more visible. The result is that in several channels, there are glitches very near the boundary of each GPS second. The peak time of these glitches seems to be about 10 milliseconds after the start of the second. The glitch does not repeat identically every second. There is one shape in the first second, then one with an opposite polarity in the next second. The first two attached plots are for the SUS and SEI racks, which are shown with a 40-Hz zero-phase lowpass. The SUS has a narrow spike, and the negative spike is larger than the positive one. The SEI signal is more complicated. The ETMY L3 MASTER signal, which is the DARM output to the ESD, is shown with a 10 to 50 Hz bandpass. These glitches are more like sine-Gaussians, but the even and odd seconds still seem to have opposite polarities. There are more channels with similar glitches. We can make a more thorough investigation, and use more data and more times, to try to track down the origin of these glitches. Hopefully these glitches are responsible for the 0.5 Hz combs such that removing them will improve those.
J. Kissel
Tagging CDS in this entry.
I'd recently taken a look at the requested output of the ETMY L1 stage,
${IFO}:SUS-ETMY_L1_MASTER_OUT_*_DQ
and was interested to find ~1 [Hz] combs in the requested output. Though this isn't the 0.5 [Hz] combs that Andy mentions above, I think it's an excellent place to take the investigation further in a more focused manner -- with the point being that even the SUS's *requested* signal has a comb.
Attached is a 100-sec FFT ASD, of a typical, 1000 [sec] stretch of observation-ready data during the run (2016-01-04 04:00 UTC). Here, to give a feel for the physical amplitude of these signals: at 30 [Hz] the noise amplitude of one of these comb peaks is roughly 1e-3 [ct/rtHz] of requested DAC output, which corresponds to
1e-3 [ct/rtHz] * (20.0 [V] /2**18 [ct]) = 7.6e-8 [V/rtHz] @ 30 Hz ( * sqrt(2 * 0.01) = 1.1e-08 [V_pk])
Potentially verifiable / refutable Crack-pot Theories / Wild guesses:
- Perhaps there is some of this glitching in the inter-process communication (IPC) on the reflected memory (RFM) data transfer from the corner to the end station, that's only exposed for requested drives that have such a huge dynamic range? For whatever channels in the signal chain that are stored, can you reproduce the same combs by filtering those channels offline?
- Recall that the power supplies for the Hartmann Wavefront Sensor (HWS) were replaced some time ago, see Integration Issue 1062. Has anyone made a before-and-after comparison on this searched other sources for such combs in auxiliary channels? Perhaps forming a BruCo-like search where this UIM / L1 stage control signal is the response instead of DARM?
- Keith has already done a long-term study of the analog-to-digital converters (ADCs), looking for combs: see, eg. pg 44-49 (yeah!) of G1300997. He found no-such combs. Perhaps we should do a similar study on the digital-to-analog convert (DAC) side of things? I could also imagine a similar set up for a set of RFM channels that make the 4km journey along the arms.
In response to Jeff: The SUS-ETMY MASTER signal is just a filtered version of DARM, so if DARM has the comb so do the drive signals. I don't think that tells us where in the loop they originate. But you're right, this could be a digital problem or an electronics one involving something synched to GPS. Keith T. and Annamaria both suggested that the power supply of the timing fanout might be involved. That of course can be perfectly synched to the GPS second. Annamaria showed me an ADC in the L1 corner station being used as a temporary monitor of one of these power supplies. That signal (attached) jumps downward one second and upward the next, matching what we see in the magnetometers and DARM. Could we check if there's such an effect at the H1 Y-end?
Note that the small glitches in Andy's post are exactly synchronized to GPS; this makes coupling to many power supply glitches (HWS, or trickle chargers for magnetometers, etc.) an unlikely source.
Note that at the time the DAQ was restarted during Tuesday's maintenance the DAQ dataconcentrator and both frame writers had been running continuously for 55 days 20 hours with no problems.