Daniel, Patrick, Matt
We did a little more rotation stage science today. The objective was to understand the remaining acceleration mystery, and to confirm that the resistor was helping. The on-screen EPICS values are the ones being used for acceleration and deceleration, and they now have an upper limit of 65000 (or 65s to reach the maximum speed of 100 RPM). Note that the on-screen velocity is in units of 0.01% of the maximum, so a value of 10000 gives the maximum speed of 100 RPM, and a value of 100 gives 1 RPM. (These RPM values are presumably for the motor, not the waveplate.)
We found that with the current firmware settings (which Patrick will append), the 50 Ohm resistor was not necessary, so we removed it. This means that other waveplates in the field need no hardware modification to achieve the 0.01dg accuracy we are seeing with this rotation stage.
The attached screenshot shows a move from 10W to 2W (velocity = 3000, acc = 6000, dec = 6000) and then from 2W to 10W (velocity = 300, acc = 60000, dec = 60000). Note that the higher values of acceleration and deceleration for lower velocities result in a smoother ride.
Current settings attached.
A couple of diagnostics features have been added to the code:
I reduced the calibration velocity from 3000 to 500. Driving too fast towards the home position seems to reduce its reproducibility. This test will have to be repeated by looking at the laser power.
The TCS rotation stages also got the new motor settings and can be tested. The TCS medm screens need to be updated as well. (why are they diffeerent?)
J. Kissel Continuing the work of Corey et al. have done cleaning up SDF files, (see LHO aLOG 26917), I've gone one level deeper to ensure that all snap files used in the target areas are soft links to locations in the userapps repo. There *is* a safe.snap for every front-end model / epics db, of which there are 129. Unfortunately, because they're human construction, there are less OBSERVE.snaps (112) and down.snaps (28). OBSERVE.snaps at least exist for every front-end model / epics db that existed during O1. However, weather station dbs, dust monitor dbs, and pi front-end models are new since O1, so OBSERVE.snaps don't exist for them. Further, down.snaps seem to have only been created for ISC models, the globally controlled SUS models, and the ISC-related beckhoff PLCs. We know the safe.snaps are poorly maintained, and sadly we haven't been in a configuration we'd call OBSERVE.snap worth in a long time, so they're also out of date. On top of all this, each subsystem seems to have a different philosophy about safe vs. down. Daniel, Sheila, Jamie, and myself were discussing this on Friday, we'd come to the conclusion that it is far too difficult to maintain three different SDF files. If the SDF mask is built correctly, then there should be no difference between the "down" and "safe" state. The inventors of the "safe" state are the SEI and SUS teams because they have actuators strong enough to damage hardware. As such, they've designed the front-end models such that all watchdogs come up tripped and user intervention is required to allow for excitations. So, as the model comes up, it's already "safe" regardless of it's settings. Of course, even though the IFO is "down" at that starting point, we still want the platforms to be fully isolated. So, in that sense, for the ISIs "down" is the same as "OBSERVE." And again, if all settings that change via guardian are correctly masked out, then "safe" is the same as "down" is the same as "observe" and you only need one file. So, eventually -- we should get back to having only one file per subsystem. But this will take a good bit of effort to make sure that what's controlled via guardian is masked out of every SDF, and vice versa, that what is masked out of SDF *is* controlled by guardian. The temporary band-aid idea, will be at least to make sure that every model's down is the same as it's safe. Because Corey et al. put a good bit of effort into reconciling the down and safe.snap files today, I've copied all of the down.snap's over to the safe.snaps and committed them to the repo. I've not yet gone as far as to change the safe.snap softlinks to point to the down.snaps, but that will be next. Anyways -- this aLOGs kinda rambling, because this activity has been disjointed, rushed, and sporadic, but I wanted to get these thoughts down and give an update on the progress. In summary, at least every safe, down, and OBSERVE.snap in the target area is a soft link to the userapps repo, and all of those files in the userapps repo are committed. More tomorrow, maybe.
Thanks for the write-up here! A couple of comments/notes:
1) Does every frontend really have a safe.snap? I thought I could not find some safe.snaps for some of the ECAT (i.e. slow control) frontends. Or is there a way for the SDF Overview medm to not display *all* SDF files?
2) If we manage to get to ONE SDF file, what' will we name it? Will we stay with "safe" since that's what the RCG calls out, or will we change it to a name more preferred (this was another subtle note I overheard you all discussing on Fri.)
I just noticed that both flow rate and laser temperature channel are clearly returning bogus values after the first Beckhoff restart this morning (~11:10 PT local -- both CO2X and Y). Flowrate reads 0 Gpm and laser temperature is 999 deg C for both CO2X and CO2Y. Chiller temperature is also showing weird behavior (CHILLER_OUT_GAIN_INMON). I went to check on the chillers and they are working fine (flow rate ~3 Gpm). I'm not sure what other channels are returning bogus number that TCS guardian uses to feed the chiller and ISS servo so I request those guardian nodes to be DOWN and turn off the CO2 laser for now until the issue is solved (hopefully tomorrow). I don't think it's a good idea to have them running when the temperature readback and chiller readback are bogus (although the interlock is probably looking at something else, since these values didn't trip the laser). Also we don't have the IFO at the moment.
The parameters for power and angle calculator also reset themselves to either 1 or 0 after SOME Beckhoff restart (see LASERPOWER_POWER_IN channel for example). I was told that TCS stuff is controlled by PCL3. Everytime this happens all the parameters have to be restored by hand. Not cool. So be mindful to check these parameters after every Beckhoff restart and don't just lase the laser (luckily when this happen the laser switch is turned off). It will immediately blast 5W out to the ITMs (whatever happened made the CO2 rotation stages moved too).
POPX: DC works, RF not.
DC responds to flashlight.
We tried to measure the transfer function from RF test input to the RF output on the feedthrough panel at ISCT1, but didn't obtain anything reasonable. We'll swap the head tomorrow.
As a side note, multiple-coax connectors are a pain to work with.
We discovered some build issues which meant we missed today's deadline to upgrade the CDS frontends and DAQ to RCG3.0. We are now ready for a second try tomorrow.
The main issue was tracked to a difference in the version of PERL running on the build machines, with h1build running 5.8.8 and l1build running 5.12.2. A condition statement in the perl module SUM.pm was performing a comparison with the return of a length() call with a null string ( eq "" ) which works with the later version but not with the older version. This line was changed to a numeric comparison ( == 0) which works for all versions.
Several other build issues:
Rolf has included the SUM.pm fix in the next tagged release RCG3.0.2
Betsy has some SUS changes she will install first thing Tuesday morning. After this we will do a clean build and install before shutting all systems down for the upgrade.
We inspected all the relevant optical surfaces that we have access to in the High Power Oscillator.
No damage was visible. A couple of small dust specks were found and either blown off or wiped off.
Using the bright green flashlight (thank you JeffB) we identified what looked like water spots on the output coupler and one of the quartz rotators.
They were removed by wiping with isopropanol.
We had a brief telecon with OlliP, MikeF, and MattH at LLO. The game plan for tomorrow is to propagate a low-power Front End beam through the HPO and look for alignment issues.
Then, to bring each laser head up at low power and look for optical damage (scattering centers) as the pump power increases.
We plan to continue with this work first thing tomorrow morning.
Completed the installation of the new dust monitors in the PSL enclosure this afternoon. All the new dust monitors for LHO are operational. The alarm levels have been set, so any alarms received at the operators station should be investigated as a true event. Last task is to clean up the supporting documentation.
1/2 open LLCV bypass valve, and the exhaust bypass valve fully open.
Flow was noted after 48 seconds, closed LLCV valve, and 3 minutes later the exhaust bypass valve was closed.
I have (finally) looked at the raw voltages for the AS 90 WFS for each quadrant. The attached screenshot shows the in-lock values from 26 Apr 2016. The largest value is about 1200 counts, while the smallest is around 130 counts.
The voltages at the input to the ADC are therefore between 80mV and 730mV, assuming 1638.4 cts/V. Both AS90 WFS are using the full 45dB of analog gain, so the voltages from the demod boards for the AS 90 WFS are between 0.4mV and 4mV.
I have taken dark noise spectra of each of the quadrants, but since we don't _DQ each quadrant individually, I will have to wait until next lock to get an in-lock comparison. Screenshot of the dark noise attached also.
400nV seems like a pretty small number, but it's still a factor of 10 or so more than the noise we expect from the output of the demod boards, so perhaps it's okay, although it's certainly not as awesome as it could be.
Status of H1: down for inspection of PSL / HPO
Activities:
- tomorrow the biggest Mantenance activities are
22:31UTC - Gerardo to CP3 to fill
22:52 UTC Gerardo back
~21:01 UTC I turned off the camera, frame grabber, then powercycled the computer (then turned the frame grabber and the camera back on). Only HWSX code is running at the moment. Things look good for now.
May 3 16:44 UTC Stopped HWSX code and ran HWSY code alone. HWSX code had been running fine since yesterday.
May 5th 18:20 UTC I noticed HWSY code stopped running. There has been many comuter and front end restart since I left it running so it was unclear what caused it to stop. I reran it again and going to leave it again for another day.
May 2 12:08:37 h1conlog1-master conlog: ../conlog.cpp: 301: process_cac_messages: MySQL Exception: Error: Data too long for column 'value' at row 1: Error code: 1406: SQLState: 22001: Exiting. Coincident with Bechoff restarts?
Restarted.
Restarted again. Same error.
This may be different, but we also ended up with a couple of corrupt autoburt files. There the problem seems to be that string values are not properly escaped. A carriage return character in a string will force a line break in the autoburt text file. A burt restore will then complain that the string is not terminated by double quotes.
Because we were focused on the QUAD-centric susaux model changes last month (alog 25313), we screwed up the DAQ channel lists for the HAM models (h1susauxh2, h34, h56) after a change to the master FOUROSEM library part. We turned the NOISEMONs for this library part into filter banks so they needed an _OUT in their channel names. I fixed these today for the 3 sus ham models and Dave recompiled. We have more work to do on these models as per existing ECRs. See attached for an example of how I've left these DAQ channel lists. Note, we still need to deal with the VOLTMONs and switch the T SIXOSEM library part Noisemons.
With Kissel, continued more cleanup work to remove VOLTMONs and add filterbanks to the NOISEMON and FASTIMONs of the h1susauxh2, 3h34, and h56 models. Will continue tomorrow morning to finish tweaking the DAQ channel lists of these. All under ECR E160003.
Before the RCG 3.0.1 Reboot (probably later today), there was an action to go through the SDF System to clean it up, check, or atleast note DIFFS. This is because when the RCG change occurs, ALL front ends will be rebooted! After a reboot, the frontends should check the safe.snap files and restore the systems to a "SAFE" state determined by these safe.snap files.
So this is a best effort to make sure we come back to a safe state, with no major surprises and possibly maintain all accepted changes we had.
SDF file checks were primarily addressed by Subsytem experts:
It is sounding like others are ACCEPTING diffs for their subsystems so their SDFs will look green (assuming they have the safe file selected on the SDF).
I picked a few subsystems to help lighten the load for others. Since I am not an expert on these subsystems, I only took snapshot images of all the DIFFs observed on SDF. SO, these will have some RED DIFFs!
PSL SDFs came up with these files as default on the SDF Overview: DBB(safe), FSS(safe), ISS(down), PMC(safe)
ISS came up with the down file, but it also has a safe file as well (the other ones do not have down files).
Did NOT ACCEPT any DIFFS since I am not familiar with channels here. Jason/Peter are out today so did not want to ACCEPT anything without them.
Took snapshot of:
CS ECAT came up with these files as default on the SDF Overview: PLC1(OBSERVE), PLC2(down), PLC3(OBSERVE)
All had OBSERVE files, but some did/didn’t have safe & downs.
Did NOT ACCEPT any DIFFs. Daniel/Patrick said that would be OK.
Took snapshot of:
EX ECAT PLC1(observe), PLC2(down), PLC3(observe)& PEM (safe)
The ECATs mainly had OBSERVEs (plc2 had a down though).
Took snapshot of:
EY ECAT PLC1(observe), PLC2(down), PLC3(observe)& PEM (safe)
The ECATs mainly had OBSERVEs (plc2 had a down though).
Took snapshot of:
Jenne and I have cleared out all of the SUS DOWN.snap SDF screens. Note, there are no DOWN files for the HTTS, IM, and SR3 SDF screens so we cleaned house on the SAFE.snap for these. All suses are sitting in the DOWN file with the exception of these 3 which are on SAFE. We have left red:
- Some benign diffs on the MC2 SDF because this SUS is misaligned (not a usual state for the SAFE.snap to be looking at)
- Some benign diffs on the IM SDF, commiss message, and MASTER ON/OFF - safe wants the masters to be OFF but we haven't set the sus actually to safe yet.
Other findings which we cleared out were:
- Lots of TRAMPS
- Some alignment settings
- The ITMX and ITMY L2_Drivealign P2L Gain tweek
- The ITMX R0 Opticalign Y offset
See snapshot attached of all the SDFs before we started (minus the IM which Cheryl worked through a bit for mostly alignment and TRAMPS).
I accepted the 2 CAL diffs. See screenshot for what they were.
H1 SEI was cleaned up at about 9:30 local this morning. Rich is currently working on some stuff on the BSC's, but we shouldn't lose too much at this point if it doesn't get captured before any model restarts.
Matt, Jenne
On Thursday (Apr 28) at about 6:20pm local, while Rick and Keita were in the PSL, Jenne and I made some measurements of the ISS first loop PDs. Rick told us that there was no light on the ISS PDs at the time of our measurement, so we were hoping to get a dark spectrum to compare with the ones that Sheila and I took on the 24th and to add to the ones that Peter took. I took a bunch of photos of the results, but later wasn't sure of the state of affairs for each photo, so I went out again today to get a better organized set of measurements. At present, there is no light on the ISS PDs, so this is a DARK noise measurement. The aim here is to get high-frequency (10kHz - 10MHz) noise measurements which are complementary to the ones taken by Even and Stefan on Friday using the digital system.
I started by cabling-up a readout system which included a fast o'scope, an SR785 for sub-100kHz spectra, and an RF analyzer for MHz spectra. To avoid loading the OP27 outputs with 50 Ohms, I made a Pomona box with a 4.7kOhm resistor and 33nF cap in series for the RF path. When loaded with 50 Ohms (by the RF analyzer) this gives -40dB of attenuation and a 1kHz high-pass. (The 100 Ohm output impedance of the ISS board or ISS PD doesn't cause problems.)
The first 2 pictures show the output of the PDA monitor from the ISS board. The same 36.6kHz rep-rate noise bursts that Sheila and I saw are present, though they don't look clipped the way they did when the system was running. In the RF spectrum, these bursts appear as a comb of lines spread from ~100kHz to ~1.5MHz with a 36.56kHz spacing. There was nothing interesting above 2MHz.
Photos 3-5 show the result of disconnecting the input. The noise bursts go away, but since most of the noise is above 100kHz, the SR785 spectrum looks pretty similar (just the peak at 36.6kHz is gone). The result was similar when I terminated the PDA input on the ISS board with 50 Ohms. This, and what follows, appears to exonerate the ISS servo board.
Photos 6-8 show the result of placing a 100Hz low-pass in the line coming from the PSL to the PDA input. The idea here is to attenuate the MHz content on the signal path by > 60dB so that we can see what is on the ground. The pomona box I used was something that Stefan made for his coil driver measurements: 1.6kOhm and 1uF. This may not be the optimal way of doing this, but if the RF were all on the signal side it would go away... which it did not. The RF is much smaller (factor of ~5), but still clearly visible in the time series and the RF spectrum.
Finally, photos 9 and 10 show the output of PDA directly from the PSL. While qualitatively similar to the monitor output shown in photos 1 and 2, it has more RF junk. The sub-100kHz spectrum also has a lot more content in this configuration, and while watching it I noticed that there were features moving around with time. (I made a video, but I can't post it to the log. Ask me if you want it.) This smells like RFI to me.
For the record, PDB looks similar, but with more high frequency noise and less low frequency noise. The first photo is from the servo board monitor, the second is looking directly at the signal from the PSL. (Compare to photos 1 and 9 above.)
Jenne, Rick, Peter, Matt
With Rick and Peter in the PSL, Jenne and I got a look at the noise coming from the ISS PDs without the PDs. That is, Rick disconnected the PDA cable from the detector output and terminated it with 50 Ohms. This had little effect on the mysterious 37kHz rep-rate noise bursts, so they seem to be getting picked up on the cable in the PSL. (We also tried a long cable that didn't go into the PSL, but instead was placed at various locations in the rack. We didn't find anything interesting.)
Also, turning off the lights and fans in the PSL enclosure had no effect on the noise bursts.