Modification of ISS second Loop Board to reduce the Gain

SudarshanK, GabrieleV, PeterK.

The modifications was made on the ISS second loop power stabilisation board to change the variable gain represented by  H1:PSL-ISS_SECONDLOOP_GAIN form 0 to 40dB to -11 to 31dB. This is on top of the addition 20dB gain that was added and reported in alog 14291, and controlled by H1:PSL-ISS_SECONDLOOP_ADD_GAIN.

We also made some modifications on the Guardian scripts that engages the second loop. Now the second loop can be engaged from the IMC guardian window although some fault checking will need to be implemented.

The in-loop and out-of-loop RPN measurement with the loop closed, gain set at maximum, boost and integrator engaged is attached below. The loop performance looks on-par with our past measurements, about 1E-8 at 10Hz.

~1110 hrs. local -> Valved-out HAM1 turbo

removed and readded 8 channels in Conlog

For some reason these were not reconnecting, so I removed and readded them. They reconnected immediately after.

H1:GRD-IMC_LOCK_LOGLEVEL
H1:GRD-IMC_LOCK_MODE
H1:GRD-IMC_LOCK_NOMINAL_S
H1:GRD-IMC_LOCK_REQUEST
H1:GRD-IMC_LOCK_REQUEST_S
H1:GRD-IMC_LOCK_STATE_S
H1:GRD-IMC_LOCK_STATUS
H1:GRD-IMC_LOCK_TARGET_S

~0925 hrs. local -> Started HAM1 ion pump

HAM1 ion pump pumping in parallel with turbo pump for the time being -> My plan is to demonstrate that the ion pump can pump solo for 2 hours at the current pressure (3.5 x 10-6 torr) then to consult with Daniel S. about the cost/benefit of the various options for the weekend

Morning Meeting Minutes

The days of the 08:30 meeting will change to Monday, Tuesday, and Thursday. No change to the start time. This is being done to coordinate the various 3IFO and maintenance activates with the needs of the commissioning team.
 
The normal Tuesday 08:00 to 12:00 maintenance window will remain, with no changes at this time.
 
Seismic – Closed the position loops on HAM1. HAM1 HEPI is unlocked. Hugh is working on grounding/noise issues with HEPI pump controller at End-X.

Suspensions – Running transfer functions for acceptance review. General 3IFO work.

IO – Dave & Elli are working on taking SRC measurements.

OpLev – Stuart and Jason are B&K testing the ITM-Y OpLev pier. 

Vacuum – HAM1 is on the turbo pump. The HAM1 ion pump is being commissioned.     

The green X light is likely clipping at the first TMS mirror

Sheila, Kiwamu,

Yesterday, Keita suggested us looking for any obvious sign of clipping in the ISCTEX table or some TMS mirrors (alog 16923) just by eyes (of course with goggles). So, we went to the end X station and inspected various places.

There was one mirror which showed quite bright scattering. We think that this is the one clipping the green light.

If we are lucky, maneuvering the angles of TMS and input PZTs may be able to fix the issue. Otherwise this potentially would be one of the tasks for the next vent.

The pictures of the TMX mirrors can be found in ResourceSpace (https://ligoimages.mit.edu/?c=1568)

 

---

(Details)

We started from the optics on the table. We did not find a bright scattering although there were many dim scattering all over the place because it is green light. We fully opened up two irises that were in front of the HWS 2" mirror and the one after it just in case. Then we checked the mirrors on the periscope which were not easy to look at directly by eyes and sensor card. We used a camera to remotely peek these mirrors. The beam spot on each of these mirror looked fine and we did not see a bright scattering off of them.

We then tried looking for any bright mirrors on the transmission monitor table (TMS). We used a camera to peek the mirrors. According to the pictures we took, the first mirror on TMS (M8 in D1201457-v2) showed a bright scattering off of it. It is hard to say where exactly the incident beam hits, but it seems that the beam is too low. On the other hand, it is not easy to tell about the horizontal off-centering. Note that since there was a risk of accidentally block the beam and trip the PZT pointing control, we disabled the loops with the offsets held at the output. Also the ETM was miasligned at this momenet in order for Elli and Dave to do the signal-recycled Michelson test. The way we identified M8 is that we could see another mirror closely behind M8, which we think is M9. Also some pictures showed another mirror far left from M8 and we think it is M10 as the mirror holder is mounted in a opposite direction. Also, some pictures showed another bright spot which is a known scattering point from the vacuum view port.

Better Len2Pit decoupling of ETMX L1 with new top damping

Summary:

Somebody asked for a better L2P from ETMX L1 stage, so I made an attempt at making one.

With new top mass damping that was described in alog 16895 without OL damping, the measurement was done, filter was made, nothing was tricky, no hand tuning, no nonsense unstable poles, it just works.

But of course when OL damping is on, this nice L2P decoupling is broken (because the nice thing was made without OL damping).

I can probably make similarly nice decoupling filter for when the OL damping is on, but the question is exactly when such nice decoupling is necessary. OL damping on? Off? Both?

The ETMX is left with old configuration.

Details:

EX L1 drivealign L2L to oplev P, and drivealign L2P to oplev P were measured, and the latter was divided by the former. In both of these measurements, uniform noise from 0.1 to 1Hz was used and the amplitude was set so each coil outputs several thousands counts or so.

The resulting inversion function between 0.1 and 1.14Hz was fitted using happyvectfit, discarding any bad coherence data. No tuning was done except for selecting the measurement data and setting the fit order to 8.

The resulting L2P filter looks much nicer, Qs are lower,  and in general it makes sense more than the ancient filter that was eventually abandoned (first attachment).

The second attachment shows the step response of ETM oplev (blue) when a large length drive is applied to the L1 stage (brown).

Left panel shows that the step response was about an order of magnitude smaller with the new L2P filter than the old flat gain filter that is used these days.

Right panel shows that when the OL damp is on, this nice reduction is gone, both the flat filter and the new filter are about the same, they are larger than "new" in the left panel.

Note that I had to make a new oldamp filter for this, as the old OLdamp filter is incompatible with the new M0 pit damping.

more guardian epicsMutex pthread_mutex_unlock errors encountered

The following nodes have reported the epicsMutex thread error:

epicsMutex pthread_mutex_unlock failed: error Invalid argument
epicsMutexOsdUnlockThread _main_ (0x2999f20) can't proceed, suspending.

node and log file:

ISI_ITMX_ST2/@4000000054e3b1f036d0862c.u
IMC_LOCK/@4000000054efab210f2ebddc.s
SUS_PRM/@4000000054eceebe32bce24c.u
SUS_ETMY/@4000000054eceebe20a258e4.u

Still unclear why this is happening.  It happened most recently on IMC_LOCK yesterday, and the first known incident was with ISI_ITMX_ST2 on February 16, while I was testing the new guardian release before the upgrade.

guardian upgrade overview

All guardian systems have been successfully upgraded to the latest release

This is a long overdue update on the guardian upgrade performed last week.  The current installed versions are:

• guardian: r1373
• cdsutils: r441

A "final" bug fix patch was applied during the Tuesday 2/24 maintenance period, after which the guardian machine (h1guardian0) was rebooted.  All nodes recovered without issue.  There have been a couple of small issues that I'll note below.

RELEASE NOTES

OP/MODE split

The functionality of the MODE switch has been split into three independent interfaces:

• OP: The operational code execution state, can be set to one of three values:
  • EXEC: primary execution mode during which usercode is continually executed
  • PAUSE: stop executing user code.  Any currently running state method will continue until completion
  • STOP: kills the worker process immediately,

• MODE: The execution model
  • EXEC: primary graph traversal model.  Node continually tries to reach REQUEST state
  • MANAGED: node "STALLS"  after jumps and waits for a new REQUEST in order to continue.  Used during intra-node management.
  • MANUAL: direct state execution, graph is ignored (see below)

• LOAD: momentary switch set to 'True' to prompt a user code reload

RELOAD improvements

Code reload now happens seamlessly in the background, without interupting the current state code execution at all.  The current state is no longer interupted or restarted.  (triggered by setting the LOAD momentary switch to 'True')

The only known limitation occurs when certain changes to the currently running state are loaded.  If the node is currently in the RUN method of a state and the new code references an attribute or variable that was expected to have been set in MAIN, you will encounter an exception.  You should be able to bypass this problem by re-requesting the current state, which causes the current state to be re-executed from the beginning (i.e. MAIN).

request any STATE in the graph

The REQUEST interface now allows for requesting any state in the system.  "Requestable" states are now only used to populate the REQUEST drop-down menu on the guardian MEDM interfaces. 

A new STATES MEDM screen, accessed via the "all" button next to the REQUEST drop-down or via "guardmedm --states ...", allows for selecting any state in the system. 

The buttons are colored the same as states in the graph.  The "targets" to the right indicate the current STATE (inner), REQUEST (middle), NOMINAL (outer).

The system handles state requests exactly the same, regardless if the state is requestable or not.  In EXEC mode, guardian will follow the graph to the requested state, and hold there once it arrives.

NOTE: this is intended only as an aid to commissioning/debugging, so that intermediate states can be requested without having to modify the code to add/remove states from the request list.  However, we should continue to persue the same philosophy of only making "requestable" the states in which the system is intended to come to rest.  The REQUEST drop down menu is still intended to be the primary request interface.  This way it will always be clear which state are intended "final" states of the system.

MANAGER registration and overhaul

Managers now register themselves with their subordinates.  The current manager is recorded in the MANAGER channel, and a new display on the main MEDM control panel displays the current manager.

NOTE: if the user manually overrides this by selecting a different mode, or if another manager steals the node, the managing node will need to be told to go back through a state where it runs set_managed() to re-acquire the subordinates.

MANUAL MODE

When in MANUAL mode, the graph is completely ignored and the REQUEST state is immediately executed, dropping whatever else the node was doing at the time.

NOTE: This mode should only be used with caution by those who understand the system their controlling.  The graph is there to purposely constrain the dynamics of the system.  Ignoring these contraints can easily put the system into a bad state if you're not careful.

This mode can be accessed via the MANUAL button in the STATES MEDM screen.

"protected" states

A new "redirect = False" flag can be used on states that should never be left until they return True.  This is useful for FAULT states that should not be exited until the fault clears, even if another goto state is selected (e.g. DOWN).

weighted edges

Edges can now have weights, which can be used to break path degeneracies.  Guardian always chooses paths with the lowest total edge weight sum.

execution time recorded

The execution time of user code is now recorded in the EXECTIME (current execution time) and EXECTIME_LAST (execution time of last cycle) records.  Indicators of these values are now on the main control screen.

automatic code archiving

All usercode is committed to a per-node git code archive upon every restart or reload.  This gives us a complete record of execatly what code was running at any given point in time.

The archive root directory is:

• /ligo/cds/lho/h1/guardian/archive/

An integer representation of the archive git SHA1 commit id is recorded in the new ARCHIVE_ID channel, which is also displayed on the main and compact control screens:

compact MEDM control screen

A new compact control screen can be access via e.g. "guardmedm --compact SUS_ETMX":

setpoint monitoring

A new setpoint monitoring system has been added.  The ezca object now records all EPICS writes performed by the usercode.  If the "ca_monitor=True" flag is set in the module, guardian checks the current value of all setpoint channels to determine if they differ from where they were set by guardian:

If any differences are detected, the SPM box on the control screen goes yellow:

By clicking on the "SPM DIFFS" button, a screen will open showing the current list of differences:

For filter modules, as shown above, just the SWSTAT value is recorded FOR THE ENTIRE MODEULE, even if not all buttons are touched by guardian.   This allows guardian to cover the full state of filter modules, since the front end SDF monitoring can not be told to watch individual states only.  The SPM DIFF screen shows filter engaged differences as shown above.

NOTE: this feature is still experimental, and there are likely kinks that need to be worked out.  In particular, anything that "legitimately" sets values that have been touched by guardian outside of guardian, e.g. a subprocess script or a BURT restore, will cause the SPM to report differences.  This is kind of unavoidable, since there's no way for guardian to know if changes that occur outside of its purview are legitimate or not.

Subsystem commissioners should experiment with this feature and report any issues to me.

improved notifications

Notifications (USERMSGs) now cycle through the USERMSG display on the main control window.  There's also separate USERMSG MEDM screen where each individual message can be viewed.

Apple Magsafe2 adapters installed in control room using MagCozy tethers

Jim, Dave

We have installed 10 Apple Magsafe2 power adapters in the control room on the Magsafe power cords. To hopefully prevent these from wandering or getting lost, we followed Rana's advice and purchased MagCozy tethers for these, please see attached photographs.

A reminder, please do not use the magsafe cable which is bundled in the iMac to Thunderbolt Monitor cables. This cable is not long enough to use without pulling on the display port cable which has lead to breakages in the past. If we need more Apple power cords in the control room, please contact me and I will purchase some more.

👍

RFM IPC errors on end station SUS models fixed

Jim, Dave

First the good news, the RFM receive error rates on the end station SUS models have been fixed. Now the bad news, we don't know why what we did would fix the problem.

Prior to today's work, the RFM loop was such that the LSC front end communicated directly with the end station SUS front end. The loop continued on to the end station ISC, then back to the corner through OAF and ASC back to LSC. The error rate at both end station SUS systems for both channels being sent by the OMC model was about one every 5 seconds. The OMC sender was running at 13uS, so we should have zero errors at the end station. Indeed, we added the very same receivers in the PEM model (running on ISC front end) and verified a zero error rate there.

Working at EY, we first removed ISC from the RFM loop. The SUS errors went to zero (even though ISC was not inbetween LSC and SUS, we just shorted the loop by one node).

We reinserted ISC, and the SUS errors came back as before.

We then switched the position of SUS and ISC on the end station RFM switch, so now ISC is between LSC and SUS, and SUS error rate went to zero. We left EY in this configuration.

We then went to EX and repeated the above, with the same results. EX now also has SUS/ISC swapped from this morning.

To check once again that we know the direction the loop takes through the VMIC 5596 RFM switch, we repeated our DTS measurements (see X1 alog  for details). We are sure that prior to the change LSC communicated directly with SUS, and now ISC is inbetween.

After many hour of running we have not had a single RFM IPC error.

Operators should now closely investigate any red blocks on the CDS Overview MEDM. The only ones we would expect are the occassional ADC errors on certain IOP models.

We will try to reproduce the original SUS errors on the DTS to investigate why removing ISC or moving the ISC fixes SUS's problem.

Continuing HAM1 pumping/view port activities

Kyle 

Soft-closed GV5 and GV7 -> Resumed rough pumping of HAM1 -> New CDS vacuum signals were added for HAM1 pressure gauge pair (much thanks to Richard M., Filiberto C. and Dave B.) -> Associated vacuum computer(s) had to be rebooted to facilitate this -> CP1 level marginally impacted -> Switched pumping over to HAM1 turbo -> Connected leak detector (LD) and helium leak tested all view ports on HAM1 -> Pressure ~1 x 10-5 torr, LD baseline < 5 x 10-10 torr*L/sec, sprayed audible flow of helium for 20 seconds on air side of each window (through hole in VP protector) -> No LD response -> Disconnected LD and resume pumping with HAM1 turbo 

For tonight I am pumping HAM1 with its turbo backed by the HAM pump cart and am leaving GV5 and GV7 soft-closed.  

Added HAM1 vacuum gauge pair to VE EPICS VME slow controls

Richard, Kyle, Filiburto, Patrick, Dave

the h0vely EPICS system was modified to add the records to monitor/control the new Pirani/Cold-Cathode gauge pair connected to HAM1. HVE-LY:X0.db database file was created and added to the startup.h0vely file. The h0vely VME crate was rebooted and the new database was started. The vaccum MEDM screens were updated to show the new signals.

The H0EDCU_VE.ini file was updated and the DAQ restarted. The Pirani and Cold-Cathode calculated pressures in Torr are being recorded by the DAQ with the channel names HVE-LY:X0_100ATORR and HVE-LY:X0_100BTORR.

The vacuum overview MEDM screen which is being displayed on the CDS Web page was also updated for offsite monitoring.

Temporary disruption of commissioning

The realization that I had not completed all of the required steps following Tuesday's pumping of HAM1 has required that I interrupt commissioning.  How soon the Corner Station can be opened to the End Stations is under discussion.  Until then, I am running rotating pumps on and near HAM1.  I am sorry for the inconvenience.  

HEPI Pump System Service/Upgrade/Improvement at EndXBSC9 ETMX

Well, unlike at EndY, the pump station maintenance did not pay off as well.  Bottom line: remaining coherences seen in the Z and HP dofs are not reduced as much as they are at EndY after the maintenance.  Possible reason--while I found lots of Accumulators that needed charging (just like at EndY,) the explicit grounding of the power supply common legs did not make the Pressure Sensing noise better.  In fact something in this process has made the sensor noise worse; but it was't the adding of the ground.

Details:  First I added the explicit jumper from the common legs on the power supply to the supply ground plug but this had no observable affect on the striptool I was watching.  I figured it was doing no harm.  After Guardian brought ISI/HPI down to OFFLINE, I ramped the pressure down with the servo.  Then the motor was greased and Accumulators were charged: most Accumulator were essentially uncharged and two on the pump station were leaking.  I was able to play with the Schaeder Valve and stop the leaks (may be iffy.)  After the Accumulators were charged, the system was brought back online and Guardian reisolated with no problems.

See the first attachment for the second trends where the system was down and then back on.  This is where I first saw that the noise on the pressure sensor channel more than doubled.  What happened to cause the noise to change like this?  I plugged the ground wire in before bring the pump down and didn't see an noise increase.  Is the power supply flaky?  I did have to move the servo box around to access the Accumulators...

It is clear this is bad when you look at the second attachment with the Pump Pressure ASD: it is a few to several factors worse than Sunday(Reference traces.)  The remaining attachments are the coherences between the Pump Pressure and the HEPI L4C & ISI T240s.  The coherences have improved suggesting the Accumulator serve us well.  But the improvements are not as good as seen at EndY where the Pump Pressure Noise dropped  5 to x10.

HAM1 HEPI has controls installed, might work

Hugh, Jim

Short version: Control loops have been installed on HAM1 HEPI. It's a hack, it could get ugly. Or it could totally work.

Since unlocking HAM1 HEPI was discussed, Hugh and I decided to see if we could quickly run through the commissioning scripts. I tried to look at the transfer functions we took in June of 2013, but only got errors. When I look for the mat files, I found that the data have never been collected. Hugh reported having troubles, so we figured that our data collection scripts must have broken and Hugh was never able to get help to fix the problem. For now, we've copied the isolation filters from HAM2 HEPI (the HEPI contollers for HAM are all generic, should be okay). There other things missing: blend filters (which are 1  for IPS and 0 for the L4C, anyway), actuator symmetrization filters (which are just gains that should be close to 1) and L4C symmetrization filters (not needed). I've also installed the Z sensor correction filter, like we use on the BSC HEPI, which might allow us to get some inertial isolation between .1 and 1 hz. It would be best to have data to at least compare transfer functions to HAM2, but maybe we'll get lucky.

SDF Monitoring rolled-out for rest of Suspensions

This morning, Stuart and I used the SDF front end to take new SAFE.SNAP snapshots of the balance of the suspensions - recall, he had done the BS previously, see alog 16896.

 

Repeating his alog instructions on how to perform this:

- Transition the Suspension to a SAFE state via Guardian
- On the SDF_RESTORE MEDM screen available via the Suspension GDS_TP screen select FILE TYPE as "EPICS DB AS SDF" & FILE OPTIONS as "OVERWRITE" then click "SDF SAVE FILE" button to push a SDF snap shot to the target area (which is soft-linked to userapps).
- This safe SDF snapshot was then checked into the userapps svn:
/opt/rtcds/userapps/release/sus/h1/burtfiles/
M        h1susbs_safe.snap

 

Note, please don't use the BURT interface to take SAFE.SNAP snapshots anymore, as the SDF monitoring will become disabled.  All other snapshots are free to be taken via BURT, however.

 

Also, we found many alignment values not saved on IFO_ALIGN which, after confirming with commiss, we saved.

[Betsy W, Stuart A, Jamie R]

After taking safe SDF snapshots for the IM Suspensions, we found that Guardian had crashed for the IM2 and IM3 when we had attempted to transition them from SAFE to ALIGNED states. Oddly IM1 and IM4 still transitioned fine.

Upon checking the Guardian log it was apparent it was falling over for IM2 & IM3 immediately after reading the alignments from the *.snap files. Under initial inspection there was nothing obviously different or wrong with the IM2 & IM3 alignment files.

After contacting Jamie, he noticed an extra carriage return at the end of the IM2 & IM3 alignment files, which was causing issues for Guardian. Removal of the carriage return, and reloading Guardian rectified the problem.