Displaying reports 56701-56720 of 83192.Go to page Start 2832 2833 2834 2835 2836 2837 2838 2839 2840 End
Reports until 15:52, Monday 16 May 2016
H1 CDS (GRD)
david.barker@LIGO.ORG - posted 15:52, Monday 16 May 2016 (27227)
guardian EPICS CA problems this morning, details

T.J, Ed, Jim, Patrick, Dave:

The ALS_XARM node was reporting no connection to the epics channels H1:ALS-X_LOCK_ERROR_FLAG and H1:SYS-MOTION_Y_SHUTTER_A_OPEN. This was preventing any further IFO locking. Logging into h1guardian0 as user controls and issuing caget commands, we were able to get H1:ALS-X_LOCK_ERROR_FLAG's value, but not H1:SYS-MOTION_Y_SHUTTER_A_OPEN (reported data "not complete"). Both caget's complained that two IOCs were serving the data, the h1slow-h1fe epics gateway (spans between the slow controls 10.105 network and the front end 10.101 network) and the IOC itself, h1ecatx1. On the other hand, caget on a control room workstation could get both channels.

Since restarting the end-x Beckhoff slow controls system seemed the most impact, we followed the sequence:

reload ALS_XARM node

restart ALS-XARM node

restart h1slow-h1fe gateway

reboot h1guardian (then we remembered that we had decided that power cycles were preferred)

power-cycle h1guardian0

restart h1ecatx1 epics IOC*

* Patrick reminded us that the Beckhoff IOC could be restarted without the PLCs needing a restart, so this was far less impactful than anticipated. This ultimately fixed the problem.

h1guardian0 was scheduled for a reboot tomorrow to load patches, so this reboot was brought forward one day.

We still don't know why h1ecatx1 IOC was accepting all CAGET commands from the control room network caget's, only partial caget's from h1guardian0, and none from the guardian nodes.

postscript: when we power-cycled h1guardian0 we kept the epics gateway off to ensure CA clients would connected directly to h1ecatx1 and not use the epics-gateway. The nodes did not start up the same way as the previous reboot, with more nodes not connecting the channels. The h1ecatx1 channels were in the list, but so were channels on h1hwsmsr (a 10.105 machine) and h1seib3 (on the same front end network). After about 15 minutes TJ reported these channels reconnected, and the only thing we had done was to restart the epics gateway, or perhaps wait long enough.

H1 SUS
jeffrey.kissel@LIGO.ORG - posted 15:35, Monday 16 May 2016 - last comment - 13:39, Tuesday 17 May 2016(27223)
Model Prep for Individual Coil Driver Switching of HSTS M3 Stages
J. Kissel, B. Weaver
Work Permit: #5880
FRS Ticket: #5489
Integration Issue: #FRS 5506
ECR: E1500045

As a band-aid solution to the ganged-coil-driver-state-switching-lock-losses-during-acquisition, recently identified to be PRM (see LHO aLOG 27158), we're going to modify the M3 stage of the HSTS's library part to have individual control over coil switching similar to what was done with the M2 stage of the Beam Splitter some time ago (see E1500045).

Because of the breakdown of suspension-type library parts, we must changes to
/opt/rtcds/userapps/release/sus/common/models/
HSTS_MASTER.mdl
MC_MASTER.mdl
and instead of copying the un-library-parted BSFM_MASTER M2 COILOUTF bank, we created a new library part to be copied into the HSTS_ and MC_MASTER_M3 stages, called
/opt/rtcds/userapps/release/sus/common/models/FOUROSEM_COILOUTF_MASTER_INDIVIDUAL_CTRL.mdl
We've confirmed that these updated models compile, and they've been committed to the userapps repo.

We'll work on MEDM screens tomorrow morning, once we've compiled the updated models and we have new channels. There will be some settings (namely the coil driver state settings) that will be lost from the channel name switch, so we'll make sure to replace those in the SDF system accordingly. We'll also make our best attempt EVER at preserving the alignment of these SUS after the restart.

Images attached to this report
Comments related to this report
betsy.weaver@LIGO.ORG - 15:39, Monday 16 May 2016 (27225)

Here are some screen captures of the HSTS BIO and EULER2OSEM matricies which we will be incorporated into new MEDM screens tomorrow.

Images attached to this comment
jeffrey.kissel@LIGO.ORG - 13:39, Tuesday 17 May 2016 (27239)
J. Kissel, B. Weaver, [J. Betzwiser remotely]

As usual, we were only able to find bugs in the above SUS model changes for individual coil driver state switching after we'd finished making the MEDM screen modifications. 

After checking the coil driver compensation functionality after the model changes, we were immediately reminded that the STATE MACHINE logic for those HSTSs stages which have modified TACQ drivers is different from those stages without modification. The beam splitter M2 stage -- which we'd used as an example for the new infrastructure -- which uses an unmodified TACQ driver, where as the recycling-cavity HSTS's (PRM PR2 SRM and SR2 -- at H1 at least) have both their M2 and M3 TACQ drivers modified for extra actuation range. Once we reminded ourselves which suspensions had which drivers modified, we found it best to create a new library part in the 
/opt/rtcds/userapps/release/sus/common/models/STATE_BIO_MASTER.mdl
library capable of individual coil state switching library that uses the correct compensation switching code for the modified driver. It's obscure, but it's as simple as copying over the existing INDIVIDUAL_CTRL block, then changing 
inline TACQ $SUS_SRC/CD_STATE_MACHINE.c
to
inline TACQ_M2 $SUS_SRC/CD_STATE_MACHINE.c
in each of the cdsFunctionCall block.

I've attached screenshots of the updated STATE_BIO_MASTER library and the innards of the parts inside. Note that I've 
- changed the names of the library blocks associated with the TACQ drivers at the top level to better differentiate between the differences. 
- made the inline function call visible under the cdsFunctionCall block (right-click > Properties ... > Block Annotation tab > double click on "Description" block property token to add to the list of things for annotation > hit OK) so that the difference in the code call is obvious without digging into the properties.

These new library blocks (and the renamed existing blocks for the unmodified driver) were hooked up to the HSTS master models according to their arrangement of TACQ driver modifications: [For H1 ONLY see E1400369 and E1200931]
HSTS_MASTER   MC1, MC3               No TACQ Driver Mods

MC_MASTER     MC2                    M2 stage driver modified

RC_MASTER     PRM, PR2, SRM, SR2     Both M2 and M3 stages modified

Also, for convenience, remember you can find details of the modification in L1200226, but in summary, the modified driver is a factor of 10 stronger than an unmodified driver (and there's no longer frequency response in the output impedance network, which is why the digital compensation has to change).
Also, also, the state machine diagrams that outline how the digital compensation works with the analog driver state can be found in T1100507

These updated models have been committed to the sus/common/models/ directory of the userapps repo, so one needs not do anything different than the above update instructions to receive the bug fix. Since the distinction between whether no, one, or two of an HSTS's TACQ drivers are modified is made at the top-level of the model, i.e. by which of the HSTS_, MC_, or RC_MASTER blocks are used, there shouldn't be a need to change any top-level stuff at LLO to receive this model update. Just update that sus/common/model/ corner of the repo, and recompile, re-install, and re-start.
Images attached to this comment
H1 CDS (DAQ, SUS)
david.barker@LIGO.ORG - posted 15:34, Monday 16 May 2016 - last comment - 17:52, Monday 16 May 2016(27224)
new SUS PI models installed, DAQ restarted

Tega, Ross, Jim, Dave:

We installed new code for the models h1susitmpi, h1susetmxpi and h1susetmypi. The new code required a DAQ restart, which TEAM-PI obtained permission from TEAM-COMMISSIONING for this afternoon.

The purpose for the change was to make the code more efficient and claw back some CPU time on the h1susitmpi model which was running long (15-16uS for a 64k model). This was successful, it is now running in the 9-10uS range. ETMX is unchanged at 7uS, there is hint that ETMY is running one micro-second longer from 3uS to 4uS.

Comments related to this report
tega.edo@LIGO.ORG - 17:52, Monday 16 May 2016 (27231)
Tega, Ross, Jim and Dave.

Detailed changes made to the PI models.
PI_MASTER:
1. Added OMC_PI_MODE library part.
2. Replicate the functionality of the "SUS_PI_DAMP" block in "SUS_PI_COMPUTE" and "ETM_DRIVER". 
3. Removed the down-conversion blocks from SUS_PI_DAMP to avoid unnecessary computation in the h1susitmpi model. 
4. Renamed OMC_PI as OMC_PI_DOWNCONV to better reflect functionality.
5. Rearranged the library parts so that Simulink blocks related to the OMC_DCPD are on the right whilst blocks that process the QPD data are on the left. 

h1susetmxpi:
1. Replace the ETMX_PI_DAMP block with the new library parts: SUS_PI_COMPUTE (block name: ETMX_PI_DAMP) and ETM_DRIVER (block name: ETMX_PI_ESD_DRIVER). 
2. Moved the down-conversion blocks out of ETMX_PI_DAMP into a single block at the top of the model.
3. Added OMC_DCPD data into the PI control path using a switch that takes either the processed signals from the QPDs (ETMX_PI_DAMP)  or the processed signals from the OMC_DCPDs(ETMX_PI_OMC_DAMP)". 

h1susetmypi:
1. Replace the ETMX_PI_DAMP block with the new library parts: SUS_PI_COMPUTE (block name: ETMY_PI_DAMP) and ETM_DRIVER (block name: ETMY_PI_ESD_DRIVER). 
2. Moved the down-conversion blocks out of ETMY_PI_DAMP into a single block at the top of the model.
3. Changes needed to process OMC data are on hold for now.

h1susitmpi:
1. Updated the links for ITMX_PI_DAMP and ITMY_PI_DAMP blocks to the new library part: SUS_PI_COMPUTE.

The attached images show the before & after snapshots for each model.
Images attached to this comment
H1 TCS
nutsinee.kijbunchoo@LIGO.ORG - posted 14:37, Monday 16 May 2016 (27216)
HWSMSR crashed Sunday 3 AM

Now restored. Restarting the h1hwsmsr computer did the trick.

 

Here's more detail on what happened:

I came in this morning noticed a connection error on DIAG_MAIN. Opened up the HWS ITMs code and saw that every channel went white. There was no restart in the morning that could have affected TCS. I powercycled the msr computer and was able to rerun the code.

Dave also mentioned that various computer crashes between 2-4 AM local was normal during ER6. He didn't see this problem during O1. We also looked at the memory and CPU usage. Nothing is overloaded

Images attached to this report
H1 CDS
david.barker@LIGO.ORG - posted 13:32, Monday 16 May 2016 - last comment - 13:54, Monday 16 May 2016(27221)
weekend timeline

we had some system problems over the weekend, here is a summary of the time-line. (These would appear to be unrelated events)

The last issue caused Guardian problems with the ALS_XARM node. We did the following to try to fix it:

The power up of h1guardian sans epics-gateway gave even more CA connection errors, with HWS IOC and h1seib3 FEC. These were very confusing, and seemed to go away when the h1slow-h1fe epics gateway was restarted which added to the confusion. We need to reproduce this error.

After Patrick restarted the h1ecatx1 IOC the guardian errors went away.

Comments related to this report
jameson.rollins@LIGO.ORG - 13:54, Monday 16 May 2016 (27222)

Rebooting the entire guardian machine just because one node was having a problem seems like extreme overkill to me.  I would not recommend that as a solution, since obviously it kills all other guardian processes, causing them to loose their state and current channel connections.  I don't see any reason to disturb the other nodes because one is having trouble.  Any problem that would supposedly be fixed by rebooting the machine should also be fixed by just kill and restarting the affected node process.

The actual problem with the node is not specified, but the only issue I know of that would cause a node to become unresponsive and immune to a simple "guardctrl restart" is the EPICS mutex thread lock issue, which has been reported both at LLO and LHO, and both with solutions that don't require rebooting the entire machine.  Presumably the issue being reported here is somehow different?  It would be good to have a better description of what exactly the problem was.

H1 CDS
patrick.thomas@LIGO.ORG - posted 12:56, Monday 16 May 2016 - last comment - 13:00, Monday 16 May 2016(27219)
Error on h1ecatx1 IOC
See attached screenshot.
Images attached to this report
Comments related to this report
patrick.thomas@LIGO.ORG - 13:00, Monday 16 May 2016 (27220)
Restarted the IOC.
LHO VE (CDS, VE)
patrick.thomas@LIGO.ORG - posted 12:37, Monday 16 May 2016 (27218)
Forced PT100 Cold Cathode on
Forced PT100 Cold Cathode on in Beckhoff on h0vacly per Chandra's request (see attached). It is now reading ~1.01e-07.
Images attached to this report
H1 SEI (SEI)
travis.sadecki@LIGO.ORG - posted 10:01, Monday 16 May 2016 (27215)
HEPI Pump Trends - past 45 days

Attached are HEPI Pump Trends for the past 45 days.  To my untrained eye, I don't see any egregious excursions in pump pressures.  SEI folks should review.

This completes FAMIS Request 4520.

Images attached to this report
LHO VE
chandra.romel@LIGO.ORG - posted 09:46, Monday 16 May 2016 - last comment - 09:50, Monday 16 May 2016(27212)
HAM11 annulus IP
Labeled HAM11 annulus IP (physically on HAM12) fell out of scale at around noon yesterday. 

Labeled HAM12 annulus IP (physically on HAM11) needs to be replaced. i.e. HAM11 annulus IP is working hard

https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=26804
Images attached to this report
Comments related to this report
chandra.romel@LIGO.ORG - 09:50, Monday 16 May 2016 (27214)
Diagonal pressure continues to trend down
Images attached to this comment
LHO VE
chandra.romel@LIGO.ORG - posted 09:20, Monday 16 May 2016 (27209)
CP3 overfill
9am local

1/2 turn on LLCV bypass --> took 22 seconds to overfill.

Lowered LLCV back to 20% (from 21%). Hot weather last week likely cause of long overfill times last Wed. and Fri.

*watch exhaust pressure after tomorrow's Dewar fill

Images attached to this report
H1 General
edmond.merilh@LIGO.ORG - posted 08:42, Monday 16 May 2016 - last comment - 09:21, Monday 16 May 2016(27208)
Monday Morning Meeting Minutes

SEI - No major maintenance plans scheduled. Ongoing tweaking with BRS.

SUS - model changes for HSTS scheduled for tomorrow.

VAC - GV measurements and manual CP3 overfill scheduled for tomorrow

CDS - Cables to be pulled for ITM ESD tomorrow. Auto restart of workstations to take place tomorrow morning.

PSL - the team sees no pressing reason to go into the enclosure at this time other than to do some DBB aligning tomorow, possibly.

Comments related to this report
chandra.romel@LIGO.ORG - 09:21, Monday 16 May 2016 (27210)
*CP3 Dewar fill from Norco truck tomorrow
H1 General
edmond.merilh@LIGO.ORG - posted 08:24, Monday 16 May 2016 (27206)
Shift Summary - Day Transition
TITLE: 05/16 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 0.0Mpc
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    Wind: 4mph Gusts, 1mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.08 μm/s 
QUICK SUMMARY:
TITLE: 05/16 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 0.0Mpc
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    Wind: 4mph Gusts, 1mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.08 μm/s 
QUICK SUMMARY:
 
H1 PSL (PSL)
peter.king@LIGO.ORG - posted 07:10, Monday 16 May 2016 - last comment - 08:10, Monday 16 May 2016(27204)
laser status
The laser was off this morning.  The chiller indicated that there was a "Flow sensor 1" error.
Comments related to this report
peter.king@LIGO.ORG - 08:10, Monday 16 May 2016 (27205)
Looking at the data from the various flow sensors, maybe, just maybe, the problem is with the flow sensor attached
to the auxiliary circuit (which monitors flows to the power meters ...).  The attached plot seems to imply that the
flow to the power meters dropped before the crystal chiller flow declined.

    Would need to check that the power meters are really attached to this part of the cooling circuit because the
diode chiller was running this morning.

For reference the cooling system information is under
https://dcc.ligo.org/DocDB/0067/T1100373/002/LIGO-T1100373-v2%20Coolant%20system%20operating%20and%20maintenance%20manual.pdf
Images attached to this comment
LHO VE
chandra.romel@LIGO.ORG - posted 14:15, Friday 13 May 2016 - last comment - 09:22, Monday 16 May 2016(27181)
CP3 overfill
1:30pm local

1/2 turn open on LLCV bypass --> took 33:17 min. to overfill CP3. 

Raised LLCV from 20% to 21%.
Comments related to this report
chandra.romel@LIGO.ORG - 09:22, Monday 16 May 2016 (27211)
Temp induced
Images attached to this comment
chandra.romel@LIGO.ORG - 08:05, Saturday 14 May 2016 (27197)
CP3 Dewar is being filled this Tuesday so LLCV may need to be lowered again.
H1 SUS
sheila.dwyer@LIGO.ORG - posted 00:16, Friday 13 May 2016 - last comment - 15:45, Monday 16 May 2016(27158)
coil driver switching causing locklosses

It looks like we need to do something about the triple coil drivers that we switch in lock, especially PRM M3.  We have lost lock a good fraction of the times that we have tried to switch in the last month or so.  Screenshot is attached, I also filed an FRS ticket hoping that someone might make an attempt to tune the delays while people are in the PSL tomorrow morning.  FRS ticket #5489 

Images attached to this report
Comments related to this report
rana.adhikari@LIGO.ORG - 12:17, Friday 13 May 2016 (27178)ISC, Lockloss

Is the DAC spectrum post-switch using up a large fraction of the range? If the noise in the PRC loop has change a little bit it could make this transition less risky.

sheila.dwyer@LIGO.ORG - 12:25, Saturday 14 May 2016 (27198)

Here's the PRM drive from last night's lock, in which we just skipped the PRM M3 BIO switching leaving the low pass off (BIO state 1). It seems like we should have plenty of room to turn the low pass on without saturating the DAQ.  

Images attached to this comment
jeffrey.kissel@LIGO.ORG - 15:45, Monday 16 May 2016 (27226)
FRS Ticket #5489 closed in favor of a long-term fix Integration Issue #5506 (which is now also in the FRS system) and for an eventual ECR. Work permit #5880 indicates we'll install the fix tomorrow. See LHO aLOG 27223.
Displaying reports 56701-56720 of 83192.Go to page Start 2832 2833 2834 2835 2836 2837 2838 2839 2840 End