Displaying report 1-1 of 1.
Reports until 14:28, Tuesday 09 June 2015
H1 CDS (SUS)
david.barker@LIGO.ORG - posted 14:28, Tuesday 09 June 2015 - last comment - 15:41, Wednesday 10 June 2015(19030)
status of 18bit DAC calibrations

We are seeing two issues with the autocal of the 18bit DAC cards (used almost exclusively by suspension models). The first is a failure of the autocal; the second is the autocal succeeding but taking longer than normal.

There are three calibration completion times: ~5.1S for unmodified DAC, ~5.3S for modified DAC, ~6.5S for modified long-running DAC

h1sus2b, h1sush34, h1susex, h1susey all have no DAC issues

h1sush2a: the third DAC is taking 6.5S to complete. This DAC is shared between PRM and PR3 models. First two channels are PRM M1 Right and Side. Last six channels are PR3 M1 T1,T2,T3,Left,Right,Side.

h1sush56: this has unmodified cards. 4th DAC is failing autocal

h1susb123: This one is strange:

On first autocal run after a computer reboot: 7th DAC failed autocal

On first restart of all models: 1st DAC failed autocal, 7th DAC succeeded

On second restart of all models: 1st DAC failed autocal, all others good (consistent restart behavior)

In all cases, 5th DAC card is running long for autocal (6.57S).

In the current situation with the 7th card now succeeding and the 1st DAC failing, the 1st DAC is driving ITMY M0 (F1,F2,F3,LF,RT,SD) and M0 (LF,RT)

Comments related to this report
jeffrey.kissel@LIGO.ORG - 15:41, Wednesday 10 June 2015 (19052)CDS, DetChar, ISC, SUS
I've been asked to translate / expand on the above entry, and I figure my reply was worth just commenting on the aLOG itself. If one person asks, many more are just as confused.

---------
We know that DetChar have seen Major Carry Transition or "DAC" [digital-to-analog converter (DAC)] glitches in some of the detectors interferometric signals, that have been pin-pointed to be from a few select stages of a few select suspensions (see e.g. LHO aLOGs 18983, 18938, or 18739).

To combat the issue, yesterday, we restarted all the front-end processes (including the Input Output Processor [IOP] process) on the four corner station SUS computers:
h1sush2a (controlling MC1, MC3, PRM and PR3 all in HAM2)
h1sush2b (controlling IMs 1 through 4)
h1sush34 (controlling MC2 and PR2 in HAM3 and SR2 in HAM4)
h1susb123 (controlling ITMY, BS, and ITMX in BSCs 1, 2 and 3 respectively)

Restarting the IOP process for any front-end who is coupled with an I/O chassis that runs 18-bit DAC cards performs the auto-calibration (autocal) routine on those 18-bit DACs, recalibrating the voltage bias between the (2-bit + 16-bit cards) of the 18-bit DAC to reduce Major Carry transition glitching. When the front-end computer is rebooted or power-cycled, the IOP process is started first, and therefore runs the auto-calibration routine as well. After autocalibration is complete, the user models for each suspension are started. 

Note that the other 3 SUS "control" computers,
h1sush56 (controlling SRM and SR3 in HAM5 and the OMC in HAM6 )
h1susex (controlling ETMX and TMSX)
h1susey (controlling ETMY and TMSY)
were NOT restarted yesterday, but have been restarted several times over recent history.

Each of these front-end or I/O chassis contains many DAC cards, because it (typically) controls many suspensions (as described above), hence the distinction between the card numbers in each front-end. Said differently, each DAC card has 8 channels -- because of initial attempts to save money and conserve space, the above mentioned corner station computers / IO chassis have some DAC cards that control OSEMs of two different suspensions. There's a map of which DAC card which controls which OSEM in the wiring diagrams; there is a diagram for each of the above mentioned computers / I/O chassis; each diagram, named after the chambers the I/O chassis controls can be found from the suspension electronics drawing tree, E1100337.

Recall that we have recently swapped out *most* of the suspensions' 18-bit DAC cards for a "new" 18-bit DAC card with upgraded EPROMs (see LHO aLOGs 18557 and 18503, ECR E1500217, and Integration Issue 945). This is what Dave means when he references "modified" vs. "unmodified" DAC cards. All DAC cards in h1sush56 remain unmodifed, where as the other corner-station DAC cards in all other SUS I/O chassis have all been upgraded.

Dave also describes that 18-bit DACs are used "almost exclusively by suspension models." The PCAL and PEM DACs are also 18-bit DACs. Every other fast DAC (used by TCS, SEI, PSL, ISC, etc.) is a 16-bit DAC which have not been found to have any issues with major carry transition glitching. Note that because the tip-tilt suspensions (the OMs and RMs, collectively abbreviated as the HTTS for HAM Tip-Tilt Suspensions) were purchased under ISC dollars, who made the decision to do this (as with many other things) differently -- they have 16-bit DACs. 

After we've restarted the IOP front-end process, we track its start-up log file to ensure that the auto-calibration was successfully completed. We've found that there are two "failure" modes to this auto-calibration process that get reported in these log files. 
(1) The IOP process log reports that the auto-calibration process has failed.
(2) The IOP process log reports that the auto-calibration process took longer than is typical. 
We don't know the practical result of either of these two failure modes means. This was brought up on the CDS hardware call today, but no one had any idea. I've pushed on the action item for Rolf&Co to follow up with the vendor to try to figure out what this means.

As for (2), 
- the typical time for an IOP process to complete the auto-calibration of one unmodified DAC card is 5.1 [sec]. 
- the typical time for a modified DAC card with the new EEPROM is 5.3 [sec].
- the atypical "errant" time appears to be 6.5 [sec]. 
But, again, since we have no idea what "running long" means, our only reason to call this a "failure mode" is that it is atypical.

So re-writing Dave's above status (which is a combined status that is the results of yesterday's IOP model restarts and other previous IOP model restarts) with the jargon explained a little differently and in more detail:
- h1sus2b, h1sush34, h1susex, h1susey all have no DAC auto-calibration issues. All DAC cards in these front ends / I/O chassis report a successful auto-calibration routine.
- h1sush2a: of the all the DAC cards in this I/O chassis, only the 3rd DAC (which is s controls the some of the TOP, M1 mass OSEMs of PRM and some of the TOP, M1 OSEMs of PR3) is suffering from failure mode (2).
- h1sush56: the 4th DAC is suffering from failure mode (1).
- h1susb123: this computer's IOP process was restarted several times. 
	- Upon the first autocal, which was performed as a result of rebooting the computer, 
		- the 1st and 7th DAC card suffered from failure mode (1), 
		- the 5th DAC card suffered from failure mode (2),
		- all other DAC cards reported success.
	- Upon the second autocal, which was performed as a result of restarting the IOP process, 
		- the 1st DAC card suffered from failure mode (1), 
		- the 5th DAC card suffered from failure mode (2),
		- all other DAC cards reported success.
	- Upon the thir autocal, which was performed as a result of restarting the IOP process again, 
		- the 1st DAC card suffered from failure mode (1), 
		- the 5th DAC card suffered from failure mode (2),
		- all other DAC cards reported success.
	The first DAC card is shared between the two TOP, M0 and R0 masses of ITMY. (note Dave's typo in the initial statement)

For previous assessments, check out LHO aLOG 18584.
Displaying report 1-1 of 1.