I have just discovered that some hang-ups in the calibration pipeline are due to dataValid errors in many CAL and PEM channels. a partial list of the channels that have errors (dataValid code 19373) are:

H1:PEM-CS_ACC_BEAMTUBE_MCTUBE_Y_DQ  H1:PEM-CS_ACC_BEAMTUBE_XMAN_Y_DQ H1:PEM-CS_ACC_BEAMTUBE_YMAN_X_DQ
H1:PEM-CS_ACC_BSC1_ITMY_X_DQ H1:PEM-CS_ACC_BSC1_ITMY_Y_DQ H1:PEM-CS_ACC_BSC1_ITMY_Z_DQ H1:PEM-CS_ACC_BSC2_BS_Y_DQ
H1:PEM-CS_ACC_BSC3_ITMX_X_DQ H1:PEM-CS_ACC_BSC3_ITMX_Y_DQ H1:PEM-CS_ACC_EBAY_FLOOR_Z_DQ H1:PEM-CS_ACC_HAM2_PRM_Y_DQ
H1:PEM-CS_ACC_HAM2_PRM_Z_DQ H1:PEM-CS_ACC_HAM3_PR2_Y_DQ H1:PEM-CS_ACC_HAM4_SR2_X_DQ H1:PEM-CS_ACC_HAM5_SRM_X_DQ
H1:PEM-CS_ACC_HAM6_OMC_X_DQ H1:PEM-CS_ACC_HAM6_OMC_Z_DQ H1:PEM-CS_ACC_IOT1_IMC_X_DQ H1:PEM-CS_ACC_IOT1_IMC_Y_DQ
H1:PEM-CS_ACC_IOT1_IMC_Z_DQ H1:PEM-CS_ACC_IOT2_INPUTOPTICS_Y_DQ H1:PEM-CS_ACC_ISCT1_REFL_Y_DQ H1:PEM-CS_ACC_ISCT6_OMC_X_DQ
H1:PEM-CS_ACC_LVEAFLOOR_BS_X_DQ H1:PEM-CS_ACC_LVEAFLOOR_BS_Y_DQ H1:PEM-CS_ACC_LVEAFLOOR_BS_Z_DQ
H1:PEM-CS_ACC_LVEAFLOOR_HAM1_Z_DQ H1:PEM-CS_ACC_LVEAFLOOR_XCRYO_Z_DQ H1:PEM-CS_ACC_LVEAFLOOR_YCRYO_Z_DQ
H1:PEM-CS_ACC_OPLEV_ITMX_Y_DQ H1:PEM-CS_ACC_OPLEV_ITMY_X_DQ H1:PEM-CS_ACC_PSL_PERISCOPE_X_DQ H1:PEM-CS_ACC_PSL_TABLE1_X_DQ
H1:PEM-CS_ACC_PSL_TABLE1_Y_DQ H1:PEM-CS_ACC_PSL_TABLE1_Z_DQ H1:PEM-CS_ACC_PSL_TABLE2_Z_DQ H1:PEM-CS_ACC_PSL_TABLE3_Z_DQ
H1:PEM-CS_ADC_4_29_2K_OUT_DQ H1:PEM-CS_LOWFMIC_LVEA_VERTEX_DQ H1:PEM-CS_MAG_EBAY_LSCRACK_QUAD_SUM_DQ
H1:PEM-CS_MAG_EBAY_LSCRACK_X_DQ H1:PEM-CS_MAG_EBAY_LSCRACK_Y_DQ H1:PEM-CS_MAG_EBAY_LSCRACK_Z_DQ
H1:PEM-CS_MAG_EBAY_SUSRACK_QUAD_SUM_DQ H1:PEM-CS_MAG_EBAY_SUSRACK_X_DQ H1:PEM-CS_MAG_EBAY_SUSRACK_Y_DQ
H1:PEM-CS_MAG_EBAY_SUSRACK_Z_DQ H1:PEM-CS_MAG_LVEA_INPUTOPTICS_QUAD_SUM_DQ H1:PEM-CS_MAG_LVEA_INPUTOPTICS_X_DQ
H1:PEM-CS_MAG_LVEA_INPUTOPTICS_Y_DQ H1:PEM-CS_MAG_LVEA_INPUTOPTICS_Z_DQ H1:PEM-CS_MAG_LVEA_OUTPUTOPTICS_QUAD_SUM_DQ
H1:PEM-CS_MAG_LVEA_OUTPUTOPTICS_X_DQ H1:PEM-CS_MAG_LVEA_OUTPUTOPTICS_Y_DQ H1:PEM-CS_MAG_LVEA_OUTPUTOPTICS_Z_DQ
H1:PEM-CS_MAG_LVEA_VERTEX_QUAD_SUM_DQ H1:PEM-CS_MAG_LVEA_VERTEX_X_DQ H1:PEM-CS_MAG_LVEA_VERTEX_Y_DQ
H1:PEM-CS_MAG_LVEA_VERTEX_Z_DQ H1:PEM-CS_MAINSMON_EBAY_1_DQ H1:PEM-CS_MAINSMON_EBAY_2_DQ H1:PEM-CS_MAINSMON_EBAY_3_DQ
H1:PEM-CS_MAINSMON_EBAY_QUAD_SUM_DQ H1:PEM-CS_MIC_EBAY_RACKS_DQ H1:PEM-CS_MIC_LVEA_BS_DQ H1:PEM-CS_MIC_LVEA_HAM7_DQ
H1:PEM-CS_MIC_LVEA_INPUTOPTICS_DQ H1:PEM-CS_MIC_LVEA_OUTPUTOPTICS_DQ H1:PEM-CS_MIC_LVEA_VERTEX_DQ
H1:PEM-CS_MIC_LVEA_XMANSPOOL_DQ H1:PEM-CS_MIC_LVEA_YMANSPOOL_DQ H1:PEM-CS_MIC_PSL_CENTER_DQ
H1:PEM-CS_RADIO_EBAY_NARROWBAND_1_DQ H1:PEM-CS_RADIO_EBAY_NARROWBAND_2_DQ H1:PEM-CS_RADIO_LVEA_NARROWBAND_1_DQ
H1:PEM-CS_RADIO_LVEA_NARROWBAND_2_DQ H1:PEM-CS_RADIO_ROOF1_BROADBAND_DQ H1:PEM-CS_RADIO_ROOF2_BROADBAND_DQ
H1:PEM-CS_RADIO_ROOF3_BROADBAND_DQ H1:PEM-CS_RADIO_ROOF4_BROADBAND_DQ H1:PEM-CS_SEIS_LVEA_VERTEX_QUAD_SUM_DQ
H1:PEM-CS_SEIS_LVEA_VERTEX_X_DQ H1:PEM-CS_SEIS_LVEA_VERTEX_Y_DQ H1:PEM-CS_SEIS_LVEA_VERTEX_Z_DQ
H1:PEM-CS_TEMPERATURE_BSC1_ITMY_DQ H1:PEM-CS_TEMPERATURE_BSC3_ITMX_DQ H1:PEM-CS_TILT_LVEA_VERTEX_T_DQ
H1:PEM-CS_TILT_LVEA_VERTEX_X_DQ H1:PEM-CS_TILT_LVEA_VERTEX_Y_DQ

H1:TCS-ITMX_CO2_ISS_IN_AC_OUT_DQ H1:TCS-ITMX_CO2_ISS_OUT_AC_OUT_DQ H1:TCS-ITMY_CO2_ISS_IN_AC_OUT_DQ
H1:TCS-ITMY_CO2_ISS_OUT_AC_OUT_DQ H1:TCS-ODC_CHANNEL_OUT_DQ H1:ODC-MASTER_CHANNEL_OUT_DQ

H1:CAL-CS_TDEP_REF_INVA_CLGRATIO_TST_REAL H1:CAL-CS_TDEP_REF_INVA_CLGRATIO_TST_IMAG
H1:CAL-CS_TDEP_REF_CLGRATIO_CTRL_REAL H1:CAL-CS_TDEP_REF_CLGRATIO_CTRL_IMAG
H1:CAL-CS_TDEP_PCALY_LINE2_REF_D_REAL H1:CAL-CS_TDEP_PCALY_LINE2_REF_D_IMAG
H1:CAL-CS_TDEP_PCALY_LINE2_REF_C_NOCAVPOLE_REAL H1:CAL-CS_TDEP_PCALY_LINE2_REF_C_NOCAVPOLE_IMAG
H1:CAL-CS_TDEP_PCALY_LINE2_REF_A_USUM_REAL H1:CAL-CS_TDEP_PCALY_LINE2_REF_A_USUM_IMAG
H1:CAL-CS_TDEP_PCALY_LINE2_REF_A_TST_REAL H1:CAL-CS_TDEP_PCALY_LINE2_REF_A_TST_IMAG H1:CAL-CS_TDEP_ESD_LINE1_REF_C_IMAG
H1:CAL-CS_TDEP_DARM_LINE1_REF_A_USUM_REAL H1:CAL-CS_TDEP_DARM_LINE1_REF_A_USUM_INV_REAL
H1:CAL-CS_TDEP_DARM_LINE1_REF_A_USUM_INV_IMAG H1:CAL-CS_TDEP_DARM_LINE1_REF_A_USUM_IMAG
H1:CAL-CS_TDEP_DARM_LINE1_REF_A_TST_REAL H1:CAL-CS_TDEP_DARM_LINE1_REF_A_TST_IMAG H1:CAL-CS_TDEP_DARM_LINE1_UNCERTAINTY H1:CAL-CS_TDEP_PCALY_LINE1_UNCERTAINTY H1:CAL-CS_TDEP_PCALY_LINE2_UNCERTAINTY
H1:CAL-CS_TDEP_PCALY_LINE3_UNCERTAINTY H1:CAL-CS_TDEP_SUS_LINE1_UNCERTAINTY H1:CAL-CS_CARM_DELTAF_DQ
H1:CAL-CS_LINE_SUM_DQ H1:CAL-DARM_CTRL_WHITEN_OUT_DBL_DQ
H1:CAL-DARM_ERR_WHITEN_OUT_DBL_DQ H1:CAL-DELTAL_CTRL_DBL_DQ H1:CAL-DELTAL_CTRL_PUM_DBL_DQ
H1:CAL-DELTAL_CTRL_TST_DBL_DQ H1:CAL-DELTAL_CTRL_UIM_DBL_DQ H1:CAL-DELTAL_EXTERNAL_DQ
H1:CAL-DELTAL_RESIDUAL_DBL_DQ H1:CAL-INJ_ODC_CHANNEL_OUT_DQ

This seems to be going on and off. The channel list above was generated from data from gps 1162941549 (November 11, 2016 23:18:52UTC = ~15:19PST). This may be related to the oaf work. at the time.