CSP LMC Troubleshooting

Since the CSP LMC is the entry point of all the CSP sub-elements, the State, ObsState and HealthState of the CSP LMC have to reflect also the State, ObsState and HealthState of all the CSP subsystems. In order to identify the cause of a particular condition of the CSP LMC, the starting point is to understand the subsystems status. For doing that, we have defined some tables to link the status of the CSP LMC and the one of the subsystems.

Recover from a FAULT obsState

If something goes wrong with the observation, the obsState of the Mid CSP.LMC Subarray could transition to FAULT. Note that this condition differs from the one where the State of a TANGO Device is in FAULT (see below). To recover from this situation, issue the Restart command. This command will release also all the resources, bringing the subarray into an EMPTY obsState.

Considering the deployment and proxies defined in the example page, the sequence of operations and responses is:

csp_sub1.obsState -> FAULT

csp_sub1.restart()

csp_sub1.commandResult -> ('restart', '1')
csp_sub1.obsState -> RESTARTING
csp_sub1.commandResult -> ('restart', '0')
csp_sub1.obsState -> EMPTY

If the CSP LMC Subarray fails to recover from its FAULT observing state or if its status becomes misaligned with those of its subordinate subsystems, an alternative option is to utilize the Off command.

Recover from a FAULT State

When an internal error occours in the Mid CSP.LMC Subarray or in the Controller, the DevState (State) of the device can transition to FAULT. To recover from this condition, the Reset command needs to be issued. If executed successfully, this command will restore the device to a state determined by by the aggregation of the states of its subordinate systems. The same applies to the observing mode for the CSP Subarray.

Please be aware that when the Mid CBF device state is DISABLED, the state of the corresponding CSP LMC device is indicated as FAULT. Even after executing the Reset command, the CSP LMC device remains in the FAULT state.

In such a scenario, it’s advisable to first set the CSP LMC device’s AdminMode to OFFLINE, then switch it back to ONLINE, and verify whether the state of both the Mid CBF device and the CSP LMC device has changed.

Recover from a subarray FAILED healthState

When the DishID-VCC map configuration fails, the subarrays healthState goes to FAILED. In this condition, the ON command cannot be issued to the MID CBF device. To solve this situation, a new successful LoadDishCfg command has to be issued.

Recover subarray using the Off command

According to the ADR-8, the Off command is designed to be executable from any observing state. Its purpose is to disable all signal processing capabilities of a subarray and release all allocated resources.

Depending on the current observing state of the Mid CSP.LMC Subarray, the Off command can be substituted with a series of commands to properly bring the device to the desired final state. For instance, if one of the subordinate subarrays is in the READY state, the Off command is replaced by the following sequence of commands:

  • GoToIdle

  • ReleaseAllResources (or RemoveAllReceptors)

  • Off

In the case where one or more subarrays are in a transitional state, the approach that works in most cases is to replace the Off command by the following sequence of commands, executed consecutively:

  • Abort: This command transitions the subarray from its current observing state to ABORTED. It can be initiated from all observing states except EMPTY and FAULT. If the subarray is in one of these states, this step is skipped, and the first command invoked must be Restart.

  • Restart: This command transitions the subarray from ABORTED to either EMPTY or ON.

  • Off: Finally, this command transitions the subarray from ON to OFF.