## 3.4. Array Monitoring

Once observing, the telescope will (mostly) look after itself.

The Fire and Primary Monitoring (PMon) systems monitor a number of critical telescope systems (e.g. power, cryogenics). Critical problems will trigger alarms which are distributed to site staff.

Observers need to monitor their own observations to ensure data quality. During observing, vis and spd should be checked semi-regularly to ensure that the data “looks ok”. In particular checking that, while on the calibrator, delays are close to zero, phases are flat(ish) and amplitudes remain constant. (There are a number of modes where some of these checks may be inappropriate, however, get help if it isn't obvious why these are not the case.)

The program assistance will often indicate that there is a problem with the system. Many of the error messages can be ignored if they are not relevant to the observations, however any unexpected errors should be followed up.

The MoniCA task displays monitor point information from antennas and other places.

### 3.4.1. Wind Stows

The anemometer at the observatory measures local wind speeds every 2.5 sec. There are two (independent) systems that stow the array in the face of high winds:

• xbones stows the array if the average wind speed for a 10 sec period is above 40km/hr

• the PMon rack takes control of the array and stows if it detects a wind speed above 50 km/hr

Observers should respect this and not restart observing until the wind speeds have died down. In fact, if PMon has issued the stow you will have to get the Duty Astronomer to call the on-call person to reset PMon (which is located in Narrabri).

### 3.4.2. Storm Stows

In addition to high winds, electrical storms present a real danger to the ATCA. Storms and lightning strikes within a 56 km radius of the array are closely monitored. Threat levels (from 0 - no threat, to 4 - high threat) are assigned based on the number of approaching lightning strikes and those within 9 km of the array. When the threat level reaches 3, the back up generators are turned on. At a threat level of 4, xbones stows the array.

### 3.4.3. Heat Stows

On very hot days (above 40°C) especially with a strong wind, the cryogenic compressors can become overwhelmed. In order to protect the receivers, if the ambient temperature in the internal system of the antennas reaches 50°C, xbones stows the array and turns the antennas to shadow the compressors from the wind.

### 3.4.4. Weather Monitoring

Weather monitors (for wind, storms and heat) can be access in MoniCA. Alternately, the ATCA Weather Station also provides information about the weather.

### 3.4.6. Troubleshooting

The first action in the face of an observing problem should always be to wait for at least a minute. (There are obvious exceptions to this rule, e.g. fires and similar catastrophic events that require quick action.) If waiting doesn't fix the problem, stop, restart and wait for at least another minute.

The following flowcharts provide details on how to recover from some of the most common observing problems. They explain what to do in the case of No Correlation (or funny correlation), Antenna(s) not 'on source' and VNC Problems , respectively.

If you are unable to fix or diagnose your problem after following the flowchart you should call your Duty Astronomer. Sometimes you may be able to diagnose your problem but not be qualified to execute the required fix and in this case you should also call your Duty Astronomer. This will be the case if you need to reboot a CABB block , for example.

### 3.4.7. Alarm notification during unattended observing

It is recognised that observers will not always be able to constantly monitor their observations. For those periods when the telescope is unattended, there are ways to get alerted to problems that need urgent action. This section describes the unattended observing alarm notification system available for the ATCA.

#### 3.4.7.1. Twitter feed

The core component of the unattended alarm system is the Twitter feed. Whenever an alarm is generated by MoniCA, or an alarm status goes away, a message is posted to the Twitter feed describing the problem.

A couple of example tweets are shown in Figure 3.9. Each tweet starts with the UTC date and time that the alarm state changed. The date is listed as year-month-day, but where the century part is omitted, to save characters. All alarms are given the hashtag #ALARM; this may currently seem redundant, but this Twitter feed will likely be expanded to include problems that do not generate MoniCA alarm states.

If an alarm has been triggered, then the next part of the tweet will be a hashtag describing how important the alarm is. An #INFORMATION alarm simply presents a condition report, but likely does not require any action. #MINOR alarms can be ignored until it is convenient, where you should probably investigate #MAJOR and #SEVERE alarms as quickly as possible.

If an alarm state has gone away, there will be no severity hashtag.

The rest of the tweet is then some guidance text about which system has caused the alarm. The guidance text is not designed to help you respond to or correct the problem; training on how to deal with problems that arise with the ATCA is still required. Rather, the tweet is there to quickly alert you to the problem.

The Twitter feed is still under active development, to make it as useful for observers as possible. All feedback about your experiences with the Twitter feed is welcome, as are suggestions on ways in which it can be made more useful.

The alarm notification system is based around Twitter because of the variety of methods available to receive tweets on mobile devices. The simplest way of course is to install the official Twitter client on your mobile device, and subscribe to the @jamie_atca_sss Twitter feed from your account. In that way, you can make your mobile device respond in one of several ways to any alarm tweets.
If you'd rather not set up a Twitter account, you can quite easily get tweets via the Twitter SMS system. This way, you can still be notified of alarms when you have your phone with you. The official Twitter SMS service can be used as described on the SMS follow web page. For example, for Australians, you may get the alarm tweets via SMS by sending an SMSfollow jamie_atca_sss” (without the quotes) to the number 0198089488. You will then get a return message if this has been successful, that will also tell you how to stop following. In addition, you'll get the most recent tweet from the alarm feed.