Once observing, the telescope will (mostly) look after itself.
The Fire and Primary Monitoring (PMon) systems monitor a number of critical telescope systems (e.g. power, cryogenics). Critical problems will trigger alarms which are distributed to site staff.
Observers need to monitor their own observations to ensure data quality. During observing, vis and spd should be checked semi-regularly to ensure that the data “looks ok”. In particular checking that, while on the calibrator, delays are close to zero, phases are flat(ish) and amplitudes remain constant. (There are a number of modes where some of these checks may be inappropriate, however, get help if it isn't obvious why these are not the case.)
The program assistance will often indicate that there is a problem with the system. Many of the error messages can be ignored if they are not relevant to the observations, however any unexpected errors should be followed up.
The MoniCA task displays monitor point information from antennas and other places.
xbones stows the array if the average wind speed for a 10 sec period is above 40km/hr
the PMon rack takes control of the array and stows if it detects a wind speed above 50 km/hr
Observers should respect this and not restart observing until the wind speeds have died down. In fact, if PMon has issued the stow you will have to get the Duty Astronomer to call the on-call person to reset PMon (which is located in Narrabri).
In addition to high winds, electrical storms present a real danger to the ATCA. Storms and lightning strikes within a 56 km radius of the array are closely monitored. Threat levels (from 0 - no threat, to 4 - high threat) are assigned based on the number of approaching lightning strikes and those within 9 km of the array. When the threat level reaches 3, the back up generators are turned on. At a threat level of 4, xbones stows the array.
On very hot days (above 40°C) especially with a strong wind, the cryogenic compressors can become overwhelmed. In order to protect the receivers, if the ambient temperature in the internal system of the antennas reaches 50°C, xbones stows the array and turns the antennas to shadow the compressors from the wind.
Weather monitors (for wind, storms and heat) can be access in MoniCA. Alternately, the ATCA Weather Station also provides information about the weather.
The first action in the face of an observing problem should always be to wait for at least a minute. (There are obvious exceptions to this rule, e.g. fires and similar catastrophic events that require quick action.) If waiting doesn't fix the problem, stop, restart and wait for at least another minute.
The following flowcharts provide details on how to recover from some of the most common observing problems. They explain what to do in the case of No Correlation (or funny correlation), Antenna(s) not 'on source' and VNC Problems , respectively.
If you are unable to fix or diagnose your problem after following the flowchart you should call your Duty Astronomer. Sometimes you may be able to diagnose your problem but not be qualified to execute the required fix and in this case you should also call your Duty Astronomer. This will be the case if you need to reboot a CABB block , for example.
It is recognised that observers will not always be able to constantly monitor their observations. For those periods when the telescope is unattended, there are ways to get alerted to problems that need urgent action. This section describes the unattended observing alarm notification system available for the ATCA.
The core component of the unattended alarm system is the Twitter feed. Whenever an alarm is generated by MoniCA, or an alarm status goes away, a message is posted to the Twitter feed describing the problem.
Figure 3.9. Some example tweets from the ATCA alarm Twitter feed.
A couple of example tweets are shown in Figure 3.9. Each tweet starts
with the UTC date and time that the alarm state changed. The date is listed as year-month-day, but
where the century part is omitted, to save characters. All alarms are given the hashtag
#ALARM; this may currently seem redundant, but this Twitter feed will likely be
expanded to include problems that do not generate MoniCA alarm states.
If an alarm has been triggered, then the next part of the tweet will be a hashtag describing how
important the alarm is. An
#INFORMATION alarm simply presents a condition report,
but likely does not require any action.
#MINOR alarms can be ignored until
it is convenient, where you should probably investigate
#SEVERE alarms as quickly as possible.
If an alarm state has gone away, there will be no severity hashtag.
The rest of the tweet is then some guidance text about which system has caused the alarm. The guidance text is not designed to help you respond to or correct the problem; training on how to deal with problems that arise with the ATCA is still required. Rather, the tweet is there to quickly alert you to the problem.
The Twitter feed is still under active development, to make it as useful for observers as possible. All feedback about your experiences with the Twitter feed is welcome, as are suggestions on ways in which it can be made more useful.
The alarm notification system is based around Twitter because of the variety of methods
available to receive tweets on mobile devices. The simplest way of course is to install the
official Twitter client on your mobile device, and subscribe to the
@jamie_atca_sss Twitter feed from your account. In that way, you can make
your mobile device respond in one of several ways to any alarm tweets.
If you'd rather not set up a Twitter account, you can quite easily get tweets via the Twitter
SMS system. This way, you can still be notified of alarms when you have your
phone with you. The official Twitter SMS service can be used as described
on the SMS follow web page.
For example, for Australians, you may get the alarm tweets via SMS by sending
an SMS “
(without the quotes) to the number
0198089488. You will then get a return message if this has been successful, that
will also tell you how to stop following. In addition, you'll get the most recent tweet from the
You can also use the free online web service IFTTT (if-this-then-that) to respond in a customisable way to tweets from the ATCA alarm system. You will need to sign up to IFTTT, and connect your Twitter account to their Twitter channel, but after that you may specify any number of ways to get notification of tweets from the alarm system. For example, there is a published recipe that takes an alarm tweet and sends you an SMS. But you could also make your smart lights at home turn red, or get a phone call (in certain regions only).
Using IFTTT may also be a more convenient way to get SMS from the alarm feed, because you can easily turn it on or off via the web interface or their mobile app, rather than having to send an SMS each time you want to start or stop getting alarm notifications.