Australia Telescope Compact Array

The time assignment and scheduling process

The process to assign and schedule time on the ATNF telescopes has many steps. This focuses on the processes relevant for the ATCA and Mopra telescopes.

Sketch of relevant directories and files

The top-level directory for the time assignment/scheduling process is $OPER/sched. Each semester (or previously each term) has its own subdirectory, e.g. 06apr which contains the information for both the ATCA and Mopra for that semester. This semester subdirectory also has some information for Parkes, the LBA and Tidbinbilla.

  tacdb.csv     Copy of the comments from the TAC database for all
                previous proposals.
  scripts       A subdirectory containing a collection of Perl scripts to
                help with the scheduling process.
  scheduler     A subdirectory containing a copy of Dave McConnell's
                scheduling program (java).
  06apr         A typical semester subdirectory.

Before the TAC meeting ...

Prepare availability information: The call for proposals (put out by Jessica Chapman) is normally circulated one month before the proposal deadline. About two weeks before the call for proposals goes out, the plans and availability for the upcoming semester are crystallised. The call for proposals will typically highlight array configurations that are offered, availability of new observing facilities (receivers, back-ends) and any restrictions that might help observers to make sensible use of the facility during the upcoming semester.
Ring binder of proposals: After the proposal deadline, Jessica Chapman will organise for the distribution of paper copies of the proposals. The paper copies of the proposals are used frequently during the time assignment and scheduling process, and occasionally throughout the semester. There is often extra supporting material for the time assignment process. Put them in ring binders!
Create directories and download XML files: To simplifier the lot of the ATCA/Mopra scheduler, the scheduler should also download the XML files associated with all the ATCA and Mopra proposals. Create the semester subdirectory, download the XML files to this directory and unpack/rearrange them. Make sure read access to the proposals is restricted as the information is private.
```
  % mkdir $OPER/sched/06apr
  % cd $OPER/sched/06apr
     ... download XML files to this directory ...
     ... store these in subdirectories "atca" and "mopra" of the semester directory ...
  % find atca mopra parkes lba tid -exec chmod o-rw {} \;
```
The XML files (coversheet.xml and observations.xml) need to be placed in the proposal-specific subdirectories of atca and mopra (e.g. the XML files for proposal C007 is in the direction atca/C007). It is convenient to place any proposal-specific material (including copies of the PDF files) in these subdirectories as well.
Generating the .txt files: The XML files contain much of the information needed for the scheduling process. The process to get the information into the scheduler is a three-step process: generate a text file from the XML; edit the text format; finally convert from text format to the scheduler format. To convert to text format, use
```
  % ../scripts/xml2txt.pl    > ca.txt
  % ../scripts/xml2txt.pl -m > mp.txt
```
These files are in a simple, compact, text format listing the requested sources, their RA/DEC or requested LST range, the requested array configurations, good and bad scheduling dates, etc. A typical proposal will be summarised in about 15 lines.
Generate schedulers coversheets: I find it convenient to have an additional page for each proposal which contains comments on the proposal from previous semesters, as well as place for me to write miscellaneous notes. Consequently I generate an extra "schedulers coversheet" page for each schedule. To generate this, use
```
  % ../scripts/proj_fmt.pl ca.txt mp.txt
```
This will produce a file proj_fmt.ps that can be printed. It is helpful to print these on coloured paper so that they act as a simple separator between proposals. Put these into the ring binders along with the proposals.
Consider proposals: Jessica Chapman will send out an Excel spreadsheet to complete comments on ATCA and Mopra proposals - the pre-grade comments. This comes with instructions and deadline a few days before the TAC meeting. At this stage the ATCA/Mopra scheduler
- Reads through each proposal, and considers it from a technical and logistical point of view, and completes the Excel spreadsheet for the TAC. The TAC members are usually not experts at radio interferometry (indeed many will not be observational astronomers), and so they will not appreciate technical aspects of a proposal that may make it difficult or impossible.
- completes the ca.txt and mp.txt files (see below). This consists of correcting/reformatting the information generated by xml2txt.pl, filling in missing information, and breaking the proposal into a collection of slots.
- makes notes about additional aspects of the proposal that are not apparent from the cover sheet. For example there will often be requests to synchronise the proposal with other observing, or to minimise the number of trips to Australia by overseas observers.
Note this step takes 10s of hours of continuous hours of work. For example if there are 120 proposals and 15 minutes are needed to consider each proposal, then it will take 30 solid hours of work.
Fine tune availability and maintenance requirements for the semester: In the few weeks before the TAC meeting, fine tune the availability of different systems, or the requirements and constraints of various maintenance/installation/commissioning work. You will need to be pro-active in getting information from the Narrabri group leaders and the Marsfield engineering staff. This may affect what you offer at the TAC meeting, and the amount of time required outside astronomy. Also before the TAC meeting it is generally possible to determine the (approximate) dates for any VLBI sessions, through discussion with Jim Lovell, Tasso and John Reynolds.
Generate the scheduling tool files: Once the ca.txt and mp.txt files are complete, then the files required by the scheduling tool can be generated:
```
  % ../scripts/txt2scd.pl ca.txt > CA_2006APRT1.scd
  % ../scripts/txt2scd.pl mp.txt > MP_2006APRT1.scd
```
The script txt2scd.pl does a number of checks for validity of the input, but it is not boiler-plate rock solid. Errors in the input to txt2scd can get through to the scheduling files. You might load the scheduling files into the scheduling tool to see if it accepts them.
Write the TAC report: My TAC reports have included usage statistics (derived from the usage suite), details of the NAPA and ToO programs that have happen since the last TAC report (cut/paste from the web NAPA/ToO page, which should be kept up-to-date), comments on proposal statistics and maintenance requirements for the coming semester (oversubscription, skews in LST ranges requested, imbalances between 3mm and centimetre proposals in configurations, etc). The script to help generate some of these statistics is pre_scd. This produces output in a .csv format, and so the output can be put into Excel to produce "charts", as well as simplifying some cut-and-pasting of numbers. To generate the .csv file, use
```
  % ../scripts/pre_scd.pl CA_2006APRT1.scd > pre.csv
```
Distribute this report as appropriate - certainly e-mail it to Jessica Chapman, and make sure paper copies are available during the TAC meeting.

The ca.txt and mp.txt files

The rationale for having intermediate text files (between the XML files and the scheduling file), is that a fair bit of work is required to correct and enter all the proposal detail. A simple to handle, compact format is required. The power of modern text editors can also be harnessed. The GUI provided by the scheduler could be used in principle. But in practise this is way too cumbersome to enter more than a small amount of information.

These text files consist of a short preamble describing the semester followed by an entry for each proposal. A typical preamble will be of the form:

  Observatory: atca
  Arrays: 6C; 6A; 6D; 1.5D; 750B; EW352; H214; H168; H75; Split
  Term: 2006APRS 
  Maintenance: 2-day; 3-day; 2-day; 4-day; 3-day; 2-day; 3-day; 2-day; 3-day
  Holidays: 14/4-17/4; 25/4; 12/6
  VLBI: 156
  ------------------------------------------------------------------------

This gives the observatory name (atca or mopra), the array configurations to be scheduled during the semester (the first one should be the one that carries over from the previous semester, and a configuration name of none should be used for Mopra), the semester name, a preliminary guess of the maintenance periods needed in the semester, the holidays during the semester (do not schedule maintenance on these days!), and the VLBI session(s) length in hours. For multiple entries, separate them with semicolons.

The following is a fairly typical proposal description in the ca.txt file. With one exception, each line starts with a field description and a colon. The exception to this is the "Comment:" field: every line up to the line of minus signs is considered to be comments.

  Project id: C015
  Title: SNR 1987A
  PI: Staveley-Smith
  Email: Lister.Staveley-Smith@csiro.au
  Requested time: 4x8; 2x12
  Array: 4xany6(any1.5); 2x6a
  Band: 4x20/13+3/6; 3; 12mm
  BW: 128/128
  Source: SN1987A
  RADEC: 05:35:28,-69:16:11
  Good dates:
  Bad dates: 14/08-25/08
  Service obs: No
  Comments:
  Flux monitoring sessions spaced by at least 4 weeks
  Bad dates are IAU-GA
  ------------------------------------------------------------------------

Most of the fields are self-explanatory, any many will be set by xml2txt.pl. This script does some checking of reasonableness of the parameters, and puts comments starting with three hash marks where it sees something that looks odd.

Important to the contents of these text files, and to the scheduling process, is the "slot" - a slot is a chunk of time that can be usefully/sensibly scheduled. For the ATCA, if the proposal was to do a 12-h synthesis, a slot would correspond to a 12-h chunk of time or perhaps a little more. If you want to do a full synthesis, then it makes little sense to schedule much less than 12 hours. At Mopra, where there are no requirement to do a full synthesis, the amount of time in a slot is more flexible. Similarly at the ATCA, if the proposal is not after a synthesis (eg snapshots, monitoring etc), then the amount of time in a slot can be more relaxed. The exact way that a proposal is formed into a collection of slots will depend very much on the observing mode and the objectives of the proposal. It requires an astronomical understanding of the proposal.

One of the tasks of the scheduler is to form a proposal into a collection of slots. Sometimes this is very obvious (as in the case above), sometimes it requires some thought (and some proposers are quite good at making a suggested way to schedule their request), sometimes it is quite clear that the proposers have given no thought to this (their request might not be practically achievable as stated).

The initial slots generated by xml2txt.pl correspond to the entries of the OPAL observations table. Sometimes these correspond to sensible slots. At other times, one stands in awe at the lateral thinking of some minds and the different ways to (ab)use OPAL's repeat field!

When done, the fields "Requested time", "Array", "Band", "BW" and "Source" fields contain descriptions of all the slots of the proposal. Each field gives a list of the request for a particular slot. So if a proposal is broken into 10 slots, then there will be parameter entries in these fields that correspond to each of these 10 slots.

To make it compact, a repeat count (an integer followed by the x character) can be used against a parameter. In the above example, the "Requested time" has six entries, four slots of 8 hours, and two slots of 12 hours.

Also to make it compact and simple, when all entries for one field would be the same, then the repeat count can be omitted - the repeat count is implicitly determined. The only exception to this is the "Requested time" field, where this implicit repeat count cannot be used. In the example above, the source name, bandwidth and RA/DEC are the same for all slots - and so the repeat count can be omitted.

When an entry is long, then you can break it over multiple lines, by ending a line with the backslash character (for line continuation). Some entries require a little more explanation:

Bands

This gives the observing bands used during the observation, with possibilities being 20, 13, 6, 3, 12mm and 3mm. When two bands are observed simultaneously, separate them by a slash character. When multiple bands are observed by frequency switching, combine them with a plus character. For instance, in the example about, 20/13+6/3 indicates that the proposal is switching between two dual-frequency set-ups. The dual bands are 20 and 13cm, and 6 and 3 cm.

Good and Bad dates

A range of dates are given in the form dd/mm-dd/mm. A year is not required (this is worked out). When there are multiple ranges, separate them with a semicolon.

RADEC

The RA and DEC of the source are used to work out when the source is up. Sometimes it is more appropriate to give this as an LST range. In this case, use the form lst(hh:mm,hh:mm) to give an LST range instead of a source RA,DEC. For example

  RADEC: lst(01:00,13:00)

would request a slot to be scheduled between 1:00 and 13:00 LST.

Some proposers are not proficient at completing OPALs observations tables, and so they expect you to do their work of entering some source information. The script expsrc.pl takes a text file (or a fragment of a text file), and fills source RA/DEC where these are missing. Using NED, Simbad and some source naming rules, it deduces the RA/DEC for a source with a given name. Note it is not infallible and source names are not always unique. Also note that the precision of the RA and DEC needed for the scheduling process is quite modest - degree rather than arcsecond accuracy is adequate. Do not be concerned if the coordinates generated are not of astrometric accuracy. A typical use would be

  % ../scripts/expsrc.pl ca.txt > new-ca.txt

Array

This can be any of the standard array names (6a, 6b, ... ew352, ew367, h214, h168, h75), plus a few generic names (any6, any1.5, any750). Where there is an alternative, then this can be given in parentheses. In the example above,

  Array: 4xany6(any1.5); 2x6a

the proposer prefers four 6km arrays (any of the 6km configurations would do), but the proposer will accept 1.5km arrays if necessary. In addition two array configurations should be 6A. When there are multiple alternatives, separate these by commas within the parentheses.

The scheduling tool

The scheduling tool (written by Dave McConnell and Craig West) is a tool that assists in the "jigsaw" part of the scheduling process. It keeps a tally on slots, shows you where slots can fit into the schedule and allows you to add and remove them from the schedule. It is the tool responsible for producing the standard layout of the schedule.

Setting up the scheduling tool

The scheduling tool is written in java. It requires various files that it needs (from time to time) to be in subdirectory atnf\scheduler relative to the current working directory. The tool works on Windows platforms, but I have had some problems working with it on Solaris (I have never explored these problems). I have worked around these two characteristics by restricting use of the tool to Windows, having a directory containing the scheduling files and having the scheduling tool software in a atnf\scheduler subdirectory of this. Invoke the tool (from the scheduling file directory) with the equivalent of

  java atnf.scheduler.Program

I have found it convenient to create a shortcut for using the scheduling tool on the Toolbar, and shortcuts on the Desktop to the relevant scheduling file directories. The tool also runs ok on kaputar, just make a symbolic link in the current schedule directory (e.g., 06apr) to operations/scheduler/atnf :

ln -s ../scheduler/atnf atnf

Typing java atnf.scheduler.Program then works.

The scheduling tool can produce postscript output. I find the postscript to be a particularly helpful way to view the schedule. Consequently almost always when I am using the scheduling tool, I also have gsview also running. Note the sort of postscript generated by the scheduling tool does not contain perfect DSC directives. In my settings for gsview, under the Options menu, I have DSC Warnings set to Off, Auto Redisplay enabled, and Ignore DSC not enabled.

If you have gsview running simultaneously and Auto Redisplay is enabled, then after you generate the postscript, gsview will repaint as soon as the focus shifts to its window.

Handling files and versions

I run the scheduling tool on a Windows machine, whereas all the remaining tools are UNIX-based. I tend to always have two copies of the schedule that I am working one: one sitting on the Windows disk, and the other sitting on a UNIX disk. I copy the scheduling file back and forth between the two systems (too) frequency.

I am sure others can find a neater solution for all of this.

The scheduling files are in an XML-like format. You will, on occasions, need to go inside these files with a text editor and change parameters "manually".

The scheduling files have names such as CA_2006APRT1.scd, which indicates it is version 1 of the 2006APR semester schedule for the Compact Array. The file name and version number is also embedded inside the file itself (I do not know the rationale for this). To increment the version number of a scheduling file, use

  % ../scripts/newver.pl CA_2006APRT1.scd > CA_2006APRT2.scd

When the scheduling tool saves a file, regardless of the name of the input file and of the filename embedded in the file itself, the tool works out what it thinks the output should be. If you change the start dates of the semester significantly, the scheduling tool will start saving the output to a file with a different name.

Using the scheduling tool

After invoking the scheduling tool, it will pop up a dialogue box to select an (existing) schedule file to work with. Double click on the appropriate file. At this point, the scheduling tool opens its main window.

Below is a screen shoot of this window.

The "File" button allows you to save the current file, to generate postscript of the current file (select the menu item "Print" - which just generates postscript), and to exit. Generate the postscript frequently, and look at the result with gsview.

Note if exiting, it does not warn you if you are exiting with outstanding changes left unsaved! It also can generate various reports, which I do not use.

The scheduling tool is know to get itself into a knot from time to time. It is wise to save your work often.

The tool can be pedantic about the way you enter text. Complete any text that you enter with a carriage return.

The "View" button allows you to view the number of hours committed to slots that have a higher rating that the rating given in the "Select Rating" field. I do not use this much. The "Select Array" field does not select anything (it has no function).

Below the main window is broken down into five boxes.

The proposal box: This box (near the top left) has a scrollable listing of all the proposals in the current scheduling file. It gives the project Ident, the PI and the number of slots. Use the slider and click on a line to select a proposal.
To create a proposal, put the project Ident next to the "Create" button, and press "Create", enter the PI in the PI field, and enter the title in the entry area to the lower right of the "Delete" button.
To delete the selected proposal, use the "Delete" key. Make sure there are no scheduled slots before deleting (the tool gets itself into a knot if there are scheduled slots in a deleted proposal).
The proposal slots box: This is a table of slots in the centre of the tool which extends across the full width of the tool. This lists all the slots of the currently selected proposal. Click on a slot to select it. Most of the fields in this table are edittable. The exceptions are the slot number ("i" - the leftmost column), the scheduled hours ("Sch(h)") and the scheduled date ("Date" - the rightmost column).
Some columns allow you to drag a value down its length. This is particularly useful for the "Rank" column (should be called the Rating column): at the TAC meeting you will usually want to change the rating of all slots of a proposal from "X" (not rated) to a particular value. To do this
- Double click and enter the rating value in the top cell of the column.
- Left click in the "Date" cell.
- Left click and hold down in the top cell of the column again.
- Press and hold down F9, and drag the value down the column.
Once you have burnt this strange key sequence into your brain, you will wonder what was so strange about it in the first place.
You can create a blank slot with the "Create" button. Once you have one slot, it is usually preferable to generate new slots with the "Copy" button (it makes a copy of the currently selected slot). The "Delete" button deletes a slot. Make sure a slot is not scheduled before deleting it!
The proposal slots displayed are those for the example ca.txt entry given above. Note the first two slots have ratings of "E" and has dates which are before the formal start of the semester. These are carry over slots from the previous semester. Also note that in slot 5, the Array has been changed to "any" (from "any 6"). This slot was scheduled in a special array which was considered equivalent to a 6 km configuration.
The Slot RA/DEC/LST/Dates box: This box, in the upper right of the tool window, shows the RA and DEC of the source of the currently selected slot, as well as preferred (good), bad and required dates. There is also a proposal note area. The "LMST Start", "LMST End" and "Use LST limits" boxes and toggle allow you to set the LST range of interest for the slot directly.
All the fields are edittable.
The schedule box: This box extends across the width of the window, but near the base of the window. There is a band of information which gives the current state of the schedule. The top part of the band indicates the part of a day, in green, where the current selected slot could be scheduled (given the LST range it requires). The middle part of the band gives blocks according to scheduled slots. Astronomy slots are in green,\ reconfiguration slots are in yellow and maintenance slots are in blue. The blank parts are Director's Time, or places where other slots can be added. The bottom part of the band gives dates (AEST). Weekends are colour grey (holidays are not highlighted, unfortunately).
You can click on scheduled slots in the middle part of the band to select this slot (the Proposal box and Proposal Slots box will update to the selected slot). You might need to click more than once to get its attention.
There are two sliders: the bottom slider selects which range of times the band represents. The top slider is used to position over a section of time. The width of the top slider corresponds (generally!) to the number of hours in the currently selected slot. The date, time, duration and array configuration to the right of the "Remove" button reflect the current position and width of this slider. You will become accustomed to the top slider being cock-eyed when it is fully zoomed out.
There are a number of buttons: the "Zoom in" and "Zoom out" do as their names suggest with the scheduling band. The "Add" button adds a slot to the schedule. It adds it at the time range implied by the top slider. You will probably want to adjust the top slider so that a slot starts on a half hour. The "Remove" button removes a scheduled slot out of the schedule.
The < and > buttons move the current slot, if it is scheduled, to start half an hour earlier or later respectively. The << and >> buttons move the scheduled slot as early or as late as possible. The two buttons in the diagram that are blank (but they are not in the tool itself) shorten or lengthen the currently scheduled slot by half an hour. They always shorten or lengthen at the end of the of the slot, so you might need to use the shorten/length buttons in concert with the < and > buttons to get the effect you want.
The "Add" button will not let you add a slot if the selected time range is outside the acceptable LST range. Use the "Relax LST" toggle to override this check. It will also not let you add a slot when the array configuration is incompatible. To override this, change the array configuration of the slot. Naturally it will also not allow you to schedule two slots on top of each other.
The Times for scheduled slot box: If the currently selected slot is scheduled, this gives the times of the slot. Note the date is an AEST date, whereas the date inn the Proposal Slots box is a UT date.

The scheduling tool recognizes a few special cases:

Specifying NASA for the project id results in a red box in the output. The box will normally say "NASA", but specifying a source name starting with "!" will replace this text with whatever follows the exclamation mark (useful for special events, like open days).
If the observatory is Parkes, reconfigs are labelled as receiver changes. No checking is done whether observations are added in the right receiver configuration.

At the TAC meeting ...

Having read the proposals, returned pregrade comments, written the TAC report and generated the scheduling file (e.g. CA_2006APRT1.scd), you are now ready for the TAC meeting. The TAC meeting consists of three sections - general business, observatory report and detailed discussion of each proposal. During the detailed discussion, a rating is assigned to each proposal (indeed potentially to each slot). The TAC might vote on giving a proposal less (and sometimes more) time or to not consider the proposal schedulable (to ``fail'' the proposal). Failing a proposal does not necessarily reflect badly on the proposal: it might be that the proposal was not technically possible this semester, or that it should be considered at a different time. The TAC might also vote after changing some of the characteristics of the slots. The TAC might also leave a question open where the scheduler needs to contact the proposers for further clarification. All is fair game at a TAC meeting.

Ratings are generally numbers from 0.00 to 5.00. Additionally a number of special ratings are given by letters:

  X    Proposal yet to be rated.
  T    Maintenance/test/reconfiguration time.
  F    Proposal not to be scheduled ("fail").
  E    Scheduled, carry-over project from a different semester. Proposals
       with these ratings will generally appear near the edge of semesters.

So during the course of the meeting, you will be involved in the discussion (mainly from the technical and logistical aspects of the proposal), as well as using the scheduling tool (see below) to modify the slots that are to be considered for scheduling and to record the ratings against each slot.

The scheduling process

The caDo and mpDo scripts

I have found it convenient to have a short script which runs four other scripts that I commonly use in scheduling. These four scripts each produce listing files of one form or other. It is rare that you would want to look at the current state of all four listings at a particular point in the scheduling. But then again it does not hurt to produce the listings either. The script to produce these listings, caDo and mpDo, has a few parameters at its head that you will want to adapt - its pretty clear what they are. caDo and mpDo run the following four scripts:

ltimetable.pl: This produces a .csv file, ca-timetable.csv that is used as part of the planning for reconfiguration dates and length of array configurations.
ldates.pl: Date constraints listing for each slot, dates.txt, and its postscript version dates.ps.
llst.pl: LST range listing for each slot, lst.txt and its postscript version lst.ps.
jlist.pl: This produces a file with name such as ca-1.jlist. This file is not used until the scheduling is complete.

The files dates.ps and lst.ps are big at the start of the scheduling process: they are designed to be printed on either A3 or A4 paper depending on their size.

Inevitably there will be conflicting requirements amongst the proposals that come through the TAC process. Some proposers will request an array configuration early in the semester, whereas others will ask for it late. The approach I have taken to scheduling is to let the constraints and requirements of the best-rated proposals determine the skeleton of the schedule. To do this, a (soft) cut-off rating is determined. Proposals with ratings above this threshold are considered (although not equally) when making the skeleton. Proposals below this are ignored. For the ATCA, set the ratings threshold so that the slots above this rating would consume 80-90% of projected available time.
For the ATCA, the first step in the scheduling process is to work out the order and approximate dates of the array configurations. This will be significantly influenced by observers good and bad dates and by possible restrictions imposed by maintenance and installation work.
The Excel file ca-timetable.xls, contains two worksheets. The second worksheet is the imported ca-timetable.csv file. You assign a configuration to each of the slots in this worksheet (hopefully the requested configuration). The first worksheet gives the a list of configurations, the "efficiency" of each configuration (a working guess at the efficiency is 75%, but different configurations such as the hybrid arrays tend to have more idle time), the start and end times of the semester, and the order in which the configurations are to be performed.
By inspecting the ldates.txt listing, by adjusting the order of configurations, and by perhaps changing the configuration used for a program, you can iterate towards an ordering which satisfies most peoples requests. Note you need to avoid reconfigurations on weekends, public holidays and the Christmas/New Year period.
Having worked out the order and dates of the reconfigurations, its a matter of going through and scheduling with the scheduling tool. Usually 2-3 days of maintenance time is scheduled per fortnight. Often this is scheduled to coincide with a reconfiguration. Reconfigurations are generally at least a 32 hour period. This consists of the physical move, the pointing and baseline recalibration, and then an extra day for extra maintenance (and as a safety net).

When the schedule is "complete"

When the schedule is complete, release it to group leaders, relevant Marsfield engineering staff, VLBI people plus Jessica Chapman. Jessica Chapman also gets the ca-1.jlist file, which she enters into the TAC database. Jessica does some checking on the schedule for reasonableness, and the others can give feedback on whether the schedule is OK from their prospective.

A few days after this internal check, and having dealt with any feedback, the schedule can be released on the web. Before doing this, you will want to feather together the previous schedule with the one just completed. This is so that the schedule pages show a continuous use of time. The script to do this is scdmerge.pl. Typical use would be

  % ../scripts/newver.pl CA_2006APRT1.scd > CA_2006APRT2.scd.tmp
  % ../scripts/scdmerge.pl 20060320 ../05oct/CA_2005OCTT5.scd CA_2006APRT2.scd.tmp >  CA_2006APRT2.scd

This generates a new version of the schedule (version 2) and then feathers in observations after 20/03/2006 (the date given by the first argument in yyymmdd form) from CA_2005OCTT5.scd into CA_2006APRT2.scd.

To generate the web-based information, use

  % ../scripts/push.pl -n ca

This command pushes the information for the Compact Array for the next term. (the -n flag above tells push to put the information in the directory for the next term). To push the Mopra schedule, use mp rather than ca with the push command. Also to update the schedule after the start of the term, omit the -n flag.

The push script does the following:

It generates and copies the pages for the schedule itself.
It generates and copies the schedule summary page.
It generates and copies the 13cm (microwave restrictions) schedule.

Note: It does not generate a swap schedule. This is still done by hand. It also does not change the web page that refers to the schedule - you will need to go into this web page and modify it to indicate the availability of the next schedule.

You will want to advertise the availability of the new schedule on the ATNF home page ``whats new'' section.

Jessica Chapman (or Jo Houldsworth) will send out an e-mail to relevant people when the schedule is available - let them know when you are ready for this.

After the dust has settled

After the schedules are done, and the change requests have died down, there are a few small jobs to complete:

The ATCA Live! page on the Narrabri web has taken on a new role with the Narrabri Visitors Centre becoming unstaffed. Visitors to the visitors centre can see what the Array is currently doing by looking at the ATCA Live! screen that is permanently running. Consequently it is helpful to make this as informative as possible to these visitors, as well as the those spread over the internet. The atca_live CGI script will display project descriptions and daily happenings, when it has this information. The process to generate this information takes input from two places:
- the events.txt file. This gives a description of daily happenings.
- the XML files, which contain project titles and outreach abstracts.
The events.txt file is a sequence of entries which give the description of something that is happening over a particular span of dates. These might be special astronomical events (e.g. eclipses), events of interest to space buffs (e.g. NASA mission events) or local events (special observations, synthesis workshops, special meetings). Each entry has the form
```
  Date: dd/mm/yy-dd/mm/yy
    ... description of event ...
  ------------------------------------------------------------------------
```
where dd/mm/yy is a date, and ... description of event ... can be a multi-line HTML fragment.
Three special one-line descriptions are recognised:
```
  Reconfig, start=hh:mm, end=hh:mm
  Maint, start=hh:mm, end=hh:mm
  VLBI, start=hh:mm, end=hh:mm
```
Here hh:mm is a local time in a 24-hr clock (note this is not AEST - a daylight savings correction is added when needed). These three one-line descriptions correspond to reconfiguration days, maintenance days and VLBI sessions. These are the most common, and are expanded into more descriptive text by outreach.pl
To generate the files needed by atca_live (both the events, and the project abstracts), use
```
  % ../scripts/outreach.pl
```
The script post_scd.pl generates a .csv file that can contain some interesting statistics. Occasionally others ask for these statistics.
The script to generate the schedulers coversheets, proj_fmt.pl uses comments from the TAC database and from the pre-grades Excel spreadsheet. These should be updated at the end of the time assignment/scheduling process, ready for next time. To update the TAC database descriptions
Using Filemaker Pro, open the ("new") TAC database on Jeeves,
```
  \\Jeeves-mm\databases\NFS_TACDB_new
```
select proposals for the relevant semester, and then export the following fields to a .csv file:
```
  Ident
  Title of proposal
  Semester
  TAC_rating
  Time_given
  TAC_comments
```
Say we saved these in newcomments.csv, then append this to the existing tacdb.csv file. Make sure read access to this is restricted, as the information is private.
```
  % cd $OPER/sched
  % cat tacdb.csv newcomments.csv > junk.csv
  % mv junk.csv tacdb.csv
  % chmod o-rw tacdb.csv
  % rm newcomments.csv
```
To save the pre-grade comments appropriately, create a .csv file in the semester directory,
```
  pregrades-sum.csv
```
from the completed pre-grades Excel spreadsheet. This .csv file should contain simply the project idents and your own comments. So use Excel to edit the pre-grades spreadsheet (discard superfluous columns and rows), and then export it as .csv format. Make sure read access to this is restricted, as the information is private.
Having updated tacdb.csv and created pregrades-sum.csv, then proj_fmt.pl will find them when the whole process starts again.
On the day that the new semester starts, you will want to edit the schedules web page to reflect the new semester, to convert the "current" schedule directory into a previous schedule directory, and rename "next" to "current".

Original: Bob Sault (28-Feb-2006)
Modified: Bob Sault (15-May-2006)