This loads a font easier to read for people with dyslexia.
This renders the document in high contrast mode.
This renders the document as white on black
This can help those with trouble processing rapid screen movements.
Decrease text size Increase text size Reset text size
Accessibility menu
Search icon
Menu

BIGCAT Change Log

2025-10-30

Changes to caobs

  • Made another attempt at fixing a bug that may occasionally cause the correlator to be told to run, but not have it produce data, by limiting the Pause command to only be issued when the correlator is actually running.
  • Fix an issue where the mm LO would not be set after the first initial setting.

Changes to correlator

  • Fixed runcorr.py to clear old information from previous configure messages when receiving new messages. This bug I think was responsible for runcorr crashes when switching to 16cm, where fewer subbands are specified; without clearing the old information, runcorr was launching correlators that were not included in the Config message, and thus would immediately exit with an error.

Known Bugs

  • Caobs does not handle malformed schedules very gracefully, and has been known to crash when told to load them. We advise all observers to carefully ensure that their schedule file is valid. A common issue is when a UTC schedule is specified but not all scans are labelled with UTC times. For most normal observations, UTC schedules are not what you want anyway, and you should choose relative timing in the scheduler.
  • Issuing a "corr stop" in caobs before the correlator has started (ie. shortly after the first start/track command is given) will always fail, and make it difficult/impossible to control the correlator without restarting everything. To avoid this problem, we recommend observers carefully start their schedule so they don't need to stop and restart immediately, and if they do make a mistake, to either restart without "corr stop" or to wait for a minute before doing that.
  • The 7mm delay solution has not yet been tested.
  • SSPD has a severe memory leak when asked to produce and display integrated data, and so should not be used to do so.

Software versions used

  • tag: production_2025-10-30
  • mc: cdebdcd4103db89776e2824ad42789b4271d5a09
  • correlator: 798ee3252bd28fd0988346262141854bae76253d
  • ingest: 9b7db7b3a50e496ce8859158c1dc1bd82f5089bc

2025-10-28

Changes to caobs

  • Fixed a bug that might have been causing the correlator to run but not produce data, which is a failure mode that we have seen. Caobs now no longer issues the same scan number more than once per correlator run.
  • Enable Update pointing scans to work properly, along with the Offset pointing type. The pointing corrections made are now broadcast to clients and shown in the caobs command-line client.

Changes to correlator

  • Move to the correlator version that supports VLBI tied-array mode. No changes to non-VLBI operations should be noticed.

Other changes

  • Corrected 15mm delay model.
  • Fixed phase corrections for caobs, so now all frequency combinations within the 16cm, 4cm and 15mm bands should look alright, with no phase discontinuities.

Known Bugs

  • Caobs does not handle malformed schedules very gracefully, and has been known to crash when told to load them. We advise all observers to carefully ensure that their schedule file is valid. A common issue is when a UTC schedule is specified but not all scans are labelled with UTC times. For most normal observations, UTC schedules are not what you want anyway, and you should choose relative timing in the scheduler.
  • Issuing a "corr stop" in caobs before the correlator has started (ie. shortly after the first start/track command is given) will always fail, and make it difficult/impossible to control the correlator without restarting everything. To avoid this problem, we recommend observers carefully start their schedule so they don't need to stop and restart immediately, and if they do make a mistake, to either restart without "corr stop" or to wait for a minute before doing that.
  • The 7mm delay solution has not yet been tested.
  • SSPD has a severe memory leak when asked to produce and display integrated data, and so should not be used to do so.

Software versions used

  • tag: production_2025-10-28
  • mc: 09b925f0be505ade6fd0f6dc48e9e73a04b2778a
  • correlator: 5fa82d1f9182a8ca598a697fb8796e9a288a3953
  • ingest: 9b7db7b3a50e496ce8859158c1dc1bd82f5089bc

2025-10-23 Revision 2

After observations using the production_2025-10-23 software showed that some of the bugs that we thought were fixed were not, we made some further alterations in the green time starting at 07:30 UTC.

Changes to caobs

  • Fixed a bug that was causing scan dataset labelling issues when sending scans in advance, which is normal behaviour. This doesn't seem to have caused any issues with the output data, but was causing error messages from ingest.

Changes to ingest

  • Ingest now seems to properly close scans when it should. Previous testing was done only by stopping and starting scans, not running the schedule with multiple scans in the same execution block, but now both modes of observing seem to work as intended.

Changes to correlator

  • Fix runcorr.py to launch only those GPU streams that are requested by caobs. While runcorr.py continues to output messages saying that a GPU stream it supports was not requested, this no longer causes runcorr.py to exit.

Known Bugs

  • Caobs does not handle malformed schedules very gracefully, and has been known to crash when told to load them. We advise all observers to carefully ensure that their schedule file is valid. A common issue is when a UTC schedule is specified but not all scans are labelled with UTC times. For most normal observations, UTC schedules are not what you want anyway, and you should choose relative timing in the scheduler.
  • Issuing a "corr stop" in caobs before the correlator has started (ie. shortly after the first start/track command is given) will always fail, and make it difficult/impossible to control the correlator without restarting everything. To avoid this problem, we recommend observers carefully start their schedule so they don't need to stop and restart immediately, and if they do make a mistake, to either restart without "corr stop" or to wait for a minute before doing that.
  • Caobs can do pointing scans and determine the az/el offsets from them, but cannot use the updated model for scans that ask for the updated model. This is less a bug than a feature that has yet to be implemented.
  • The mm delay solution has not yet been tested.
  • SSPD has a severe memory leak when asked to produce and display integrated data, and so should not be used to do so.

Software versions used

  • tag: production_2025-10-23_rev2
  • mc: 51700b07ec8883a66a2eb55689d76bb8e2af724e
  • correlator: 30267297dde3beedf70b7e35bac38bbc402968a5 (caveat: only the runcorr.py changes from the previous commit have been incorporated, the old correlator binaries are still being used)
  • ingest: 9b7db7b3a50e496ce8859158c1dc1bd82f5089bc

2025-10-23

Changes to caobs

  • Fixed the issue that was causing on-source flagging problems when changing bands.
  • Improved handling of start/stop/starts before the correlator is properly configured by the first start. This should mean the system is much less likely to lose control of the correlator processes.
  • Fixed the caobs_server launcher on skull, to make it less likely to launch more than one server at a time.

Changes to ingest

  • Fixed issues that were preventing correct writing of more than one frequency configuration to a single dataset; this should (and has been tested to) work now.
  • Ingest now is much better at closing scans in the dataset.
  • Changed the output of ingest in the terminal to be less unnecessarily verbose.

Other changes

  • Corrected 16cm delay model.

Known Bugs

  • Issuing a "corr stop" in caobs before the correlator has started (ie. shortly after the first start/track command is given) will always fail, and make it difficult/impossible to control the correlator without restarting everything. To avoid this problem, we recommend observers carefully start their schedule so they don't need to stop and restart immediately, and if they do make a mistake, to either restart without "corr stop" or to wait for a minute before doing that.
  • Ingest sometimes fails to close scans properly in the output (A)SDM, although the situation has improved in this software release. This results in datasets that cannot be converted to MeasurementSet. Thankfully, the fix for this is easy, using the fixasdm.py script on panthera-ingest.
  • Caobs can do pointing scans and determine the az/el offsets from them, but cannot use the updated model for scans that ask for the updated model. This is less a bug than a feature that has yet to be implemented.
  • The mm delay solution has not yet been tested.
  • SSPD has a severe memory leak when asked to produce and display integrated data, and so should not be used to do so.
  • Runcorr.py will exit if a Config message is sent without the subbands that it is controlling. This is not what we want most of the time.

Software versions used

  • tag: production_2025-10-23
  • mc: 9f6bdf7ddcbdb1f72ef3c9c78befce6bdd9a3605
  • correlator: 2ac59b18dbaf5cbd534f84e21b0add36a7be03d2
  • ingest: 9b61c29539696d96ddb9f57a5a71e684a9be8b67

2025-10-21

BIGCAT operations begin.

Known Bugs

  • Ingest does not support more than a single frequency configuration in a dataset
  • Ingest regularly fails to close scans properly in the output (A)SDM. This results in datasets that cannot be converted to MeasurementSet. Thankfully, the fix for this is easy, using the fixasdm.py script on panthera-ingest.
  • Caobs fails to control the correlator if the first scan that launches the correlator is stopped before the correlator is properly instructed to begin processing data. The only fix for this is to shut everything down and restart it.
  • Caobs cannot properly tell when the antenna is on-source after instructing the antenna to change receivers. The fix for this is to stop the scan and restart it, at which point caobs will recognise that the correct receiver is on axis (if it is).
  • Caobs can do pointing scans and determine the az/el offsets from them, but cannot use the updated model for scans that ask for the updated model. This is less a bug than a feature that has yet to be implemented.
  • The 16cm delay solution is not very good, and the mm delay solution has not yet been tested.
  • SSPD has a severe memory leak when asked to produce and display integrated data, and so should not be used to do so.
  • It is easily possible to get the system into an uncontrollable state by starting more than one caobs_server.
  • Runcorr.py will exit if a Config message is sent without the subbands that it is controlling. This is not what we want most of the time.

Original: Jamie Stevens (23-Oct-2025)
Modified: Jamie Stevens (23-Oct-2025)