Automated Operation

How to set up your PANOPTES unit automated nightly operation

WIP: PEAS shell not used anymore, the section needs to be updated

Overview

Most of the time the unit should be started in automatic mode, which just means that POCS will make the decisions about when to start and stop operations (based on safety conditions), what targets to observe (based on the scheduler and the constraints), and how the data is handled during cleanup.

This document is assuming that you have installed the latest version of POCS which uses Docker images (like mini virtual machines) to guarantee that all the required dependencies are properly installed.

See Installing POCS for more on installation, and the Docker overview to learn more about docker.

Each Docker image is responsible for a separate "service". For example, we have a config-server service that allows us to look up configuration items. We have an aag-weather service that is responsible for the weather. And of course we have the pocs service that runs POCS and controls the unit.

Running POCS in an automated way involves starting up a few components. The docker containers start a few of these items automatically (the config server, the messaging hub, and PAWS) but a few need to be started explicitly as of this writing.

The overall process is as follows:

1. Setup the POCS environment, including configuration.

2. Start the Docker containers.

3. Load PEAS Shell to start the power and weather sensor monitoring.

4. Load POCS Shell to connect to the mount and cameras and begin observations.

This process is described in more detail in the Automatic Operation section below. Additionally, some helpful commands regarding the containers and shells are given in the Appendix at the end of this document.

Container	Purpose	Image*
config-server	Dynamic configuration server that can be queried during run-time for values.	panoptes-utils
aag-server	AAG weather data, read and plot.	panoptes-pocs
paws	Runs a copy of the PAWS website in a local Flash server.	paws
peas-shell	Environmental monitoring, including power and weather	panoptes-pocs
pocs-shell	Runs POCS itself.	panoptes-pocs

Automatic Operation

Each of the steps below should be performed in a separate terminal. You can use a tool like tmux or byobu (see Appendix for instructions) to run each terminal in the same window. This also has the advantage of making the programs long running in case the terminal window is closed. If you are using ssh to connect to the machine you will definitely want to use one of the tools so that the programs do not terminate when your connection is closed.

Setup

For first time operation it is important to create and edit a configuration file that is specific to your PANOPTES unit.

Edit Configuration File

Configuration files exist in /var/panoptes/conf_files, with an initial pocs.yaml provided by default. The config server is set to read a “local” copy of this file that will override default values. We create the local copy separate from the default so you can always return to the default.

Copy the default file to a local version. Note that the name should match pocs_local.yaml exactly.

# Go to main panoptes directory
cd $PANDIR/conf_files/

# Copy the default config file to local version
cp pocs.yaml pocs_local.yaml

You should then edit the pocs_local.yaml file for values appropriate to your unit. The main configuration items to edit are:

name: Argus    # the name of your unit, can be anything
pan_id: PAN001 # assigned pan_id, leave at PAN000 if not assigned
location:
 elevation: 3400 m       # use with ‘m’ for ‘meters’
 latitude: 19.23 deg     # use with ‘deg’ for degrees
 Longitude: -155.23 deg  # use with ‘deg’ for degrees, must include + or -
 name: MountainTopObs    # name of site location

Other potential items to edit include the devices with port numbers. The defaults should be okay but in some cases the numbers will be reversed. For example, by default the mount is on port /dev/ttyUSB0 and the weather station is /dev/ttyUSB1, but you may need to swap the two numbers. The same is true of the environment sensors.

YAML

mount:
 serial:
 port:
weather:
 aag_cloud:
 serial_port:
environment:
 camera_board:
 serial_port:
 control_board:
 serial_port:

Starting PANOPTES Services

There exists a convenience script in $POCS that will start all of the appropriate services at once. In a new terminal enter:

You should see some feedback with the starting of the different services. The command will then “block”, meaning that you cannot use the terminal again. We leave this running in the terminal the entire time we are using POCS.

Connecting to PEAS Shell

The PEAS shell is run inside the peas-shell container and can be started in a new terminal via:

This will load the “shell” environment for PEAS, which will let you enter additional PEAS-specific commands. You should see something like:

Welcome to PEAS Shell! Type ? for help PEAS >

We want to start the weather sensors, the control board sensors, and the camera board sensors:

PEAS > load_weather PEAS > load_control_board PEAS > load_camera_board

Each command you type will provide you with some brief feedback about the loading. Note that if you are testing something and only want to load one of the items that is okay. For full operation all items need to be loaded.

Once the sensors have been loaded, start the services:

This may take a few seconds. You can then check values with the following:

PEAS > last_reading control_board PEAS > last_reading camera_board PEAS > last_reading weather

Note that initial weather reading will show Unknown for most values. The weather is updated approximately every 90 seconds so you need to wait a bit to check for valid values.

PEAS Shell Commands

Below are descriptions of the most commonly used commands in PEAS shell, in the order that they might be entered. You can view the full list of available commands within PEAS by typing ?, and see what an individual command does by entering help <command>.

Tip: In both POCS and PEAS, press Tab after starting to type out a command to autocomplete it. Additionally, after typing the command followed by a space, you can press Tab twice to view any additional arguments it takes.

Loading sensors

load_control_board load_weather load_camera_board	Enter any of these commands after starting PEAS to load the respective instrument.
load_all	Shortcut to load the control board, weather station, and camera board all at once.

Controlling sensors, viewing instrument readings

start	Run this after loading instrument(s) in the previous step. PEAS will start taking measurements from these sensors.
last_reading <device>	Gets the last reading from the specified sensor. Arguments: camera_board, control_board, weather Useful for checking on your unit’s status (ie. verifying unit is parked during the daytime, from camera_board accelerometer readings) and that all instruments are behaving as expected.
turn_off_relay <device>	In the event that an instrument stops working (ie. if the camera_board stops taking readings), it may require a full power cycle. Use this command to turn off the power relay for the specified device. Arguments: fan, mount, weather, camera_box Tip: you can verify that the command worked by running last_reading control_board and checking that the current for the respective device is 0. If not, try re-entering the command.
turn_on_relay <device>	Use this command to turn on the power relay for the specified device. Usually done a few seconds after turn_off_relay when power cycling the device. Arguments: fan, mount, weather, camera_box Tip: you can verify that the command worked by running last_reading control_board and checking that the current for the respective device is nonzero. If not, try re-entering the command.

Misc.

status	Get the entire system status, pretty-printed.
toggle_debug	Toggle whether debugging messages are output to the PEAS log file for a given sensor. Arguments: camera_board, control_board, weather
help <command>	Provides a brief description of the given PEAS shell command. Arguments: see full list of commands by typing ?.

Stopping PEAS processes

stop	Stop the loop and cancel next call after running start.
exit	Exit PEAS shell completely.

Starting POCS Shell

Similar to the PEAS shell, the POCS shell is run inside the pocs-shell container and can be started in a new terminal via:

Similar to the PEAS shell, this lets you type additional POCS specific commands. You should see a prompt such as:

Welcome to POCS Shell! Type ? for help POCS >

To run in automated mode you need to set up the system and then run the system. However, before doing that you should open up the log files in another terminal so that you can see any output from the system.

View log files

In a new terminal enter:

cd $PANDIR/logs grc tail -F pocs-shell.py-all.log

Note that you may need to install grc first via sudo apt install grc.

Start POCS

First start POCS shell and then run:

POCS > setup_pocs POCS > run_pocs

The system is then officially running in an automated fashion and will wait until conditions are okay to start actual work. Note that PEAS must be running with sensors on in another terminal before you can do run_pocs.

Using simulator values

Sometimes you might want to run the system with various simulators. For instance, if you are testing your unit indoors and it is daytime, you don’t want the real weather or night setting. You can enter simulator values one at a time during setup. Here we run with simulated cameras, weather, and nighttime settings:

POCS > setup_pocs simulator=weather simulator=cameras simulator=night

Possible simulator values are: camera, dome, mount, night, power, weather

POCS Shell Commands

Below are descriptions of the most commonly used commands in POCS shell, in the order that they might be entered. You can view the full list of available commands within POCS by typing ?, and see what an individual command does by entering help <command>.

Setting up POCS

setup_pocs # Or: setup_pocs simulator=<name> simulator=<name> simulator=<name> (etc)	Setup and initialize a POCS instance. You can optionally specify hardware or conditions to simulate by adding on any number of space-separated simulator=<name> arguments to the end of the command. Possible simulator names: camera, dome, mount, night, power, weather

Manually controlling mount

unpark	To keep the unit safe during the daytime, the mount should be parked with the head unit and camera lenses facing the ground by default (ie. after POCS exits). The mount must be unparked using this command to run any of the following manual mount controls.
go_home	Send the unit to the mount’s predefined home position, which is aimed directly at either the north or south celestial pole.
polar_alignment_test	Capture images of the pole and compute alignment of mount. By default, the images from each polar alignment test will be saved to $PANDIR/images/drift_align in a timestamped directory. See the Polar Alignment guide for further instructions.
park	Send the unit back to the park position when you’re finished controlling the mount manually. Note that this should be done by default when POCS shuts down properly.

Misc.

help <command>	Provides a brief description of the given POCS shell command. Arguments: see full list of commands by typing ?.

Automated observing

run_pocs

The unit will continue to observe targets from the POCS scheduler file as long as POCS detects that conditions are safe (ie. the unit is connected to AC power, the weather is clear, it is nighttime) or until you press CTRL-C to interrupt the automated observing sequence. Note: if the unit’s cameras are in the process of taking an exposure, please wait until the exposure finishes and the shutters close before stopping the observing sequence with CTRL-C. You can watch the POCS log file output to see when the exposure finishes.

Exiting POCS

exit	Exit POCS shell completely.

Appendix

Running Containers

Starting Containers

There exists a convenience script in $POCS that will start all of the appropriate containers at once.

There are some options that can be viewed by removing the “up” parameter. Perhaps the most common option might be to run all of the containers in the background as a daemon:

This will return control to the terminal and log any output from the containers.

Container Status

The regular docker cli commands can be used on the containers. For instance, check on the status of all running containers you can:

Restart Containers

The regular docker cli commands can be used on the containers. For instance, to restart just the pocs-shell container, one would do:

docker restart pocs-shell

This might be necessary at certain times. For instance, the docker containers aren’t capable of detecting hardware changes in the underlying host system, so if the hardware changes (e.g. the mount usb is unplugged from the NUC and reattached) then the container would need to be restarted.

Stopping Containers

Individual containers can be stopped similar to the restart method however if all of the containers need to be stopped at once you can use the pocs-docker.sh script:

You should allow time for the containers to shut down gracefully.

Running POCS

PEAS Shell

The PEAS shell is run inside the peas-shell container and can be started via:

POCS Shell

The POCS shell is run inside the pocs-shell container and can be started via:

POCS Commands

Sometimes it is necessary to run commands inside a running docker container. In general, this can be accomplished via the docker cli commands:

docker exec -it pocs-shell /bin/zsh

Here the exec specifies that we want to execute something, the -it says we want an interactive terminal, pocs-shell specifies the container name we want to access, and /bin/zsh represents the command we want run inside the container. This example will start the ZSH shell terminal, allowing you to run regular linux commands.

Because it might be common to run things inside the pocs-shell container, a convenience command is provided:

cd $POCS bin/pocs-cmd jupyter-console

This will start the jupyter-console running from inside the pocs-shell container. Without any parameters passed, the command will start the ZSH shell:

Attach to byobu session

When observing, you can use byobu (byobu.org) to split your terminal into multiple smaller panels with different processes running in each. That way, you can view POCS, PEAS, and log files all at once, which is especially convenient when logging in remotely. It also allows other people on your team to join the same byobu session and see how the unit is doing.

By default, byobu should already be installed on Ubuntu. To attach, enter byobu in a new terminal window on the NUC. Then, you can jump between panels by pressing CTRL-A, letting go, then pressing an arrow key ↑ ↓ → ← in the direction you want to move. More commands (ie. splitting screens, resizing screens, creating windows etc) are listed below.

Byobu cheat sheet

CTRL-A then ? - help

CTRL-A then \ - kill byobu (& all running processes)

CTRL-A then CTRL-D - detach without kill

Windows management

CTRL-A then C - create new window

CTRL-A then K - kill current window

CTRL-A then P - previous window

CTRL-A then N - next window

CTRL-A then [window number] - jump to a specific window (see

bottom bar of terminal for all

window numbers)

Panels management

CTRL-A then % - split current panel vertically

CTRL-A then | - split current panel horizontally

CTRL-A then TAB - next panel

CTRL-A then ↑, - switch panels (with arrow keys)

CTRL-A then ↓,

CTRL-A then →,

CTRL-A then ←

Hold CTRL-A and press ↑, - resize panels vertically

Hold CTRL-A and press ↓

Hold CTRL-A and press ←, - resize panels horizontally

Hold CTRL-A and press →

Scroll up page buffer

CTRL-A then [ - scroll with ↑ ↓ arrows, then press

Enter to exit

If you’re on a team, try to make your terminal as large as possible () when using byobu, since you might restrict the size that the session appears to anyone else attached.

General observing setup

The following is a basic example of how you might apply the above information when setting a PANOPTES unit up to observe in survey mode (featuring PAN012!)

Note: add how to set scheduler and choose integration time

Step 1: Log into NUC

Either in-person or via ssh.

bStep 2: Start byobu

Step 3: Start docker containers

5Step 4: Set up byobu panels

Here, byobu panels will be set up to show PEAS, POCS, and the log files for both.

Note that this is completely up to your preference -- in the future, if there’s a process you find yourself using a lot that would be helpful to view while observing, then you can adjust the panels to add it. Likewise, if something shown here is unhelpful to you (ie. you may find yourself not using the PEAS log very often) then you can set up byobu without that window to make more space for other things.

Step 5: Start sensors with PEAS

Step 6: Start observing with POCS

Repeat for POCS. In the top right window, cd into $POCS and do bin/pocs-shell. In the bottom window, cd into $PANDIR/logs and enter the command to view the pocs-shell.py-all.log file. Then enter setup_pocs and run_pocs to start the observing process. Keep an eye out on the POCS log for errors during setup. It may take a few minutes for everything to be setup, but if there were no errors your unit will begin observing the targets listed in its scheduler file.

Note that once you’ve set your unit up with byobu, you won’t need to repeat these steps -- your setup will be saved the next time you log in, until you stop the byobu program on the NUC.