Automated Operation
How to set up your PANOPTES unit automated nightly operation
WIP: PEAS shell not used anymore, the section needs to be updated
Overview
Most of the time the unit should be started in automatic mode, which just means that POCS will make the decisions about when to start and stop operations (based on safety conditions), what targets to observe (based on the scheduler and the constraints), and how the data is handled during cleanup.
This document is assuming that you have installed the latest version of POCS which uses Docker images (like mini virtual machines) to guarantee that all the required dependencies are properly installed.
See Installing POCS for more on installation, and the Docker overview to learn more about docker.
Each Docker image is responsible for a separate "service". For example, we have a config-server
service that allows us to look up configuration items. We have an aag-weather
service that is responsible for the weather. And of course we have the pocs
service that runs POCS and controls the unit.
Running POCS in an automated way involves starting up a few components. The docker containers start a few of these items automatically (the config server, the messaging hub, and PAWS) but a few need to be started explicitly as of this writing.
The overall process is as follows:
Setup the POCS environment, including configuration.
Start the Docker containers.
Load PEAS Shell to start the power and weather sensor monitoring.
Load POCS Shell to connect to the mount and cameras and begin observations.
This process is described in more detail in the Automatic Operation section below. Additionally, some helpful commands regarding the containers and shells are given in the Appendix at the end of this document.
Container
Purpose
Image*
config-server
Dynamic configuration server that can be queried during run-time for values.
panoptes-utils
aag-server
AAG weather data, read and plot.
panoptes-pocs
paws
Runs a copy of the PAWS website in a local Flash server.
paws
peas-shell
Environmental monitoring, including power and weather
panoptes-pocs
pocs-shell
Runs POCS itself.
panoptes-pocs
Automatic Operation
Each of the steps below should be performed in a separate terminal. You can use a tool like tmux or byobu (see Appendix for instructions) to run each terminal in the same window. This also has the advantage of making the programs long running in case the terminal window is closed. If you are using ssh to connect to the machine you will definitely want to use one of the tools so that the programs do not terminate when your connection is closed.
Setup
For first time operation it is important to create and edit a configuration file that is specific to your PANOPTES unit.
Edit Configuration File
Configuration files exist in /var/panoptes/conf_files, with an initial pocs.yaml provided by default. The config server is set to read a “local” copy of this file that will override default values. We create the local copy separate from the default so you can always return to the default.
Copy the default file to a local version. Note that the name should match pocs_local.yaml exactly.
You should then edit the pocs_local.yaml file for values appropriate to your unit. The main configuration items to edit are:
Other potential items to edit include the devices with port numbers. The defaults should be okay but in some cases the numbers will be reversed. For example, by default the mount is on port /dev/ttyUSB0 and the weather station is /dev/ttyUSB1, but you may need to swap the two numbers. The same is true of the environment sensors.
Starting PANOPTES Services
There exists a convenience script in $POCS that will start all of the appropriate services at once. In a new terminal enter:
You should see some feedback with the starting of the different services. The command will then “block”, meaning that you cannot use the terminal again. We leave this running in the terminal the entire time we are using POCS.
Connecting to PEAS Shell
The PEAS shell is run inside the peas-shell container and can be started in a new terminal via:
This will load the “shell” environment for PEAS, which will let you enter additional PEAS-specific commands. You should see something like:
Welcome to PEAS Shell! Type ? for help PEAS >
We want to start the weather sensors, the control board sensors, and the camera board sensors:
PEAS > load_weather PEAS > load_control_board PEAS > load_camera_board
Each command you type will provide you with some brief feedback about the loading. Note that if you are testing something and only want to load one of the items that is okay. For full operation all items need to be loaded.
Once the sensors have been loaded, start the services:
This may take a few seconds. You can then check values with the following:
PEAS > last_reading control_board PEAS > last_reading camera_board PEAS > last_reading weather
Note that initial weather reading will show Unknown for most values. The weather is updated approximately every 90 seconds so you need to wait a bit to check for valid values.
PEAS Shell Commands
Below are descriptions of the most commonly used commands in PEAS shell, in the order that they might be entered. You can view the full list of available commands within PEAS by typing ?, and see what an individual command does by entering help <command>.
Tip: In both POCS and PEAS, press Tab after starting to type out a command to autocomplete it. Additionally, after typing the command followed by a space, you can press Tab twice to view any additional arguments it takes.
Loading sensors
load_control_board load_weather load_camera_board
Enter any of these commands after starting PEAS to load the respective instrument.
load_all
Shortcut to load the control board, weather station, and camera board all at once.
Controlling sensors, viewing instrument readings
start
Run this after loading instrument(s) in the previous step. PEAS will start taking measurements from these sensors.
last_reading <device>
Gets the last reading from the specified sensor.
Arguments: camera_board, control_board, weather
Useful for checking on your unit’s status (ie. verifying unit is parked during the daytime, from camera_board accelerometer readings) and that all instruments are behaving as expected.
turn_off_relay <device>
In the event that an instrument stops working (ie. if the camera_board stops taking readings), it may require a full power cycle. Use this command to turn off the power relay for the specified device.
Arguments: fan, mount, weather, camera_box
Tip: you can verify that the command worked by running last_reading control_board and checking that the current for the respective device is 0. If not, try re-entering the command.
turn_on_relay <device>
Use this command to turn on the power relay for the specified device. Usually done a few seconds after turn_off_relay when power cycling the device.
Arguments: fan, mount, weather, camera_box
Tip: you can verify that the command worked by running last_reading control_board and checking that the current for the respective device is nonzero. If not, try re-entering the command.
Misc.
status
Get the entire system status, pretty-printed.
toggle_debug
Toggle whether debugging messages are output to the PEAS log file for a given sensor.
Arguments: camera_board, control_board, weather
help <command>
Provides a brief description of the given PEAS shell command.
Arguments: see full list of commands by typing ?
.
Stopping PEAS processes
stop
Stop the loop and cancel next call after running start.
exit
Exit PEAS shell completely.
Starting POCS Shell
Similar to the PEAS shell, the POCS shell is run inside the pocs-shell container and can be started in a new terminal via:
Similar to the PEAS shell, this lets you type additional POCS specific commands. You should see a prompt such as:
Welcome to POCS Shell! Type ? for help POCS >
To run in automated mode you need to set up the system and then run the system. However, before doing that you should open up the log files in another terminal so that you can see any output from the system.
View log files
In a new terminal enter:
cd $PANDIR/logs grc tail -F pocs-shell.py-all.log
Note that you may need to install grc first via sudo apt install grc.
Start POCS
First start POCS shell and then run:
POCS > setup_pocs POCS > run_pocs
The system is then officially running in an automated fashion and will wait until conditions are okay to start actual work. Note that PEAS must be running with sensors on in another terminal before you can do run_pocs.
Using simulator values
Sometimes you might want to run the system with various simulators. For instance, if you are testing your unit indoors and it is daytime, you don’t want the real weather or night setting. You can enter simulator values one at a time during setup. Here we run with simulated cameras, weather, and nighttime settings:
POCS > setup_pocs simulator=weather simulator=cameras simulator=night
Possible simulator values are: camera, dome, mount, night, power, weather
POCS Shell Commands
Below are descriptions of the most commonly used commands in POCS shell, in the order that they might be entered. You can view the full list of available commands within POCS by typing ?, and see what an individual command does by entering help <command>.
Setting up POCS
setup_pocs
# Or:
setup_pocs simulator=<name> simulator=<name> simulator=<name>
(etc)
Setup and initialize a POCS instance. You can optionally specify hardware or conditions to simulate by adding on any number of space-separated simulator=<name> arguments to the end of the command.
Possible simulator names: camera, dome, mount, night, power, weather
Manually controlling mount
unpark
To keep the unit safe during the daytime, the mount should be parked with the head unit and camera lenses facing the ground by default (ie. after POCS exits).
The mount must be unparked using this command to run any of the following manual mount controls.
go_home
Send the unit to the mount’s predefined home position, which is aimed directly at either the north or south celestial pole.
polar_alignment_test
park
Send the unit back to the park position when you’re finished controlling the mount manually. Note that this should be done by default when POCS shuts down properly.
Misc.
help <command>
Provides a brief description of the given POCS shell command.
Arguments: see full list of commands by typing ?.
Automated observing
run_pocs
The unit will continue to observe targets from the POCS scheduler file as long as POCS detects that conditions are safe (ie. the unit is connected to AC power, the weather is clear, it is nighttime) or until you press CTRL-C to interrupt the automated observing sequence.
Note: if the unit’s cameras are in the process of taking an exposure, please wait until the exposure finishes and the shutters close before stopping the observing sequence with CTRL-C. You can watch the POCS log file output to see when the exposure finishes.
Exiting POCS
exit
Exit POCS shell completely.
Appendix
Running Containers
Starting Containers
There exists a convenience script in $POCS that will start all of the appropriate containers at once.
There are some options that can be viewed by removing the “up” parameter. Perhaps the most common option might be to run all of the containers in the background as a daemon:
This will return control to the terminal and log any output from the containers.
Container Status
The regular docker cli commands can be used on the containers. For instance, check on the status of all running containers you can:
Restart Containers
The regular docker cli commands can be used on the containers. For instance, to restart just the pocs-shell container, one would do:
docker restart pocs-shell
This might be necessary at certain times. For instance, the docker containers aren’t capable of detecting hardware changes in the underlying host system, so if the hardware changes (e.g. the mount usb is unplugged from the NUC and reattached) then the container would need to be restarted.
Stopping Containers
Individual containers can be stopped similar to the restart method however if all of the containers need to be stopped at once you can use the pocs-docker.sh script:
You should allow time for the containers to shut down gracefully.
Running POCS
PEAS Shell
The PEAS shell is run inside the peas-shell container and can be started via:
POCS Shell
The POCS shell is run inside the pocs-shell container and can be started via:
POCS Commands
Sometimes it is necessary to run commands inside a running docker container. In general, this can be accomplished via the docker cli commands:
docker exec -it pocs-shell /bin/zsh
Here the exec specifies that we want to execute something, the -it says we want an interactive terminal, pocs-shell specifies the container name we want to access, and /bin/zsh represents the command we want run inside the container. This example will start the ZSH shell terminal, allowing you to run regular linux commands.
Because it might be common to run things inside the pocs-shell container, a convenience command is provided:
cd $POCS bin/pocs-cmd jupyter-console
This will start the jupyter-console running from inside the pocs-shell container. Without any parameters passed, the command will start the ZSH shell:
Attach to byobu session
When observing, you can use byobu (byobu.org) to split your terminal into multiple smaller panels with different processes running in each. That way, you can view POCS, PEAS, and log files all at once, which is especially convenient when logging in remotely. It also allows other people on your team to join the same byobu session and see how the unit is doing.
By default, byobu should already be installed on Ubuntu. To attach, enter byobu in a new terminal window on the NUC. Then, you can jump between panels by pressing CTRL-A, letting go, then pressing an arrow key ↑ ↓ → ← in the direction you want to move. More commands (ie. splitting screens, resizing screens, creating windows etc) are listed below.
Byobu cheat sheet
CTRL-A then ? - help
CTRL-A then \ - kill byobu (& all running processes)
CTRL-A then CTRL-D - detach without kill
Windows management
CTRL-A then C - create new window
CTRL-A then K - kill current window
CTRL-A then P - previous window
CTRL-A then N - next window
CTRL-A then [window number] - jump to a specific window (see
bottom bar of terminal for all
window numbers)
Panels management
CTRL-A then % - split current panel vertically
CTRL-A then | - split current panel horizontally
CTRL-A then TAB - next panel
CTRL-A then ↑, - switch panels (with arrow keys)
CTRL-A then ↓,
CTRL-A then →,
CTRL-A then ←
Hold CTRL-A and press ↑, - resize panels vertically
Hold CTRL-A and press ↓
Hold CTRL-A and press ←, - resize panels horizontally
Hold CTRL-A and press →
Scroll up page buffer
CTRL-A then [ - scroll with ↑ ↓ arrows, then press
Enter to exit
General observing setup
The following is a basic example of how you might apply the above information when setting a PANOPTES unit up to observe in survey mode (featuring PAN012!)
Note: add how to set scheduler and choose integration time
Step 1: Log into NUC
Either in-person or via ssh.
bStep 2: Start byobu
Step 3: Start docker containers
5Step 4: Set up byobu panels
Here, byobu panels will be set up to show PEAS, POCS, and the log files for both.
Note that this is completely up to your preference -- in the future, if there’s a process you find yourself using a lot that would be helpful to view while observing, then you can adjust the panels to add it. Likewise, if something shown here is unhelpful to you (ie. you may find yourself not using the PEAS log very often) then you can set up byobu without that window to make more space for other things.
Step 5: Start sensors with PEAS
Step 6: Start observing with POCS
Repeat for POCS. In the top right window, cd into $POCS and do bin/pocs-shell. In the bottom window, cd into $PANDIR/logs and enter the command to view the pocs-shell.py-all.log file. Then enter setup_pocs and run_pocs to start the observing process. Keep an eye out on the POCS log for errors during setup. It may take a few minutes for everything to be setup, but if there were no errors your unit will begin observing the targets listed in its scheduler file.
Note that once you’ve set your unit up with byobu, you won’t need to repeat these steps -- your setup will be saved the next time you log in, until you stop the byobu program on the NUC.
Last updated