Automated Operation
How to set up your PANOPTES unit automated nightly operation
WIP: PEAS shell not used anymore, the section needs to be updated
Overview
Most of the time the unit should be started in automatic mode, which just means that POCS will make the decisions about when to start and stop operations (based on safety conditions), what targets to observe (based on the scheduler and the constraints), and how the data is handled during cleanup.
This document is assuming that you have installed the latest version of POCS which uses Docker images (like mini virtual machines) to guarantee that all the required dependencies are properly installed.
See Installing POCS for more on installation, and the Docker overview to learn more about docker.
Each Docker image is responsible for a separate "service". For example, we have a config-server
service that allows us to look up configuration items. We have an aag-weather
service that is responsible for the weather. And of course we have the pocs
service that runs POCS and controls the unit.
Running POCS in an automated way involves starting up a few components. The docker containers start a few of these items automatically (the config server, the messaging hub, and PAWS) but a few need to be started explicitly as of this writing.
The overall process is as follows:
Setup the POCS environment, including configuration.
Start the Docker containers.
Load PEAS Shell to start the power and weather sensor monitoring.
Load POCS Shell to connect to the mount and cameras and begin observations.
This process is described in more detail in the Automatic Operation section below. Additionally, some helpful commands regarding the containers and shells are given in the Appendix at the end of this document.
Automatic Operation
Each of the steps below should be performed in a separate terminal. You can use a tool like tmux or byobu (see Appendix for instructions) to run each terminal in the same window. This also has the advantage of making the programs long running in case the terminal window is closed. If you are using ssh to connect to the machine you will definitely want to use one of the tools so that the programs do not terminate when your connection is closed.
Setup
For first time operation it is important to create and edit a configuration file that is specific to your PANOPTES unit.
Edit Configuration File
Configuration files exist in /var/panoptes/conf_files, with an initial pocs.yaml provided by default. The config server is set to read a “local” copy of this file that will override default values. We create the local copy separate from the default so you can always return to the default.
Copy the default file to a local version. Note that the name should match pocs_local.yaml exactly.
You should then edit the pocs_local.yaml file for values appropriate to your unit. The main configuration items to edit are:
Other potential items to edit include the devices with port numbers. The defaults should be okay but in some cases the numbers will be reversed. For example, by default the mount is on port /dev/ttyUSB0 and the weather station is /dev/ttyUSB1, but you may need to swap the two numbers. The same is true of the environment sensors.
Starting PANOPTES Services
There exists a convenience script in $POCS that will start all of the appropriate services at once. In a new terminal enter:
You should see some feedback with the starting of the different services. The command will then “block”, meaning that you cannot use the terminal again. We leave this running in the terminal the entire time we are using POCS.
Connecting to PEAS Shell
The PEAS shell is run inside the peas-shell container and can be started in a new terminal via:
This will load the “shell” environment for PEAS, which will let you enter additional PEAS-specific commands. You should see something like:
Welcome to PEAS Shell! Type ? for help PEAS >
We want to start the weather sensors, the control board sensors, and the camera board sensors:
PEAS > load_weather PEAS > load_control_board PEAS > load_camera_board
Each command you type will provide you with some brief feedback about the loading. Note that if you are testing something and only want to load one of the items that is okay. For full operation all items need to be loaded.
Once the sensors have been loaded, start the services:
This may take a few seconds. You can then check values with the following:
PEAS > last_reading control_board PEAS > last_reading camera_board PEAS > last_reading weather
Note that initial weather reading will show Unknown for most values. The weather is updated approximately every 90 seconds so you need to wait a bit to check for valid values.
PEAS Shell Commands
Below are descriptions of the most commonly used commands in PEAS shell, in the order that they might be entered. You can view the full list of available commands within PEAS by typing ?, and see what an individual command does by entering help <command>.
Tip: In both POCS and PEAS, press Tab after starting to type out a command to autocomplete it. Additionally, after typing the command followed by a space, you can press Tab twice to view any additional arguments it takes.
Loading sensors
Controlling sensors, viewing instrument readings
Misc.
Stopping PEAS processes
Starting POCS Shell
Similar to the PEAS shell, the POCS shell is run inside the pocs-shell container and can be started in a new terminal via:
Similar to the PEAS shell, this lets you type additional POCS specific commands. You should see a prompt such as:
Welcome to POCS Shell! Type ? for help POCS >
To run in automated mode you need to set up the system and then run the system. However, before doing that you should open up the log files in another terminal so that you can see any output from the system.
View log files
In a new terminal enter:
cd $PANDIR/logs grc tail -F pocs-shell.py-all.log
Note that you may need to install grc first via sudo apt install grc.
Start POCS
First start POCS shell and then run:
POCS > setup_pocs POCS > run_pocs
The system is then officially running in an automated fashion and will wait until conditions are okay to start actual work. Note that PEAS must be running with sensors on in another terminal before you can do run_pocs.
Using simulator values
Sometimes you might want to run the system with various simulators. For instance, if you are testing your unit indoors and it is daytime, you don’t want the real weather or night setting. You can enter simulator values one at a time during setup. Here we run with simulated cameras, weather, and nighttime settings:
POCS > setup_pocs simulator=weather simulator=cameras simulator=night
Possible simulator values are: camera, dome, mount, night, power, weather
POCS Shell Commands
Below are descriptions of the most commonly used commands in POCS shell, in the order that they might be entered. You can view the full list of available commands within POCS by typing ?, and see what an individual command does by entering help <command>.
Setting up POCS
Manually controlling mount
Misc.
Automated observing
Exiting POCS
Appendix
Running Containers
Starting Containers
There exists a convenience script in $POCS that will start all of the appropriate containers at once.
There are some options that can be viewed by removing the “up” parameter. Perhaps the most common option might be to run all of the containers in the background as a daemon:
This will return control to the terminal and log any output from the containers.
Container Status
The regular docker cli commands can be used on the containers. For instance, check on the status of all running containers you can:
Restart Containers
The regular docker cli commands can be used on the containers. For instance, to restart just the pocs-shell container, one would do:
docker restart pocs-shell
This might be necessary at certain times. For instance, the docker containers aren’t capable of detecting hardware changes in the underlying host system, so if the hardware changes (e.g. the mount usb is unplugged from the NUC and reattached) then the container would need to be restarted.
Stopping Containers
Individual containers can be stopped similar to the restart method however if all of the containers need to be stopped at once you can use the pocs-docker.sh script:
You should allow time for the containers to shut down gracefully.
Running POCS
PEAS Shell
The PEAS shell is run inside the peas-shell container and can be started via:
POCS Shell
The POCS shell is run inside the pocs-shell container and can be started via:
POCS Commands
Sometimes it is necessary to run commands inside a running docker container. In general, this can be accomplished via the docker cli commands:
docker exec -it pocs-shell /bin/zsh
Here the exec specifies that we want to execute something, the -it says we want an interactive terminal, pocs-shell specifies the container name we want to access, and /bin/zsh represents the command we want run inside the container. This example will start the ZSH shell terminal, allowing you to run regular linux commands.
Because it might be common to run things inside the pocs-shell container, a convenience command is provided:
cd $POCS bin/pocs-cmd jupyter-console
This will start the jupyter-console running from inside the pocs-shell container. Without any parameters passed, the command will start the ZSH shell:
Attach to byobu session
When observing, you can use byobu (byobu.org) to split your terminal into multiple smaller panels with different processes running in each. That way, you can view POCS, PEAS, and log files all at once, which is especially convenient when logging in remotely. It also allows other people on your team to join the same byobu session and see how the unit is doing.
By default, byobu should already be installed on Ubuntu. To attach, enter byobu in a new terminal window on the NUC. Then, you can jump between panels by pressing CTRL-A, letting go, then pressing an arrow key ↑ ↓ → ← in the direction you want to move. More commands (ie. splitting screens, resizing screens, creating windows etc) are listed below.
Byobu cheat sheet
CTRL-A then ? - help
CTRL-A then \ - kill byobu (& all running processes)
CTRL-A then CTRL-D - detach without kill
Windows management
CTRL-A then C - create new window
CTRL-A then K - kill current window
CTRL-A then P - previous window
CTRL-A then N - next window
CTRL-A then [window number] - jump to a specific window (see
bottom bar of terminal for all
window numbers)
Panels management
CTRL-A then % - split current panel vertically
CTRL-A then | - split current panel horizontally
CTRL-A then TAB - next panel
CTRL-A then ↑, - switch panels (with arrow keys)
CTRL-A then ↓,
CTRL-A then →,
CTRL-A then ←
Hold CTRL-A and press ↑, - resize panels vertically
Hold CTRL-A and press ↓
Hold CTRL-A and press ←, - resize panels horizontally
Hold CTRL-A and press →
Scroll up page buffer
CTRL-A then [ - scroll with ↑ ↓ arrows, then press
Enter to exit
General observing setup
The following is a basic example of how you might apply the above information when setting a PANOPTES unit up to observe in survey mode (featuring PAN012!)
Note: add how to set scheduler and choose integration time
Step 1: Log into NUC
Either in-person or via ssh.
bStep 2: Start byobu
Step 3: Start docker containers
5Step 4: Set up byobu panels
Here, byobu panels will be set up to show PEAS, POCS, and the log files for both.
Note that this is completely up to your preference -- in the future, if there’s a process you find yourself using a lot that would be helpful to view while observing, then you can adjust the panels to add it. Likewise, if something shown here is unhelpful to you (ie. you may find yourself not using the PEAS log very often) then you can set up byobu without that window to make more space for other things.
Step 5: Start sensors with PEAS
Step 6: Start observing with POCS
Repeat for POCS. In the top right window, cd into $POCS and do bin/pocs-shell. In the bottom window, cd into $PANDIR/logs and enter the command to view the pocs-shell.py-all.log file. Then enter setup_pocs and run_pocs to start the observing process. Keep an eye out on the POCS log for errors during setup. It may take a few minutes for everything to be setup, but if there were no errors your unit will begin observing the targets listed in its scheduler file.
Note that once you’ve set your unit up with byobu, you won’t need to repeat these steps -- your setup will be saved the next time you log in, until you stop the byobu program on the NUC.
Last updated