Administrators

A Phalanx administrator is someone who is responsible for defining a Phalanx environment, installing Phalanx in that environment, and maintaining the resulting cluster. Administrators operate infrastructure, manage secrets, and are involved in the deployment, configuration, and Argo CD synchronization of applications.

Note

If you are maintaining a Phalanx environment, contact SQuaRE to be added as a contributor to the lsst-sqre/phalanx repository. Some of the steps below may require pushing your development branches to that repository rather than using a fork.

Initial install

• Phalanx requirements
• Hostnames and TLS
• Create a new Phalanx environment
• Set up secrets management
• Installing a Phalanx environment

Procedures

• Upgrade windows and sequence
• Syncing Argo CD in an environment
• Add a new static secret
• Update a static secret
• Sync secrets for an environment
• Audit secrets for an environment
• Updating the pull secret for an environment
• Migrating to the new secrets management system

Troubleshooting

• Troubleshooting the Rubin Science Platform
  • PostgreSQL cannot mount its persistent volume
  • Spawner menu missing images, Nublado stuck pulling the same image
  • User gets permission denied from applications
  • You need privileged access to the filestore
  • Login fails with “bad verification code” error
  • User keeps logging in through the wrong identity provider
  • User deleted from COmanage and needs to be restored
• Communicating an Outage
  • Putting up a banner
  • Slack

Infrastructure

• Using Google Kubernetes Engine
  • Getting GKE Kubernetes credentials
  • Managing GCP resources with Terraform
  • Set up workload identity
• Filestore
  • Privileged access to the filestore
• Kubernetes kubelet nodeStatusMaxImages setting for Nublado
  • Setting nodeStatusMaxImages
  • Background
  • Pruning cached images

Reference

• Command-line interface