Science Platform operations

The Rubin Science Platform is described in LDM-542. This document contains operational notes of interest to administrators of the Science Platform and maintainers of services deployed via the Science Platform, but not of interest to users.

For user documentation of the Notebook Aspect of the Rubin Science Platform, see nb.lsst.io.

The Science Platform uses Argo CD to manage its Kubernetes resources. The Argo CD configuration and this documentation are maintained on GitHub.

A phalanx is a SQuaRE deployment (Science Quality and Reliability Engineering, the team responsible for the Rubin Science Platform). Phalanx is how we ensure that all of our services work together as a unit.

Overview

• Introduction to Kubernetes and Argo CD
• Repository structure
  • Layout
  • Charts
  • Chart versioning
• Secrets
  • Vault
  • 1Password

For service maintainers

General development and operations

• Setting up pre-commit linting and helm-docs generation
  • Install pre-commit and helm-docs locally
  • What to expect when developing in Phalanx with pre-commit
  • Running all files
  • By-passing pre-commit
• Create a new service
  • Write the service
  • Create the Docker image
  • Create the Helm chart
• Add a secret with 1Password and VaultSecret
  • Part 1. Open the 1Password vault
  • Part 2. Create a Secret Note
  • Part 3. Sync 1Password items into Vault
  • Next steps: connecting Vault to Kubernetes with VaultSecret
• Updating a secret stored in 1Password and VaultSecret
  • Forcing reconciliation
• Updating the pull secret stored in 1Password and Vault
• Add a new service to Phalanx
  • Add the wrapper chart
  • Add the Argo CD application
• Adding an external Helm chart
  • Potential problems
  • Configure the external chart
  • Tell Argo CD about the upstream Helm repository
• Set up a local development environment with minikube
  • Start minikube
  • Deploy the minikube environment
  • Access the Argo CD UI
• Syncing Argo CD
  • Go to Argo CD for the environment
  • Sync the application
  • Revert to a previous configuration
• Upgrading a service
• Changing charts and phalanx together

Specific tasks

• Update the TAP_SCHEMA table
• Managing mobu flocks

For science platform administrators

Services

• Argo CD
  • Upgrading Argo CD
  • Argo CD authentication
• cachemachine
  • Image pruning
  • Updating “recommended”
• cert-manager
  • Setting up Route 53 for cert-manager
  • Bootstrapping cert-manager
• Gafaelfawr
  • Debugging authentication issues
  • Configuring storage
  • Recreating Gafaelfawr service tokens
• ingress-nginx
  • TLS certificates
• nublado2
  • Clear session database entry
• postgres
  • Recreating postgres PV/PVC
  • Adding a new database
• squash-api
• TAP
• vault-secrets-operator

Bootstrapping

• Bootstrapping a new deployment
  • Requirements
  • Checklist
  • Hostnames and TLS
  • Service notes
    • Gafaelfawr
    • Nublado 2
    • Portal
    • Squareone

Infrastructure

• Filestore
  • Privileged access to the filestore

Troubleshooting

• Troubleshooting the Rubin Science Platform
  • PostgreSQL cannot mount its persistent volume
  • Spawner menu missing images, cachemachine stuck pulling the same image
  • Spawner menu shows empty parentheses after recommended rather than image tag
  • Spawning a notebook fails with a pending error
  • User gets permission denied from services
  • You need privileged access to the filestore
  • User pods don’t spawn, reporting “permission denied” from Moneypenny