Science Platform operations

The Rubin Science Platform is described in LDM-542. This document contains operational notes of interest to administrators of the Science Platform and maintainers of services deployed via the Science Platform, but not of interest to users.

For user documentation of the Notebook Aspect of the Rubin Science Platform, see nb.lsst.io.

The Science Platform uses Argo CD to manage its Kubernetes resources. The Argo CD configuration and this documentation are maintained on GitHub.

A phalanx is a SQuaRE deployment (Science Quality and Reliability Engineering, the team responsible for the Rubin Science Platform). Phalanx is how we ensure that all of our services work together as a unit.

Overview

• Introduction to Kubernetes and Argo CD
• Repository structure
  • Layout
  • Charts
  • Chart versioning
• Secrets
  • Vault
  • 1Password

For service maintainers

General development and operations

• Create a new service
  • Write the service
  • Create the Docker image
  • Create the Helm chart
• Add a secret with 1Password and VaultSecret
  • Part 1. Open the 1Password vault
  • Part 2. Create a Secret Note
  • Part 3. Sync 1Password items into Vault
  • Next steps: connecting Vault to Kubernetes with VaultSecret
• Updating a secret stored in 1Password and VaultSecret
  • Forcing reconciliation
• Add a new service to Phalanx
  • Add the wrapper chart
  • Add the Argo CD application
• Adding an external Helm chart
  • Potential problems
  • Configure the external chart
  • Tell Argo CD about the upstream Helm repository
• Set up a local development environment with minikube
  • Start minikube
  • Deploy the minikube environment
  • Access the Argo CD UI
• Syncing Argo CD
  • Go to Argo CD for the environment
  • Sync the application
  • Revert to a previous configuration
• Upgrading a service
• Changing charts and phalanx together

Specific tasks

• Update the TAP_SCHEMA table
• Managing mobu flocks

For science platform administrators

Services

• Argo CD
  • Upgrading Argo CD
  • Argo CD authentication
• cachemachine
  • Image pruning
  • Updating “recommended”
• cert-issuer
  • Setting up Route 53 for cert-issuer
  • Bootstrapping cert-issuer
• cert-manager
• Gafaelfawr
  • Debugging authentication issues
  • Configuring storage
  • Recreating Gafaelfawr service tokens
• ingress-nginx
  • TLS certificates
• nublado2
  • Clear session database entry
• postgres
  • Recreating postgres PV/PVC
• squash-api
• TAP
• vault-secrets-operator

Bootstrapping

• Bootstrapping a new deployment
  • Requirements
  • Checklist
  • Hostnames and TLS
  • Service notes
    • Gafaelfawr
    • Nublado 2
    • Portal
    • Squareone

Infrastructure

• Filestore
  • Privileged access to the filestore

Troubleshooting

• Troubleshooting the Rubin Science Platform
  • PostgreSQL cannot mount its persistent volume
  • Spawner menu missing images, cachemachine stuck pulling the same image
  • Spawner menu shows empty parentheses after recommended rather than image tag
  • Spawning a notebook fails with a pending error
  • User gets permission denied from services
  • You need privileged access to the filestore
  • User pods don’t spawn, reporting “permission denied” from Moneypenny