Troubleshooting the Rubin Science Platform¶
Intended audience: Anyone who is administering an installation of the Rubin Science Platform.
Sometimes things break, and we are assembling the most common failure scenarios, and their fixes, in this document.
PostgreSQL cannot mount its persistent volume¶
Symptoms: When restarted, the postgres service pod fails to start because it cannot mount its persistent volume.
If the pod is already running, it gets I/O errors from its database, hangs, or otherwise shows signs of storage problems.
Cause: The postgres deployment requests a PersistentVolume via a PersistentVolumeClaim.
If the backing store is corrupt or has been deleted or otherwise is disrupted, sometimes the PersistentVolume will become unavailable, but the PersistentVolumeClaim will hang on to it and keep trying to futilely mount it.
When this happens, you may need to recreate the persistent volume.
Solution: Recreating postgres PV/PVC
Spawning a notebook fails with a pending error¶
Symptoms: When a user tries to spawn a new notebook, the spawn fails with an error saying that the user’s lab is already pending spawn or is pending deletion.
Cause: If the spawning of the lab fails or if the deletion of a lab fails, sometimes JupyterHub can give up on making further progress but still remember that the lab is supposedly still running.
In this case, JupyterHub may not recover without assistance.
You may need to delete the record for the affected user, and also make sure the user’s lab namespace (visible in Argo CD under the nublado-users application) has been deleted.
Solution: Clear session database entry
User gets permission denied from services¶
Symptoms: A user is able to authenticate to the Rubin Science Platform (prompted by going to the first authenticated URL, such as the Notebook Aspect spawner page), but then gets permission denied from other services.
Causes: Authentication and authorization to the Rubin Science Platform is done via a service called Gafaelfawr (see Gafaelfawr).
After the user authenticates, Gafaelfawr asks their authentication provider for the user’s group memberships and then translates that to a list of scopes.
The mapping of group memberships to scopes is defined in the values.yaml file for Gafaelfawr for the relevant environment, in the gafaelfawr.config.groupMapping configuration option.
The most likely cause of this problem is that the user is not a member of a group that grants them access to that service. Gafaelfawr will prevent the user from logging in at all if they are not a member of any group that grants access to a service. If they are a member of at least one group, they’ll be able to log in but may get permission denied errors from other services.
Solution: Debugging authentication issues
You need privileged access to the filestore¶
Symptoms: You need to do something like copy data from one instance to another, or to get a report of per-user usage on the filestore, or create a new non-world-writeable section under the filestore.
Causes: The RSP intentionally only lets you access a pod as an unprivileged user. If you need to do something that spans users or should create root-owned files, you will need some way of accessing the filestore-presented filesystem with privilege.
Solution: Privileged access to the filestore
User pods don’t spawn, reporting “permission denied” from Moneypenny¶
Symptoms: A user pod fails to spawn, and the error message says that Moneypenny did not have permission to execute.
Cause: The gafaelfawr-token VaultSecret in the nublado2 namespace is out of date.
This happened because the gafaelfawr-redis pod restarted and either it lacked persistent storage (at the T&S sites, as of July 2022), or because that storage had been lost.
Solution: Recreating Gafaelfawr service tokens
Login fails with “bad verification code” error¶
Symptoms: When attempting to authenticate to a Science Platform deployment using GitHub, the user gets the error message Authentication provider failed: bad_verification_code: The code passed is incorrect or expired.
Cause: GitHub login failed after the OAuth 2.0 interaction with GitHub was successfully completed, and then the user reloaded the failed login page (or reloaded the page while Gafaelfawr was attempting to complete the authentication). This error is normal and expected if one reloads a GitHub login error page or interrupts the GitHub login. It itself doesn’t represent a problem, and is probably a red herring distracting from whatever real problem there is. Most likely, there is some failure on the Gafaelfawr side after GitHub authentication that’s preventing the authentication from completing or making it take a long time, and the user ran out of patience and reloaded the page (which will never work).
Solution: Don’t reload the login page. Find the underlying problem and troubleshoot it. For example, if Gafaelfawr Redis storage is unavailable, Gafaelfawr may time out or fail to store the user’s token after completing GitHub authentication.
User keeps logging in through the wrong identity provider¶
Symptoms: When attempting to use a different identity provider for authentication, such as when linking a different identity to the same account, the CILogon screen to select an identity provider doesn’t appear. Instead, the user is automatically sent to the last identity provider they used.
Cause: The CILogon identity provider selection screen supports remembering your selection, in which case it’s stored in a browser cookie or local storage and you are not prompted again. Even when you want to be prompted.
Solution: Have the user go to https://cilogin.org/me and choose “Delete ALL”. This will clear their remembered selection. They can they retry whatever operation they were attempting.