<aside> 💡 This document guides you on how to rebuild a new user volume and migrate the data from the old user volume to the new one. The file system type of the new volume is according to the current filesystem setting.

</aside>

Tutorial

• Step 1: Check the PV status is Retain and get the {pv-name}

  $ kubectl get pv | grep claim-{username}
  
  pvc-dff1b4c0-6119-4fd3-9051-ec9034591176   20Gi  RWO  Retain  Bound    hub/claim-{username}  rook-block   38m
  

• Step 2: Delete user PVC.

  $ kubectl -n hub get pvc | grep '{username}'
  $ kubectl -n hub delete pvc claim-{username}
  
  persistentvolumeclaim "claim-{username}" deleted
  

• Step 3: Check the user PV is not related to PVC Storage

  • Check the PV status is Released.

    $ kubectl get pv {pv-name}
    
    NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
    pvc-dff1b4c0-6119-4fd3-9051-ec9034591176   20Gi       RWO            Retain           **Released** hub/claim-phadmin   rook-block              42m
    

  • remove claimRef from pv

    $ kubectl patch pv {pv-name} -p '{"spec":{"claimRef":null}}'
    

  • After the commend, check the PV status is Available.

    $ kubectl get pv {pv-name}
    
    NAME                                       CAPACITY   ACCESS MODES RECLAIM POLICY STATUS    CLAIM               STORAGECLASS   REASON   AGE
    pvc-dff1b4c0-6119-4fd3-9051-ec9034591176   20Gi       RWO          Retain         **Available** hub/claim-phadmin   rook-block              42m
    

• Step 4: Create a temp pvc

  • Create a temp pvc yaml

    $ vim temp-claim-{username}.yaml
    

    Note: The resource request storage need to be the same as the previous one.

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: claim-old-{username}
      namespace: hub
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi # must be the **same size**
      storageClassName: rook-block
      volumeMode: Filesystem
      volumeName: {pv-name}
    

  • Apply the temp-claim-{username}

    $ kubectl -n hub apply -f temp-claim-{username}.yaml
    
    persistentvolumeclaim/claim-old-{username} created
    

• Step 5: Create a new pvc

  • Create a new pvc yaml

    $ vim claim-{username}.yaml
    

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      annotations:
        hub.jupyter.org/username: {username}
        volume.beta.kubernetes.io/storage-class: rook-block
      labels:
        app: jupyterhub
        chart: jupyterhub-0.10.6
        component: singleuser-storage
        heritage: jupyterhub
        release: primehub
      name: claim-{username}
      namespace: hub
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi # **same size**
      storageClassName: rook-block
      volumeMode: Filesystem
    

  • Apply claim-{username}.yaml

    $ kubectl -n hub apply -f claim-{username}.yaml
    
    persistentvolumeclaim/claim-{username} created
    

• Step 6: Mount PVC.

  $ kubectl mountpvc --pvc claim-{username} --pvc claim-old-{username} -n hub rsync-pvc --image ubuntu:18.04 --command -- sleep 100000 | kubectl apply -f -
  

• Step 7: rsync data files until finishing.

  $ kubectl -n hub exec -it $(kubectl -n hub get pod -l app=rsync-pvc | cut -d' ' -f1 | grep -v NAME) bash
  

  • In bash env, you need to type the following commend:

  Note: It is very important for the slash word. Please be careful when you do the rsync command.

  $ apt-get update && apt-get install -yq rsync
  $ rsync -avP /pvcs/claim-old-{username}/ /pvcs/claim-{username}/
  

• Step 8: Delete the rsync deployment.

  $ kubectl -n hub delete deploy rsync-pvc
  

• Step 9: Delete old PVC

  NOTICE: If you finish syncing files and asking the user to check the files, then you can do the following commend.

  $ Kubectl -n hub delete pvc claim-old-{username}
  

Verification

• As the targeted user (impersonate), Launch the Notebook and open the Terminal. To verify the file system type.

  • Step 1: Login Admin Keycloak.

    • URL: http://<cluster_url>/auth/admin/primehub/console

  • Step 2: Find user account and impersonate user account.

    • Snapshot

  • Step 3: Open PrimeHub and start the notebook.

    • Snapshot

  • Step 4: Run the following command.

    $ mount | grep jovyan
    

    

Trouble Shooting

TroubleShooting: Mount volume failed: Structure needs cleaning

Reference

N/A