Skip to content

When the field "spec.database.storageClassName" is omitted in Kubegres YAML, Kubegres operator should assign the default storageClass to the deployed PostgreSql cluster #18

Description

@mologie

Hi, I found that kubeges's controller segfaults and stays down when I omit the storageClassName property. It recovers once the offending Kubegres object is removed.

Expected behavior: Kubegres would request PVCs with the default storage class, i.e. simply omit the storageClassName field when generating its PVCs.

Steps to reproduce: Apply the following yaml:

apiVersion: v1
kind: Secret
metadata:
  name: postgres
type: Opaque
stringData:
  rootpasswd: foo
  replpasswd: bar
  userpasswd: baz
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-conf
data:
  primary_init_script.sh: |
    #!/bin/bash
    set -e
    psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
    CREATE DATABASE $MY_POSTGRES_USER;
    CREATE USER $MY_POSTGRES_USER WITH PASSWORD '$MY_POSTGRES_PASSWORD';
    GRANT ALL PRIVILEGES ON DATABASE $MY_POSTGRES_USER to $MY_POSTGRES_USER;
    EOSQL
---
apiVersion: kubegres.reactive-tech.io/v1
kind: Kubegres
metadata:
  name: postgres
spec:
  image: postgres:13.2
  port: 5432
  replicas: 1
  database:
    size: 10Gi
  env:
    - name: POSTGRES_PASSWORD
      valueFrom:
        secretKeyRef:
          name: postgres
          key: rootpasswd
    - name: POSTGRES_REPLICATION_PASSWORD
      valueFrom:
        secretKeyRef:
          name: postgres
          key: replpasswd
    - name: MY_POSTGRES_USER
      value: notisvc
    - name: MY_POSTGRES_PASSWORD
      valueFrom:
        secretKeyRef:
          name: postgres
          key: userpasswd
  customConfig: postgres-conf

And observe how the manager container of kubegres-controller-manager panics immediately:

manager
2021-06-07T17:14:53.192Z    INFO controllers.Kubegres =======================================================
manager
2021-06-07T17:14:53.192Z    INFO controllers.Kubegres =======================================================
manager
2021-06-07T17:14:54.192Z    INFO controllers.Kubegres KUBEGRES {"name": "postgres", "Status": {"blockingOperation":{"statefulSetOperation":{},"statefulSetSpecUpdateOperation":{}},"previousBlockingOperation":{"statefulSetOperation":{},"statefulSetSpecUpdateOperation":{}}}}
manager
2021-06-07T17:14:54.192Z    INFO controllers.Kubegres Corrected an undefined value in Spec. {"spec.database.volumeMount": "New value: /var/lib/postgresql/data"}
manager
2021-06-07T17:14:54.192Z    INFO controllers.Kubegres Updating Kubegres Spec {"name": "postgres"}
manager
2021-06-07T17:14:54.192Z    DEBUG controller-runtime.manager.events Normal {"object": {"kind":"Kubegres","namespace":"kubegres-crash","name":"postgres","uid":"c4fcc9d9-e21c-4af3-9e33-18c1f980654e","apiVersion":"kubegres.reactive-tech.io/v1","resourceVersion":"62606471"}, "reason": "SpecCheckCorrection", "message": "Corrected an undefined value in Spec. 'spec.database.volumeMount': New value: /var/lib/postgresql/data"}
manager
E0607 17:14:54.207006 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
manager
goroutine 403 [running]:
manager
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x15f20e0, 0x22f5700)
manager
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/runtime/runtime.go:74 +0xa6
manager
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
manager
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/runtime/runtime.go:48 +0x89
manager
panic(0x15f20e0, 0x22f5700)
manager
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
manager
reactive-tech.io/kubegres/controllers/states.(*DbStorageClassStates).getSpecStorageClassName(...)
manager
    /workspace/controllers/states/DbStorageClassStates.go:80
manager
reactive-tech.io/kubegres/controllers/states.(*DbStorageClassStates).GetStorageClass(0xc000ae8b50, 0x8, 0xc000ea7be0, 0x19)
manager
    /workspace/controllers/states/DbStorageClassStates.go:62 +0x47
manager
reactive-tech.io/kubegres/controllers/states.(*DbStorageClassStates).loadStates(0xc000ae8b50, 0x8, 0xc000ea7be0)
manager
    /workspace/controllers/states/DbStorageClassStates.go:46 +0x2f
manager
reactive-tech.io/kubegres/controllers/states.loadDbStorageClass(...)
manager
    /workspace/controllers/states/DbStorageClassStates.go:39
manager
reactive-tech.io/kubegres/controllers/states.(*ResourcesStates).loadDbStorageClassStates(0xc000ae9370, 0x14d91f5, 0x8)
manager
    /workspace/controllers/states/ResourcesStates.go:75 +0xe5
manager
reactive-tech.io/kubegres/controllers/states.(*ResourcesStates).loadStates(0xc000ae9370, 0xc000843200, 0x0)
manager
    /workspace/controllers/states/ResourcesStates.go:46 +0x45
manager
reactive-tech.io/kubegres/controllers/states.LoadResourcesStates(...)
manager
    /workspace/controllers/states/ResourcesStates.go:40
manager
reactive-tech.io/kubegres/controllers/ctx/resources.CreateResourcesContext(0xc0005d22c0, 0x19a3880, 0xc000eaee70, 0x19ac260, 0xc0001ba2e0, 0x19b6f60, 0xc00044bae0, 0x19a06c0, 0xc00054a2c0, 0x0, ...)
manager
    /workspace/controllers/ctx/resources/ResourcesContext.go:105 +0x6c5
manager
reactive-tech.io/kubegres/controllers.(*KubegresReconciler).Reconcile(0xc00054a300, 0x19a3880, 0xc000eaee70, 0xc0006bf2c0, 0xe, 0xc0006bf2a0, 0x8, 0xc000eaee70, 0x40a1ff, 0xc000030000, ...)
manager
    /workspace/controllers/kubegres_controller.go:74 +0x17f
manager
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0005448c0, 0x19a37c0, 0xc00029a000, 0x16482e0, 0xc001000460)
manager
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.2/pkg/internal/controller/controller.go:263 +0x317
manager
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0005448c0, 0x19a37c0, 0xc00029a000, 0x203000)
manager
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.2/pkg/internal/controller/controller.go:235 +0x205
manager
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1(0x19a37c0, 0xc00029a000)
manager
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.2/pkg/internal/controller/controller.go:198 +0x4a
manager
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
manager
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185 +0x37
manager
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0004b0f50)
manager
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:155 +0x5f
manager
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000afbf50, 0x196dee0, 0xc000991620, 0xc00029a001, 0xc0003621e0)
manager
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:156 +0xad
manager
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0004b0f50, 0x3b9aca00, 0x0, 0xc00029a001, 0xc0003621e0)
manager
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:133 +0x98
manager
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x19a37c0, 0xc00029a000, 0xc0001547c0, 0x3b9aca00, 0x0, 0xc000704501)
manager
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185 +0xa6
manager
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x19a37c0, 0xc00029a000, 0xc0001547c0, 0x3b9aca00)
manager
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:99 +0x57
manager
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
manager
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.2/pkg/internal/controller/controller.go:195 +0x4e7
manager
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
manager
    panic: runtime error: invalid memory address or nil pointer dereference
manager
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x14aafe7]
manager
goroutine 403 [running]:
manager
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
manager
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/runtime/runtime.go:55 +0x10c
manager
panic(0x15f20e0, 0x22f5700)
manager
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
manager
reactive-tech.io/kubegres/controllers/states.(*DbStorageClassStates).getSpecStorageClassName(...)
manager
    /workspace/controllers/states/DbStorageClassStates.go:80
manager
reactive-tech.io/kubegres/controllers/states.(*DbStorageClassStates).GetStorageClass(0xc000ae8b50, 0x8, 0xc000ea7be0, 0x19)
manager
    /workspace/controllers/states/DbStorageClassStates.go:62 +0x47
manager
reactive-tech.io/kubegres/controllers/states.(*DbStorageClassStates).loadStates(0xc000ae8b50, 0x8, 0xc000ea7be0)
manager
    /workspace/controllers/states/DbStorageClassStates.go:46 +0x2f
manager
reactive-tech.io/kubegres/controllers/states.loadDbStorageClass(...)
manager
    /workspace/controllers/states/DbStorageClassStates.go:39
manager
reactive-tech.io/kubegres/controllers/states.(*ResourcesStates).loadDbStorageClassStates(0xc000ae9370, 0x14d91f5, 0x8)
manager
    /workspace/controllers/states/ResourcesStates.go:75 +0xe5
manager
reactive-tech.io/kubegres/controllers/states.(*ResourcesStates).loadStates(0xc000ae9370, 0xc000843200, 0x0)
manager
    /workspace/controllers/states/ResourcesStates.go:46 +0x45
manager
reactive-tech.io/kubegres/controllers/states.LoadResourcesStates(...)
manager
    /workspace/controllers/states/ResourcesStates.go:40
manager
reactive-tech.io/kubegres/controllers/ctx/resources.CreateResourcesContext(0xc0005d22c0, 0x19a3880, 0xc000eaee70, 0x19ac260, 0xc0001ba2e0, 0x19b6f60, 0xc00044bae0, 0x19a06c0, 0xc00054a2c0, 0x0, ...)
manager
    /workspace/controllers/ctx/resources/ResourcesContext.go:105 +0x6c5
manager
reactive-tech.io/kubegres/controllers.(*KubegresReconciler).Reconcile(0xc00054a300, 0x19a3880, 0xc000eaee70, 0xc0006bf2c0, 0xe, 0xc0006bf2a0, 0x8, 0xc000eaee70, 0x40a1ff, 0xc000030000, ...)
manager
    /workspace/controllers/kubegres_controller.go:74 +0x17f
manager
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0005448c0, 0x19a37c0, 0xc00029a000, 0x16482e0, 0xc001000460)
manager
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.2/pkg/internal/controller/controller.go:263 +0x317
manager
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0005448c0, 0x19a37c0, 0xc00029a000, 0x203000)
manager
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.2/pkg/internal/controller/controller.go:235 +0x205
manager
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1(0x19a37c0, 0xc00029a000)
manager
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.2/pkg/internal/controller/controller.go:198 +0x4a
manager
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
manager
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185 +0x37
manager
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0004b0f50)
manager
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:155 +0x5f
manager
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000afbf50, 0x196dee0, 0xc000991620, 0xc00029a001, 0xc0003621e0)
manager
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:156 +0xad
manager
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0004b0f50, 0x3b9aca00, 0x0, 0xc00029a001, 0xc0003621e0)
manager
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:133 +0x98
manager
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x19a37c0, 0xc00029a000, 0xc0001547c0, 0x3b9aca00, 0x0, 0xc000704501)
manager
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185 +0xa6
manager
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x19a37c0, 0xc00029a000, 0xc0001547c0, 0x3b9aca00)
manager
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:99 +0x57
manager
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
manager
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.2/pkg/internal/controller/controller.go:195 +0x4e7

Activity

  1. alex-arica commented on Jun 7, 2021

    @alex-arica
    Member

    Thank you for your message and for providing those details.

    I am going to investigate this issue and come back to you.

    From the spec, the field "storageClassName" is a mandatory field, please see that field in the table in the page:
    https://www.kubegres.io/doc/properties-explained.html

    However, in the logs it should not display panics but rather a validation message such as:

    "In the Resources Spec the value of 'spec.database.storageClassName' is undefined. Please set a value otherwise this operator cannot work correctly."

  2. mologie commented on Jun 7, 2021

    @mologie
    Author

    Indeed, I missed that part! However, I'd suggest to make it optional because the PVC's storageClassName field is optional too, and otherwise there is now way to reference the default storageclass. I'd gladly open another ticket independently of this issue and contribute that change.

  3. alex-arica commented on Jun 7, 2021

    @alex-arica
    Member

    Thank you for your message. As you suggested, we can make storageClassName optional.

    If that field is not provided, I can add a logic to identify the name of the default "storageClassName" in a Kubernetes cluster and set it automatically.

    I added this feature to the roadmap.

  4. mologie commented on Jun 7, 2021

    @mologie
    Author

    If that field is not provided, I can add a logic to identify the name of the default "storageClassName" in a Kubernetes cluster and set it automatically.

    I believe you will not have to set anything at all. You can omit the storageClassName property of your PVC, and it will pick the default storage class.

    https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/

    Depending on the installation method, your Kubernetes cluster may be deployed with an existing StorageClass that is marked as default. This default StorageClass is then used to dynamically provision storage for PersistentVolumeClaims that do not require any specific storage class.

    This is usually what I want when deploying PostgreSQL: The default storage class tends to be durable but fast enough storage.

  5. alex-arica commented on Jun 7, 2021

    @alex-arica
    Member

    Thank you for your recommendation. I will take that approach which simplifies the feature.

    Thank you for your great feedback.

  6. alex-arica commented on Jun 8, 2021

    @alex-arica
    Member

    Is there anything preventing you from running the command below to get the name of the default storage class in your Kubernetes cluster?

    kubectl get sc
    

    And then set the default storage class name in your Kubegres resource YAML when creating a cluster of PostgreSql?

  7. mologie commented on Jun 8, 2021

    @mologie
    Author

    Nothing whatsoever :). And that's what I did to try out Kubegres. However, I would prefer to have as little information about the cluster I am running on in my YAML resources. This makes them more portable, i.e. there is less stuff to edit/replace for moving it from one cluster to another.

  8. alex-arica commented on Jun 8, 2021

    @alex-arica
    Member

    We are using Kubegres in large systems and in those type of Kubernetes clusters have multiple storage classes based on latency capabilities. We tend to set the storage classes manually by selecting the ones with the lowest latencies. The default one is not always the fastest.

    In your case, I suppose you are working with systems where latency is not the main factor and you want to setup a cluster quickly where the default settings are acceptable.

    The argument about portability is compelling in your case. I think that's an important factor. It is good that you suggested this feature because it highlights additional use cases where Kubegres can fill some gaps.

    I am working on this feature and will commit a beta version tomorrow. I will let you know once it is available, and I would be grateful if you can test it as well?

    Once you confirm things are working fine on your side, I will make a final commit with acceptance tests and create a release/tag.

  9. mologie commented on Jun 8, 2021

    @mologie
    Author

    Will do, thank you very much for looking into this!

  10. changed the title [-]Segfault when spec.database.storageClassName is omitted[/-] [+]When spec.database.storageClassName is omitted in YAML, Kubegres should assign the default one to the deployed PosgreSql cluster[/+] on Jun 8, 2021
  11. added a commit that references this issue on Jun 8, 2021
  12. changed the title [-]When spec.database.storageClassName is omitted in YAML, Kubegres should assign the default one to the deployed PosgreSql cluster[/-] [+]When spec.database.storageClassName is omitted in YAML, Kubegres should assign the default storageClass to the deployed PosgreSql cluster[/+] on Jun 8, 2021
  13. alex-arica commented on Jun 8, 2021

    @alex-arica
    Member

    I made the changes and added new acceptance tests in GIT main.

    You can install beta Kubegres version 1.6, as follows:

    kubectl apply -f https://raw.githubusercontent.com/reactive-tech/kubegres/main/kubegres.yaml
    

    Once you are happy with your tests, please let me know and I will create a new release/tag and update the documentation.

  14. mologie commented on Jun 8, 2021

    @mologie
    Author

    Seems to work 👌 . What I tested was that 1) the segfault is fixed by inserting the objects that would previously crash, observe the crash, and then updating kubegres. It correctly deployed PostgreSQL afterwards.

    I 2) checked that it would deploy PostgreSQL with my default storage class "vsan-mirror" when omitting storageClassName, which worked correctly.

  15. self-assigned this
    on Jun 9, 2021
  16. changed the title [-]When spec.database.storageClassName is omitted in YAML, Kubegres should assign the default storageClass to the deployed PosgreSql cluster[/-] [+]When the field "spec.database.storageClassName" is omitted in Kubegres YAML, Kubegres operator should assign the default storageClass to the deployed PostgreSql cluster[/+] on Jun 9, 2021
  17. alex-arica commented on Jun 9, 2021

    @alex-arica
    Member

    I created a new tag/release 1.6 here:
    https://github.com/reactive-tech/kubegres/releases

    To install Kubegres 1.6:

    kubectl apply -f https://raw.githubusercontent.com/reactive-tech/kubegres/v1.6/kubegres.yaml
    

    I updated the documentation pages, as follows:

    Please close this ticket if you are happy with the changes.

    Thank you for your suggestions and your help!

  18. mologie commented on Jun 10, 2021

    @mologie
    Author

    Thanks very much :) I updated to the version you mentioned w/o issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions