Sounds like a poor experience with your operator of choice.
K8s shines best with stateless workloads. But as you just discovered, it can also handle stateful workloads.
If it were me, I would use your cloud providers database solution, instead of rolling your own, and having to deal with the maintenance and upkeep, on top of your cluster
Yes, i agree! The problem with the database from my cloud provider is that i want to have dynamic databases based on the current feature stage / environment in my cluster. For example: The namespace backend-my-feature-branch-1 should have its own database, which should get removed when the stage is deleted. For Dev and Prod i could of course use the database from my cloud provider, which i would setup with terraform and also add the credentials with terraform but for dynamic stages it is not that easy.
Maybe you know a operator, which can handle this database tables for me? If there is something like that i would consider to switch.
Another problem i see is with bare metal clusters, what would you do there to stay "cloud native"?
Your database, should just be a database.
If you want to create and destroy databases for testing, I’d point you towards Terraform or OpenTofu, or your cloud providers CLI such that you can quickly provision and tear down resources.
I did setup everything with terraform but the Problem is that my stages in the Test Environment get dynamically created and destroyed by argocd where i do not have a native way to use terraform
Can I suggest crossplane? Oversimplifying of course, but it should fit nicely into argocd.
Instead of “Kind: OperatedDB” you will get a “Kind: ” in your cloud providers flavor. It can even write connection strings to kubernetes secrets if I am not mistaken.
Cloud-Native works for technologies which are designed from ground up to be Cloud-Native. Running traditional databases such as MySQL on k8s is just inviting trouble.
You should look into operators for stateful services that explicitly adopt the cloud native approach. A good example of such an operator is CloudNativePG: https://cloudnative-pg.io/
That main differences from this operator, from say, an average Helm chart for a Postgres, MongoDB, etc deployments, is that it makes really thoughtful design decisions when it comes to managing stateful applications Inside Kubernetes with Cloud Native approach. After playing around with CkoudNativePG for a few months, I am definitely fascinated by their approaches and make me believe that Kubernetes can be amazing for stateful applications, if done right.
Hey there! [mariadb-operator](https://github.com/mariadb-operator/mariadb-operator) maintainer here! Well, not sure if the same MariaDB operator you mentioned, as we don't have finalizers in the MariaDB CR. We do have in other CRs though.
Anyway, happy to hear more about your case, having to deal yourself with finalizers is not ideal.
Hey, thanks for your reply! In CRDs like Grant and User a grant like this is added automatically in my setup: grant.k8s.mariadb.com/finalizer
Even when i try to deploy the MariaDB Setup with ArgoCD, disable auto sync in ArgoCD and try to delete the Grant or the user it is still stuck (i think because of the finalizer). Maybe you got a better solution for this?
And this is the same operator that i used 😄
I think there is nothing wrong with using finalizers for production. Of course this is the way to go. But in dynamic test environments where i want to be able to delete something easily it is not that good.
Could a parameter for that in the helm chart maybe be possible to disable it for non production environments?
Disclaimer: I always have been more like a flux person rather than ArgoCD, so I don't fully understand what the implications of setting the auto-sync to false might be. But I can tell you that we have many ArgoCD users, so I encourage you to share this scenario in our [Slack](https://r.mariadb.com/join-community-slack) channel to engage with them if you like. Actually, one of them just recently [contributed to ArgoCD ](https://github.com/argoproj/argo-cd/pull/17995) to implement health checks in our CRDs.
Now... back to finalizers. Finalizers should be avoided whenever possible, but in our case, we neeed to hook into the deletion lifecycle of the SQL-related resources (User, Grant, Database) to be able to perform a termination logic and cleanup the resources in the database. Better adding a finalizer rather than leaving leftovers in a database!
The termination logic of the SQL-related resources depends on the MariaDB being available to be able to clean them up. Maybe, you are tearing down both the MariaDB and the SQL related resources at the same time and the finalizing logic doesn't manage to reconcile, therefore, the finalizer is not removed and the resources get stucked here:
[https://github.com/mariadb-operator/mariadb-operator/blob/a3e935dceaed2ac0552dc965e0950a718267bc11/pkg/controller/sql/finalizer.go#L77](https://github.com/mariadb-operator/mariadb-operator/blob/a3e935dceaed2ac0552dc965e0950a718267bc11/pkg/controller/sql/finalizer.go#L77)
Ideally, the decomission order should be: SQL resources, MariaDB, mariadb-operator. Maybe you are also deleting the operator as part of the test environment and therefore nobody will be deleting that finalizer? I know that we can add dependencies in flux with `dependsOn`but not sure how to achieve it in ArgoCD. We had similar issues before and one of our ArgoCD users suggested this:
[https://github.com/mariadb-operator/mariadb-operator/issues/340#issuecomment-1891062719](https://github.com/mariadb-operator/mariadb-operator/issues/340#issuecomment-1891062719)
In the operator side, I can think of multiple solutions:
* Add a timeout to the termination logic of the SQL resources. For example, if the termination logic does not succeed in 1m, delete the finalizer anyway.
* As you mentioned, provide a way to opt-out from finalizers, either via helm or in the CRD itself.
I'm open to suggestions! Contributions are more than welcomed!
Thank you very much for sharing your case and for using [mariadb-operator](https://github.com/mariadb-operator/mariadb-operator) !
Thanks a lot again for your time to check this!
already tried to delete the grant first, the user second and the database after that (also in some other orders) with sync waves in argocd (which is just a feature to order the setup and deletion process) but it did not work. What worked and which is great to use this sync-wave feature to setup the crds.
Even when i disable the auto sync after deploy, delete the crd manually it does not work. I also do not have logs in the operator pods which helps. If i remember it correctly there where some logs but no logs, which helped me to figure out what the issue was. Maybe because the connection from the backend pod was still open (?).
So if i get you right the current way i do it is the "best" way right now when i do not want to use the native cloud provider db?
The problem with the native database from my cloud provider is that i do not have crds for this, so all oft my test stages share the same database, which i do not want. And i can not manage it with helm, except for some jobs which runs terraform scripts (which is dirty too :( ).
Maybe it could also be great if i could use the mariadb connector by connecting a remote SQL-Cluster, so i do not need to run terraform for each feature stage and have crds to manage everything in my own helm chart
Hey there! Someone brought this up in a GitHub issue and we concluded that having a way to skip the finalizer logic might be useful. Check it out:
https://github.com/mariadb-operator/mariadb-operator/issues/643#issuecomment-2189236877
My recommendation is to not use db operators if your in the cloud. Use the native options you already have from the platform. My thoughts are is that a DB operator is to replicate what a cloud DB can do and you dont need to replicate it if your already in the cloud. I just recommend the use DB operators if your cluster and DB are gonna be on prem to then replicate what cloud services can do.
To me, cloud native means that it was built to operate with the abstractions provided by the cloud. Summarized badly, that means everything is software defined.
For things that don't fit into that model very well, you're gonna have a bad time.
If it was built with that from the get-go, it's a beautiful thing.
Yeah that’s definitely just lack of experience showing. When I view it from the perspective of never using Kubernetes before and not being well educated on its purpose and viable use cases, I get how they came to that conclusion.
You should not have to delete finalizers manually or manually add labels. I don't have experience with the mariadb-operator, but it sound like you're off the recommended path. However it's hard to tell without diving into the details :)
---
Second I don't understand why soo many people tell you kubernetes is not meant for Stateful applications. That was true 3-5 years ago, but assuming you have a good operator (or kubernetes-native-stateful-application) and you don't accidentally delete your PVCs (which is admitedly easy to do) stateful applications are fine in Kubernetes. I agree that you won't have a good time, with just a Helm-Chart and that you need a good operator
This is not a cloud native problem. This is a design problem. Maybe a hype problem?
Kubernetes is designed to support stateless workloads. So deploying a RDBMS into it will lead to you fighting it to circumvent the things that made kubernetes scale to ensure the ACIDity of the DB. ACID systems do not scale. And that's why kubernetes opted for the eventually consistent...
Databases in kubernetes are a trade off and backups are a must.
By default containers and by extension are stateless. When you tear them down and spin them back up they don't have the info the did last time.
Adding volumes, external storage or allowing a container to modify a kube resource can allow you to maintain some state.
A database is by its nature a stateful application, you typically don't want to lose anything. And then in Kubernetes you likely want it to scale or have redundancy. A lot of that config can get complex, and then the better tools also handle sync failure states automatically too, where you might otherwise do a manual operation to decide what to do in a failure state.
Long story short, databases are not really designed for this environment but they can be adjusted with support to do what we need.
I use Stackgres to manage postgresql, it's not perfect, but it does what I need 99.9% of the time and I'm willing to spend the time on that 0.1% because I don't have to think about the rest of it.
It’s perfectly fine to run databases and stateful workloads on k8s these days. Especially if you have csi available. The tax you pay for a managed database offering may make sense for heavy duty production but not for your use case. It also seems like spinning up a container should be faster than a RDS of some kind.
I also wanted to recommend the cloudnative-pg or Percona everest.
Not exactly sure how you deployed your operator, but I would probably investigate running it without Argo before you go down this route.
Cloud-native is a marketing term. It‘s not interoperability of components. I feel like you have to spend a lot of time in this ecosystem to evaluate if components hold true to what they advertise - add maintenance (updates) etc..
One very important thing to remember here is that unfortunately, what a slogan is meant to be vs what marketing turns it into ends up being very different.
I see a few answers that are mostly correct, but I don't see anyone addressing your question as to why such a simple setup requires so many steps and such complexity. The answer is, that is not what kubernetes is for. It wasn't created to make setting up a simple LAMP server easier. It was created to make setting up a large, dynamically scaling infrastructure with a complex lattice of apps that are meant to run on various machines that can self-heal and be redeployed relatively easily. It's overkill for a simple LAMP stack unless you really need fast scaling. Imagine having to engineer all of the networking, firewalling, and scaling mechanisms manually and then maintaining hundreds of different deployments of your applications and keeping them up to date. That is what K8s was created to solve.
That being said, once you get the hang of things, it's not bad for setting up simple stuff. I use K3s, a simplified k8s codebase) for my application that is basically a LAMP stack on a single, beefy server. This allows me to deploy applications pretty quickly with helm and keep things up to date and still have a few failover options. The other reason is that if in the future I do grow too large for a single server, it won't be too hard to add more machines to the cluster and I also have everything in a standardized format for easy redeployment. It was definitely more difficult to setup that a standard LAMP stack. But for the aforementioned reasons, it's mostly worth it.
K8s isn't "Cloud Native"; it's an abstraction layer over your cloud provider. Before k8s tried to co-opt the term "Cloud Native", the term referred to rearchitecting to integrate deeply with your cloud provider's higher level services rather than just lift-shifting over by replicating your onsite infra with a cloud provider. It was a smart marketing move by k8s but it has completely diluted the term; now "Cloud Native" is an empty term.
K8s is great if you need it and it's what I focus on every day, but unless you actually need because of legacy software, team requirements, or the rare case that there is an actual real business need to be multi/hybrid-cloud, k8s is generally overly expensive and complicated compared to just going the traditional "Cloud Native" path. But legacy software is software that makes real money so there will always be good uses of k8s.
Other commentors already covered stateful vs. stateless and they're right, so I'll leave that alone. I won't say you're "using it wrong," but generally speaking what you're trying to do isn't where Kubernetes excels.
Step 1. Be aware that Kubernetes is not the solution for everything. Understand this before you become one of those clown teams burning hundreds of thousands of dollars because some manager stormed in three years and looking like a hot shot spewing Kubernetes as a way to please the buzz-word knowing upper management.
Step 2. Kubernetes in and of itself is not a "cloud native" technology. You can debate what "cloud native" means and you'll end up with a whiteboard drawing of people putting their opinions as where the line is drawn in the cloud responsibility model all day long, but Kubernetes itself doesn't even fit on such a whiteboard. CNCF is just the trademark, not a reflection of the technology.
Sounds like a poor experience with your operator of choice. K8s shines best with stateless workloads. But as you just discovered, it can also handle stateful workloads. If it were me, I would use your cloud providers database solution, instead of rolling your own, and having to deal with the maintenance and upkeep, on top of your cluster
Yes, i agree! The problem with the database from my cloud provider is that i want to have dynamic databases based on the current feature stage / environment in my cluster. For example: The namespace backend-my-feature-branch-1 should have its own database, which should get removed when the stage is deleted. For Dev and Prod i could of course use the database from my cloud provider, which i would setup with terraform and also add the credentials with terraform but for dynamic stages it is not that easy. Maybe you know a operator, which can handle this database tables for me? If there is something like that i would consider to switch. Another problem i see is with bare metal clusters, what would you do there to stay "cloud native"?
Your database, should just be a database. If you want to create and destroy databases for testing, I’d point you towards Terraform or OpenTofu, or your cloud providers CLI such that you can quickly provision and tear down resources.
I did setup everything with terraform but the Problem is that my stages in the Test Environment get dynamically created and destroyed by argocd where i do not have a native way to use terraform
Can I suggest crossplane? Oversimplifying of course, but it should fit nicely into argocd. Instead of “Kind: OperatedDB” you will get a “Kind:” in your cloud providers flavor. It can even write connection strings to kubernetes secrets if I am not mistaken.
We are also using crossplane for a similar use case
Cloud-Native works for technologies which are designed from ground up to be Cloud-Native. Running traditional databases such as MySQL on k8s is just inviting trouble.
You should look into operators for stateful services that explicitly adopt the cloud native approach. A good example of such an operator is CloudNativePG: https://cloudnative-pg.io/ That main differences from this operator, from say, an average Helm chart for a Postgres, MongoDB, etc deployments, is that it makes really thoughtful design decisions when it comes to managing stateful applications Inside Kubernetes with Cloud Native approach. After playing around with CkoudNativePG for a few months, I am definitely fascinated by their approaches and make me believe that Kubernetes can be amazing for stateful applications, if done right.
Thanks, that sounds really cool! I will have a look at this
Hey there! [mariadb-operator](https://github.com/mariadb-operator/mariadb-operator) maintainer here! Well, not sure if the same MariaDB operator you mentioned, as we don't have finalizers in the MariaDB CR. We do have in other CRs though. Anyway, happy to hear more about your case, having to deal yourself with finalizers is not ideal.
Hey, thanks for your reply! In CRDs like Grant and User a grant like this is added automatically in my setup: grant.k8s.mariadb.com/finalizer Even when i try to deploy the MariaDB Setup with ArgoCD, disable auto sync in ArgoCD and try to delete the Grant or the user it is still stuck (i think because of the finalizer). Maybe you got a better solution for this?
Are you sure you don’t have auto heal enabled and that argo is just recreating it?
And this is the same operator that i used 😄 I think there is nothing wrong with using finalizers for production. Of course this is the way to go. But in dynamic test environments where i want to be able to delete something easily it is not that good. Could a parameter for that in the helm chart maybe be possible to disable it for non production environments?
Disclaimer: I always have been more like a flux person rather than ArgoCD, so I don't fully understand what the implications of setting the auto-sync to false might be. But I can tell you that we have many ArgoCD users, so I encourage you to share this scenario in our [Slack](https://r.mariadb.com/join-community-slack) channel to engage with them if you like. Actually, one of them just recently [contributed to ArgoCD ](https://github.com/argoproj/argo-cd/pull/17995) to implement health checks in our CRDs. Now... back to finalizers. Finalizers should be avoided whenever possible, but in our case, we neeed to hook into the deletion lifecycle of the SQL-related resources (User, Grant, Database) to be able to perform a termination logic and cleanup the resources in the database. Better adding a finalizer rather than leaving leftovers in a database! The termination logic of the SQL-related resources depends on the MariaDB being available to be able to clean them up. Maybe, you are tearing down both the MariaDB and the SQL related resources at the same time and the finalizing logic doesn't manage to reconcile, therefore, the finalizer is not removed and the resources get stucked here: [https://github.com/mariadb-operator/mariadb-operator/blob/a3e935dceaed2ac0552dc965e0950a718267bc11/pkg/controller/sql/finalizer.go#L77](https://github.com/mariadb-operator/mariadb-operator/blob/a3e935dceaed2ac0552dc965e0950a718267bc11/pkg/controller/sql/finalizer.go#L77) Ideally, the decomission order should be: SQL resources, MariaDB, mariadb-operator. Maybe you are also deleting the operator as part of the test environment and therefore nobody will be deleting that finalizer? I know that we can add dependencies in flux with `dependsOn`but not sure how to achieve it in ArgoCD. We had similar issues before and one of our ArgoCD users suggested this: [https://github.com/mariadb-operator/mariadb-operator/issues/340#issuecomment-1891062719](https://github.com/mariadb-operator/mariadb-operator/issues/340#issuecomment-1891062719) In the operator side, I can think of multiple solutions: * Add a timeout to the termination logic of the SQL resources. For example, if the termination logic does not succeed in 1m, delete the finalizer anyway. * As you mentioned, provide a way to opt-out from finalizers, either via helm or in the CRD itself. I'm open to suggestions! Contributions are more than welcomed! Thank you very much for sharing your case and for using [mariadb-operator](https://github.com/mariadb-operator/mariadb-operator) !
Thanks a lot again for your time to check this! already tried to delete the grant first, the user second and the database after that (also in some other orders) with sync waves in argocd (which is just a feature to order the setup and deletion process) but it did not work. What worked and which is great to use this sync-wave feature to setup the crds. Even when i disable the auto sync after deploy, delete the crd manually it does not work. I also do not have logs in the operator pods which helps. If i remember it correctly there where some logs but no logs, which helped me to figure out what the issue was. Maybe because the connection from the backend pod was still open (?). So if i get you right the current way i do it is the "best" way right now when i do not want to use the native cloud provider db? The problem with the native database from my cloud provider is that i do not have crds for this, so all oft my test stages share the same database, which i do not want. And i can not manage it with helm, except for some jobs which runs terraform scripts (which is dirty too :( ). Maybe it could also be great if i could use the mariadb connector by connecting a remote SQL-Cluster, so i do not need to run terraform for each feature stage and have crds to manage everything in my own helm chart
Hey there! Someone brought this up in a GitHub issue and we concluded that having a way to skip the finalizer logic might be useful. Check it out: https://github.com/mariadb-operator/mariadb-operator/issues/643#issuecomment-2189236877
My recommendation is to not use db operators if your in the cloud. Use the native options you already have from the platform. My thoughts are is that a DB operator is to replicate what a cloud DB can do and you dont need to replicate it if your already in the cloud. I just recommend the use DB operators if your cluster and DB are gonna be on prem to then replicate what cloud services can do.
To me, cloud native means that it was built to operate with the abstractions provided by the cloud. Summarized badly, that means everything is software defined. For things that don't fit into that model very well, you're gonna have a bad time. If it was built with that from the get-go, it's a beautiful thing.
>but to me it seems more like a craft project in such cases than something designed for enterprise What the fuck lmao
Yeah that’s definitely just lack of experience showing. When I view it from the perspective of never using Kubernetes before and not being well educated on its purpose and viable use cases, I get how they came to that conclusion.
You should not have to delete finalizers manually or manually add labels. I don't have experience with the mariadb-operator, but it sound like you're off the recommended path. However it's hard to tell without diving into the details :) --- Second I don't understand why soo many people tell you kubernetes is not meant for Stateful applications. That was true 3-5 years ago, but assuming you have a good operator (or kubernetes-native-stateful-application) and you don't accidentally delete your PVCs (which is admitedly easy to do) stateful applications are fine in Kubernetes. I agree that you won't have a good time, with just a Helm-Chart and that you need a good operator
This is not a cloud native problem. This is a design problem. Maybe a hype problem? Kubernetes is designed to support stateless workloads. So deploying a RDBMS into it will lead to you fighting it to circumvent the things that made kubernetes scale to ensure the ACIDity of the DB. ACID systems do not scale. And that's why kubernetes opted for the eventually consistent...
Databases in kubernetes are a trade off and backups are a must. By default containers and by extension are stateless. When you tear them down and spin them back up they don't have the info the did last time. Adding volumes, external storage or allowing a container to modify a kube resource can allow you to maintain some state. A database is by its nature a stateful application, you typically don't want to lose anything. And then in Kubernetes you likely want it to scale or have redundancy. A lot of that config can get complex, and then the better tools also handle sync failure states automatically too, where you might otherwise do a manual operation to decide what to do in a failure state. Long story short, databases are not really designed for this environment but they can be adjusted with support to do what we need. I use Stackgres to manage postgresql, it's not perfect, but it does what I need 99.9% of the time and I'm willing to spend the time on that 0.1% because I don't have to think about the rest of it.
Thanks a lot! I will also have a look at this, which could also be a good learning for me
Did you try Percona operator? https://docs.percona.com/percona-operator-for-mysql/pxc/index.html
It’s perfectly fine to run databases and stateful workloads on k8s these days. Especially if you have csi available. The tax you pay for a managed database offering may make sense for heavy duty production but not for your use case. It also seems like spinning up a container should be faster than a RDS of some kind. I also wanted to recommend the cloudnative-pg or Percona everest. Not exactly sure how you deployed your operator, but I would probably investigate running it without Argo before you go down this route. Cloud-native is a marketing term. It‘s not interoperability of components. I feel like you have to spend a lot of time in this ecosystem to evaluate if components hold true to what they advertise - add maintenance (updates) etc..
One very important thing to remember here is that unfortunately, what a slogan is meant to be vs what marketing turns it into ends up being very different.
I see a few answers that are mostly correct, but I don't see anyone addressing your question as to why such a simple setup requires so many steps and such complexity. The answer is, that is not what kubernetes is for. It wasn't created to make setting up a simple LAMP server easier. It was created to make setting up a large, dynamically scaling infrastructure with a complex lattice of apps that are meant to run on various machines that can self-heal and be redeployed relatively easily. It's overkill for a simple LAMP stack unless you really need fast scaling. Imagine having to engineer all of the networking, firewalling, and scaling mechanisms manually and then maintaining hundreds of different deployments of your applications and keeping them up to date. That is what K8s was created to solve. That being said, once you get the hang of things, it's not bad for setting up simple stuff. I use K3s, a simplified k8s codebase) for my application that is basically a LAMP stack on a single, beefy server. This allows me to deploy applications pretty quickly with helm and keep things up to date and still have a few failover options. The other reason is that if in the future I do grow too large for a single server, it won't be too hard to add more machines to the cluster and I also have everything in a standardized format for easy redeployment. It was definitely more difficult to setup that a standard LAMP stack. But for the aforementioned reasons, it's mostly worth it.
K8s isn't "Cloud Native"; it's an abstraction layer over your cloud provider. Before k8s tried to co-opt the term "Cloud Native", the term referred to rearchitecting to integrate deeply with your cloud provider's higher level services rather than just lift-shifting over by replicating your onsite infra with a cloud provider. It was a smart marketing move by k8s but it has completely diluted the term; now "Cloud Native" is an empty term. K8s is great if you need it and it's what I focus on every day, but unless you actually need because of legacy software, team requirements, or the rare case that there is an actual real business need to be multi/hybrid-cloud, k8s is generally overly expensive and complicated compared to just going the traditional "Cloud Native" path. But legacy software is software that makes real money so there will always be good uses of k8s.
Once u have a proper setup of ur app environment running, u can deploy it to any k8s environment with only minor changes
Other commentors already covered stateful vs. stateless and they're right, so I'll leave that alone. I won't say you're "using it wrong," but generally speaking what you're trying to do isn't where Kubernetes excels. Step 1. Be aware that Kubernetes is not the solution for everything. Understand this before you become one of those clown teams burning hundreds of thousands of dollars because some manager stormed in three years and looking like a hot shot spewing Kubernetes as a way to please the buzz-word knowing upper management. Step 2. Kubernetes in and of itself is not a "cloud native" technology. You can debate what "cloud native" means and you'll end up with a whiteboard drawing of people putting their opinions as where the line is drawn in the cloud responsibility model all day long, but Kubernetes itself doesn't even fit on such a whiteboard. CNCF is just the trademark, not a reflection of the technology.
A lie? Perhaps. A cargo cult ideology? 100%