I have not had hands on experience with nomad, but as I understand it, it still requires a lot of plumbing. Something as simple as service discovery isn't, and requires a completely separate consul cluster. Similarly, there is no secrets management, which requires a vault cluster, which in turn requires a it's own consul cluster. I'm not sure what config management options there are.
So now you're stuck with 3 consul clusters, a vault cluster, whatever you choose for config management, and a nomad cluster. Feels like you didn't gain much from the simplicity of nomad.
In addition to that, knowing Hashicorp's pricing, I bet that set up would run you north of a million a year for enterprise setup.
Nomad relies on Consul for Service Discovery and K/V storage, and Vault for secrets, indeed ( Vault can use a variety of backends, including an integrated Raft-based one, Consul, object storage, etc.). One tool that does one thing well, which integrates with other tools that their thing well.
I vastly prefer having three simple Raft-based clusters to manage than the "everything and the kitchen sink" approach Kubernetes takes, with results like base64 encoded for "secrets".
And as someone doing both, Nomad+Consul+Vault are drastically easier on day one and day two. They're also usable outside of Nomad ( you can have bare metal machines outside of Nomad using Vault secrets and Consul for SD and K/V), and you can link multiple regions together.
You do indeed need some basic config management to configure the clusters. Ansible seems to have won that race sadly, and there are available playbooks.
> In addition to that, knowing Hashicorp's pricing, I bet that set up would run you north of a million a year for enterprise setup
They've changed up their pricing structure and there are more tiers and add-ons, so i doubt it ( basic Vault cluster was in the 5 figured range per year), if you need support. It's not like enterprise support for Kubernetes would come cheap either, especially if you do it the recommended way with multiple clusters and all that.
> Nomad relies on Consul for Service Discovery and K/V storage, and Vault for secrets, indeed ( Vault can use a variety of backends, including an integrated Raft-based one, Consul, object storage, etc.). One tool that does one thing well, which integrates with other tools that their thing well.
You say nope but then you prove that Nomad relies on Consul as I had mentioned, meaning, you need a Consul instance behind the scenes. If the Nomad setup recommendation is anything like the Vault recommendation, the recommendation will be to run two clusters, one for services discovery, and one for K/V storage for Nomad. I've setup an enterprise Vault instance, and their enterprise architect recommended separate instances. Which is totally fine, but it does mean two Consul clusters + a Nomad cluster.
From my experience with Consul and Vault, it is not as simple as you say it is. A team of 3 Engineers took 3 months to set up an enterprise grade cluster. There was a little bureaucracy at the time, so I can't really blame it on that. If I recall correctly, the integrated Raft-based clustering was being worked on and we were made aware of it because there was some pushback from management on separate 2 Consul clusters for K/V and SD, but I never got to see it to fruition and utilise it, so for us it was Consul. Other backends were discouraged at the enterprise level, they never really made it clear if they'd fully support us if we went with a different backend, leading me to believe that at best, they'd prefer you use Consul over something else. I mean why wouldn't they? They'd rather you pay them extra for a Consul cluster.
If Nomad is anything like my experience with Vault/Consul, then unfortunately you are still stuck with the setup I mention earlier, that is 1 Vault cluster, 1 Nomad cluster, and 3 Consul clusters (1 K/V for Vault, 1 K/V for Nomad, and 1 for service discovery). For sure having separate individual tools that does 1 thing may have their advantages, but I fail to see how this is that "much more simpler" than Kubernetes. At best this is marginally simpler.
Vault has integrated storage since multiple versions, and Nomad can very well use a single Consul cluster for both SD and K/V. ( And honestly i can't recall having two Consul clusters being recommended, and the proposal we had from Hashicorp included a Consul Enterprise cluster for Vault as part of Vault's pricing.)
So, you need three clusters - Vault, Nomad and Consul for SD and KV for Nomad. Two of which, Consul and Nomad, can run on the same machines ( it'd be suboptimal security-wise to have Vault there too).
You don't have to deal with operational aspects of consult, nomad and vault of that matter if you choose the managed kubernetes cluster of the cloud provider. If you are talking about container orchestration onpremise - the experience I had with kubernetes is terrible. There 1000 things that can go wrong. Onpremise, I would recommend using k3s which is super simple to setup or microk8s with comes with ubuntu 20.04. I am not sure what benefits you would get with nomad as compared to the lightweight kubernetes distros. One of the benefits of these lightweight solutions is that, you will basically find a helm chart for any serious application out there where as in nomad you might have to figure it out how to deploy. For instances deploying cassandra, you will find a helm chart, but nomad, you might find blog that does it or figure it out yourself. To my best of knowledge that how it is, may be I am wrong?
microk8s and k3s aren't fit or designed for production, they're mostly for testing/experimenting.
And even with managed Kubernetes, there's still a lot of complexity remaining ( GCP had to come out with a more managed service, GKE Autopilot, to address some of that), but you still have the evolution of APIs to keep track of every update you make, and there are still dozens of services that are updated each update, and each one can go wrong ( even if it rarely does).
> . One of the benefits of these lightweight solutions is that, you will basically find a helm chart for any serious application out there where as in nomad you might have to figure it out how to deploy. For instances deploying cassandra, you will find a helm chart, but nomad, you might find blog that does it or figure it out yourself. To my best of knowledge that how it is, may be I am wrong?
Indeed, and that's the main disadvantage for Nomad IMHO, the ecosystem is much smaller so there aren't that many ready-made equivalents to Helm charts and operators. Depending on how many of those you need, k8s can save you a lot of time.
I don't agree with you. What aspects of microk8s or k3s makes it experimental? It used to be the case, in case of k3s it is one of core offerings from rancher. Same goes for microk8s. Purpose of autopilot is something else, if you are talking about pure orchestration bare minimum kubernetes is actually not a bad option. API will keep evolving but basic objects like deployments and statefulsets that you need for paas like experience are quite stable.
> K3s is a highly available, certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances.
> [microk8s] Low-ops, minimal production Kubernetes,
for devs, cloud, clusters, workstations, Edge and IoT.
Microk8s started as an easy alternative to minikube for local dev.
k3s started as a simplified version of k8s for testing/experimenting on RPis, etc.
Today both seem to focus on IoT/"edge". Both do clustering and HA of the control plane though, so are in theory usable in production.
However, why would you use either of them in production? Yes, it's easier than vanilla k8s, but still has a lot of moving parts, and to top it off, it's a specific flavour by Rancher or Canonical of moving parts ( e.g. microk8s uses dqlite for storage instead of etcd). So you might stumble on specific to the platform edge cases, and you still have a big part of the k8s complexity to deal with ( in the case of microk8s it tries to abstract some of the complexity with wrappers, but when they fail, you're screwed).
> API will keep evolving but basic objects like deployments and statefulsets that you need for paas like experience are quite stable
Stable now, but Ingress was in beta for quite some time, and the when the beta API gets deprecated, you have to adapt.
So now you're stuck with 3 consul clusters, a vault cluster, whatever you choose for config management, and a nomad cluster. Feels like you didn't gain much from the simplicity of nomad.
In addition to that, knowing Hashicorp's pricing, I bet that set up would run you north of a million a year for enterprise setup.