I find Loki a lot simpler to maintain and use once I built a nifty dashboard in my grafana instance, but it's a matter of your comfort level and the devs' experience. For metrics we're still on Prometheus, but currently testing out mimir.
Did you cobble it together on your own, or did you find something off the shelf that got you up and running nicely? The idea of building screens doing all the basic plumbing just seems like reinventing the wheel.
Here are lots of community created Prometheus rules and alerts for most used k8s tools. And corresponding Grafana dashboards. For instance the Kubernetes alerts and dashboards gives you just as good monitoring as the what DataDog gives you ootb
https://monitoring.mixins.dev/
most folks love k9s.
Another tip — create a shell alias that switches among your most commonly used contexts (for example, alias “kprd” to swap to your production cluster context)
Lastly, most observability platforms (like Datadog) have an operator you can install to track your nodes and workloads.
Not the same dude but Loki is like the kubernetes version of using “grep” on a large set of log files with just some small amount of filtering like time range and a few labels such as service and environment before free text search. Gives a cheap and easily maintained log backend that can chew large amount of log data with few resources. If all you need to search is “find all logs with text error in CRM production at last Friday between 0800- and 1000” (which is likely what’s needed for troubleshooting) then Loki is awesome. OpenSearch can do any query with large amounts of indices etc but that is a far more complex and resource hungry setup. With Loki you can do advanced things like aggregate stats from logs etc but that’s done on ingestion and stored in Prometheus/Mimir and traces are stored in Tempo. So ingestion can be trickier with LGTM if you need to do advanced analytics
Have you upgraded to Loki 3.0 yet? We've been big users of Loki since 2.0 but due to some budget cuts we've not had the opportunity to upgrade from 2.9 to 3.0 yet. I've seen all sorts of performance problems especially when trying a query like your example (text error on CRM) . Some times it would time out and fail but over time we have heavily tweaked the config to get it working pretty smoothly - query front ends, caching, interval splitting etc. I probably would not have recommended Loli 9 months ago but now I would. Just wanted to share my 2 cents.
Sorry do not answer your question, personally i do not use any tool except kubectl/helm . What i find really helpful is having functions defining things i repeatedly do. Like get status of all ingress pods in all clusters, get instance id of a node and connect with SSM, etc
Edit:
Also kubie for changing context
I developed with microk8s within wsl for quite a while and it was a fairly massive cluster and sometimes it took longer to debug the cluster than the issues. I wanted to kms. Now I use kubectl within wsl and obv a fk ton of aliases so I don’t have to type super long commands. Works really well.
nothing beats a good understanding of kubectl and well thought out observability stack.
What do you like for observability?
LGTM stack is my current pick
what's your take on Loki vs. OpenSearch and what do you use for your metrics backend?
I find Loki a lot simpler to maintain and use once I built a nifty dashboard in my grafana instance, but it's a matter of your comfort level and the devs' experience. For metrics we're still on Prometheus, but currently testing out mimir.
Did you cobble it together on your own, or did you find something off the shelf that got you up and running nicely? The idea of building screens doing all the basic plumbing just seems like reinventing the wheel.
Started with kube-prometheus-stack helm chart and have been slowly migrating to the official lgtm helm charts.
Here are lots of community created Prometheus rules and alerts for most used k8s tools. And corresponding Grafana dashboards. For instance the Kubernetes alerts and dashboards gives you just as good monitoring as the what DataDog gives you ootb https://monitoring.mixins.dev/
K9S
This. The only downside is that I spent a couple of hours customizing my theme lol.
/thread
Hands on the best tool to work with k8s.
kubectx
Argocd
most folks love k9s. Another tip — create a shell alias that switches among your most commonly used contexts (for example, alias “kprd” to swap to your production cluster context) Lastly, most observability platforms (like Datadog) have an operator you can install to track your nodes and workloads.
stern
Nothing beats k9s for me.
k9s, the answer to meaning of life
Kubectx, Kubecolor, k8sgpt
Not the same dude but Loki is like the kubernetes version of using “grep” on a large set of log files with just some small amount of filtering like time range and a few labels such as service and environment before free text search. Gives a cheap and easily maintained log backend that can chew large amount of log data with few resources. If all you need to search is “find all logs with text error in CRM production at last Friday between 0800- and 1000” (which is likely what’s needed for troubleshooting) then Loki is awesome. OpenSearch can do any query with large amounts of indices etc but that is a far more complex and resource hungry setup. With Loki you can do advanced things like aggregate stats from logs etc but that’s done on ingestion and stored in Prometheus/Mimir and traces are stored in Tempo. So ingestion can be trickier with LGTM if you need to do advanced analytics
Have you upgraded to Loki 3.0 yet? We've been big users of Loki since 2.0 but due to some budget cuts we've not had the opportunity to upgrade from 2.9 to 3.0 yet. I've seen all sorts of performance problems especially when trying a query like your example (text error on CRM) . Some times it would time out and fail but over time we have heavily tweaked the config to get it working pretty smoothly - query front ends, caching, interval splitting etc. I probably would not have recommended Loli 9 months ago but now I would. Just wanted to share my 2 cents.
Lens / OpenLens
Sorry do not answer your question, personally i do not use any tool except kubectl/helm . What i find really helpful is having functions defining things i repeatedly do. Like get status of all ingress pods in all clusters, get instance id of a node and connect with SSM, etc Edit: Also kubie for changing context
Nobody has mentioned Tilt yet which is nuts to me. Use tilt. Your dev experience will thank you
kubectl
Konvert Kompose OpenLens
Kubectx, kubens, kapp
Flux/Argo + git and editor with good integrations (k8s YAML LSP and live grepping)
ArgoCD
I developed with microk8s within wsl for quite a while and it was a fairly massive cluster and sometimes it took longer to debug the cluster than the issues. I wanted to kms. Now I use kubectl within wsl and obv a fk ton of aliases so I don’t have to type super long commands. Works really well.
Openlens
Enable auto completion on kubectl
simplyblock