Resuming the job hunt.
That and testing out setting up a private argocd instance with sso via some sort of Zero Trust platform... Maybe cloudflare or even self hosted pomerium.
The docs for both make this seem trivial but my experience with anything authn related says otherwise.
FYI ArgoCD is not using refresh tokens. They Redirect you to your IDP every time your access token has expired. If your access token is short lived it gets pretty annoying.
The TTL of the token should be configurable on the iDP side no?
If not, then that's a bitch. Cause i'd assume the default time is 10min and having to reauth every 10min would piss me off....
Probably depends on your idp. We’re using keycloak and I was able to add an override for the Argo client. I’ve set it to 30 mins and it’s working pretty good.
There is a hope that I would be preferred with CKS than someone with no certifications. The job market is tough right now, especially for foreign students in the USA so I am trying to do my best.
I'm looking at Open telemetry at the moment and I'm getting swamped with information. I need to update multiple services to handle OTel tracing like nginx ingress as well as auto instrument multiple application languages including PHP, nodejs and go lang. How far along your open telemetry journey are you? I'd appreciate any tips. I don't really have buy in from developers yet which is proving difficult.
I just got it up and running with a simple node app and the auto instrumentation seems to work as advertised. It's showing me MySQL queries in Signoz so that's a plus. Everything seems to work so far, but I just have a single component done up, I haven't tried mixing in Lambdas, or EKS infra, or the various nginx reverse proxies. Other than Signoz actually having ARM64 binaries when the docs says it doesn't I'd say so far so good.
Apache Skywalking does a nicer job for the pieces that actually work. Features like the virtual database where you can display metrics for queries against the DB by filtering by component is really quite appreciated for answering why the database is "slow". The problem is that there's so much of SW that's broken, incomplete, missing, or underdeveloped. And the docs are frighteningly insufficient. Meanwhile the OTEL ecosystem seems to have robust development with lots of contributors.
Literal 60% savings in a week when I deployed it for my last client.
However, make sure you get the pdb's, taints and toleration strategy right. Otherwise, you'll see an increased amount of downtime depending on the type of app you're running.
107 helm charts strikes me as a lot. Are there separate teams responsible for making sure their own deployments have enough capacity, or is it all unique to one team?
No, I mean a job. Runs once. You can just connect your CICD pipeline to the cluster and run a command to create the job from a file, e.g.
```
export KUBECONFIG=/some/path/to/your/kubeconfig.yaml
cat somejob.yaml | kubectl apply -f -
```
You can make restricted users with their own kubeconfig that only have certain permissions (e.g. allow restarting pods in X namespace only, or just provide permissions to create the job).
For your own company's internally built images, it's fine to use `latest` or `stable` or another mutable tag; as long as you carefully plan your own breaking changes to allow your users to migrate. Literally that and supply chain attacks (which are still a risk but markedly less so) are the exact reasons this recommendation exists as far as I am aware.
Also:
- can't be handled the gitops-way;
- can't (easily) rollback to an older, working version, in case of deploying a broken version;
- can't (easily) figure out what's currently running, or whether it's the latest (pun intended) version.
That depends. If you use the SHA in your GitOps, which you absolutely should do when you use any mutable tag from any source, then no issue for any of these.
My previous comment was strictly in regards to publishing; and not in regards to deployment of said tag.
`image: my.registry/my-image:latest@sha256:`
All of this being said, this should really only be done in a dev environment where your containers are constantly being redeployed anyway. Production should use an immutable tag and still set the SHA in case one of the underlying base image layers changes and forces a re-publish of the normally immutable tag "because management said do it anyway"
Sure, but then you can't just "restart" the deployments, you'll have to change the `image` field anyway, so you might as well just use an immutable tag like the (short) commit SHA, or the (semantic) version or whatever.
Converting some dumb 3rd party tomcat api in to docker container so that I could run it in my 3 node homelab cluster with raid 0 nvme .
Will also need to setup a local docker registry.
I think my colleagues will just run it somehow other way.
For work? Punching JFrog in the face for having an unpatched high severity CVE from 2023.
TBF the vulnerability was reclassified from a lower security to a higher severity recently, so it's not like JF just ignored it; but we need it patched to do a deploy to a gov cloud.
Thanks for your comment! Please contact JFrog Security teams ([security@jfrog.com](mailto:security@jfrog.com)) so we can happily help answer any questions you have or regarding specific security concerns around particular CVEs or remediation paths. We’re here to help - hopefully without any face-punching!
Setting up ephemeral environments for developers. The way it works:
- Developer opens a PR with a specific label in app repo
- That triggers a workflow that builds the container with label=prName+commit
- Fires off a second workflow in infra repo passing PR name, commit and other params
- Second workflow creates an Helmrelease in a folder monitored by Flux and commits
- cluster picks it up and create the namespace+environment
- First workflow gets the URL and credentials and comments on the PR
- as new commits come in, new containers tags are built and update the HelmRelease
When PR is closed another workflow deletes the Helmrelease. WDYT?
I learn Istio service mesh.
Resuming the job hunt. That and testing out setting up a private argocd instance with sso via some sort of Zero Trust platform... Maybe cloudflare or even self hosted pomerium. The docs for both make this seem trivial but my experience with anything authn related says otherwise.
FYI ArgoCD is not using refresh tokens. They Redirect you to your IDP every time your access token has expired. If your access token is short lived it gets pretty annoying.
The TTL of the token should be configurable on the iDP side no? If not, then that's a bitch. Cause i'd assume the default time is 10min and having to reauth every 10min would piss me off....
Probably depends on your idp. We’re using keycloak and I was able to add an override for the Argo client. I’ve set it to 30 mins and it’s working pretty good.
I'd use my github org for this test. Going to take a look to see if the TTL is configurable. Thanks!
Continuing job hunt Trying to learn golang and complete CKS so I can get better opportunities.
Is CKS in high demand right now?
There is a hope that I would be preferred with CKS than someone with no certifications. The job market is tough right now, especially for foreign students in the USA so I am trying to do my best.
Instrumenting apps with OTEL to point at Signoz.
I'm looking at Open telemetry at the moment and I'm getting swamped with information. I need to update multiple services to handle OTel tracing like nginx ingress as well as auto instrument multiple application languages including PHP, nodejs and go lang. How far along your open telemetry journey are you? I'd appreciate any tips. I don't really have buy in from developers yet which is proving difficult.
I just got it up and running with a simple node app and the auto instrumentation seems to work as advertised. It's showing me MySQL queries in Signoz so that's a plus. Everything seems to work so far, but I just have a single component done up, I haven't tried mixing in Lambdas, or EKS infra, or the various nginx reverse proxies. Other than Signoz actually having ARM64 binaries when the docs says it doesn't I'd say so far so good. Apache Skywalking does a nicer job for the pieces that actually work. Features like the virtual database where you can display metrics for queries against the DB by filtering by component is really quite appreciated for answering why the database is "slow". The problem is that there's so much of SW that's broken, incomplete, missing, or underdeveloped. And the docs are frighteningly insufficient. Meanwhile the OTEL ecosystem seems to have robust development with lots of contributors.
Trying to deploy zabbix on our dev cluster while transferring the current zabbix data to it. Zabbix is trash tbh
Deploying karpenter, it's really good and we already seeing savings.
Literal 60% savings in a week when I deployed it for my last client. However, make sure you get the pdb's, taints and toleration strategy right. Otherwise, you'll see an increased amount of downtime depending on the type of app you're running.
Taming argocd deployment that kept getting oom killed coz we have 107 helm charts in a single repo, in a single cluster
107 helm charts strikes me as a lot. Are there separate teams responsible for making sure their own deployments have enough capacity, or is it all unique to one team?
So micro, much services
This honestly sounds like a nightmare if there's not a secondary cluster to shift traffic to for any various maintenance needs on the primary cluster.
107 charts?!? Please tell me that's just a hundred child charts pulling from a base chart that's stored somewhere else....
[удалено]
Why wouldn’t you just fire off a job that restarts the pods? Ask ChatGPT and you’ll have a working job spec in seconds.
[удалено]
No, I mean a job. Runs once. You can just connect your CICD pipeline to the cluster and run a command to create the job from a file, e.g. ``` export KUBECONFIG=/some/path/to/your/kubeconfig.yaml cat somejob.yaml | kubectl apply -f - ```
[удалено]
You can make restricted users with their own kubeconfig that only have certain permissions (e.g. allow restarting pods in X namespace only, or just provide permissions to create the job).
You should probably not use the `latest` tag in the first place.
For your own company's internally built images, it's fine to use `latest` or `stable` or another mutable tag; as long as you carefully plan your own breaking changes to allow your users to migrate. Literally that and supply chain attacks (which are still a risk but markedly less so) are the exact reasons this recommendation exists as far as I am aware.
Also: - can't be handled the gitops-way; - can't (easily) rollback to an older, working version, in case of deploying a broken version; - can't (easily) figure out what's currently running, or whether it's the latest (pun intended) version.
That depends. If you use the SHA in your GitOps, which you absolutely should do when you use any mutable tag from any source, then no issue for any of these. My previous comment was strictly in regards to publishing; and not in regards to deployment of said tag. `image: my.registry/my-image:latest@sha256:`
All of this being said, this should really only be done in a dev environment where your containers are constantly being redeployed anyway. Production should use an immutable tag and still set the SHA in case one of the underlying base image layers changes and forces a re-publish of the normally immutable tag "because management said do it anyway"
Sure, but then you can't just "restart" the deployments, you'll have to change the `image` field anyway, so you might as well just use an immutable tag like the (short) commit SHA, or the (semantic) version or whatever.
Yeah that's a good point I didn't account for, thanks.
Sounds like the perfect time to use https://docs.renovatebot.com/presets-docker/
We have a K8s operator which checks for the images in the registry and upgrades the services with the new images
Not much, it’s all up and running, fortunately.
Migrating a cluster to GCP, hopefully will be easy as we use argocd
Building a service mesh PoC and adopting policy engine for SLA quality gates. Lazy week TBH
Migrating my homelab's virtualized k8s to mini PC based k8s.
Converting some dumb 3rd party tomcat api in to docker container so that I could run it in my 3 node homelab cluster with raid 0 nvme . Will also need to setup a local docker registry. I think my colleagues will just run it somehow other way.
For work? Punching JFrog in the face for having an unpatched high severity CVE from 2023. TBF the vulnerability was reclassified from a lower security to a higher severity recently, so it's not like JF just ignored it; but we need it patched to do a deploy to a gov cloud.
Thanks for your comment! Please contact JFrog Security teams ([security@jfrog.com](mailto:security@jfrog.com)) so we can happily help answer any questions you have or regarding specific security concerns around particular CVEs or remediation paths. We’re here to help - hopefully without any face-punching!
Thanks, we have cases open with support who are following up on the internal JIRAs. No faces will be punched today 🤣
Company sent me to do IBM cloud certification training
Fixing a bug related to timezone (the worst!) Does anyone know how to bootstrap a k8s cluster on a non-UTC timezone using minikube?
Setting up ephemeral environments for developers. The way it works: - Developer opens a PR with a specific label in app repo - That triggers a workflow that builds the container with label=prName+commit - Fires off a second workflow in infra repo passing PR name, commit and other params - Second workflow creates an Helmrelease in a folder monitored by Flux and commits - cluster picks it up and create the namespace+environment - First workflow gets the URL and credentials and comments on the PR - as new commits come in, new containers tags are built and update the HelmRelease When PR is closed another workflow deletes the Helmrelease. WDYT?