A Tale of a Self Hosting Fix
This will be a short post about self hosting. There are many problems that tend to be fabricated when you self host. I use the term fabricated because most of the time they are indicative of your unique environment. This is one of those situations.
The Problem
There is a fun race condition that occurs if my back up batteries die off and I completely lose power to my cluster.
Two things happen:
- You can no longer access Gitea since the machine it runs through no longer has DNS
- You can no longer pull my PiHole image which is stored in Gitea, which my machine uses.
See the problem.
The short answer to the "but, Sean"'s is "No". Alluding to my original point, this issue is most likely generated by my setup. The DNS is hosted on a k8s cluster with an always pull configuration.
But...
Yes, I could just set the pull policy to Never, but, that impacts my changes and/or software updates.
Yes, I could use unique tags for created images, but, that just seems like an unnecessary amount of work impacting too many other workflows.
What about IfNotPresent? See, "Never"
"The Quick N' Dirty"
The glamour of three lines of code!
#!/bin/bash
# This is a script to rerun pihole, removing auto pull for the times when power outage and DNS is down.
#voodoo
sed -i 's/imagePullPolicy: Always/imagePullPolicy: Never/g' deployments/pihole.yml
kubectl rollout restart deployments/pihole
#unvoodoo
sed -i 's/imagePullPolicy: Never/imagePullPolicy: Always/g' deployments/pihole.yml
This is my easy work around. Why it works:
- Upon crash, the deployment will attempt to start up and fail
- Theoretically, there should be a cached, or multiple cached images on different nodes
- The voodoo and unvoodoo are temporary changes to a static configuration.
Yay! We Did It!
Be proud of your small wins. I couldn't find a best practice that worked in the context of my environment, so I bandaged up a fix. Is it pretty? No, not really. Does it solve a problem that really bothers me? Yes, yes it does. Now, in the case of complete outage. One script solves my manual configuration problem. The next step is automating the re-deployment of ~30 services that may be dependent on the availability of other services.
