Replicator crashes when watches expire #32
Description
When watches expire (and possibly at other times), the pod crashes because it cannot write to a log file. I believe this is related to kubernetes/kubernetes#61006. The fix for this appears to have gone in to 1.13, and this repository appears to depend on the 1.14 client, so I'm not sure what's going on there.
Name: replicator-deployment-766c46874f-54bs2
Namespace: kube-system
Priority: 0
PriorityClassName: <none>
Node: ip-172-31-15-37.us-west-2.compute.internal/172.31.15.37
Start Time: Fri, 14 Feb 2020 13:37:22 -0800
Labels: app=replicator
pod-template-hash=766c46874f
Annotations: <none>
Status: Running
IP: 100.96.6.2
Controlled By: ReplicaSet/replicator-deployment-766c46874f
Containers:
replicator:
Container ID: docker://54fa9bce17bea730e50713ab35a0ad9de93b4321ac17f4d598870fa5c4995ec8
Image: quay.io/mittwald/kubernetes-replicator:latest
Image ID: docker-pullable://quay.io/mittwald/kubernetes-replicator@sha256:9cd515802fee4859d1978a39e2a2e4278b691c54a4236df1f5e663d9c9b37c2a
Port: <none>
Host Port: <none>
State: Running
Started: Fri, 14 Feb 2020 14:35:05 -0800
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Fri, 14 Feb 2020 14:22:34 -0800
Finished: Fri, 14 Feb 2020 14:35:03 -0800
Ready: True
Restart Count: 3
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from replicator-token-xz6mg (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
replicator-token-xz6mg:
Type: Secret (a volume populated by a Secret)
SecretName: replicator-token-xz6mg
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 28m (x4 over 86m) kubelet, ip-172-31-15-37.us-west-2.compute.internal Pulling image "quay.io/mittwald/kubernetes-replicator:latest"
Normal Pulled 28m (x4 over 86m) kubelet, ip-172-31-15-37.us-west-2.compute.internal Successfully pulled image "quay.io/mittwald/kubernetes-replicator:latest"
Normal Created 28m (x4 over 86m) kubelet, ip-172-31-15-37.us-west-2.compute.internal Created container replicator
Normal Started 28m (x4 over 86m) kubelet, ip-172-31-15-37.us-west-2.compute.internal Started container replicator
^ Note how the pod restarted 4 times
-> % kubectl -n kube-system logs replicator-deployment-766c46874f-54bs2 --previous --tail=10
2020/02/14 22:35:03 updating secret default/better-tls-3.dev.source.ai-tls
2020/02/14 22:35:03 updating dependent secret kube-system/better-tls-3.dev.source.ai-tls -> jenkins/better-tls-3.dev.source.ai-tls
2020/02/14 22:35:03 updating secret jenkins/better-tls-3.dev.source.ai-tls
2020/02/14 22:35:03 secret jenkins/better-tls-3.dev.source.ai-tls is replicated from kube-system/better-tls-3.dev.source.ai-tls
2020/02/14 22:35:03 updating secret jenkins/better-tls-3.dev.source.ai-tls
2020/02/14 22:35:03 secret kube-system/better-tls-3.dev.source.ai-tls has 2 dependents
2020/02/14 22:35:03 updating dependent secret kube-system/better-tls-3.dev.source.ai-tls -> default/better-tls-3.dev.source.ai-tls
2020/02/14 22:35:03 updating secret default/better-tls-3.dev.source.ai-tls
W0214 22:35:03.908603 1 reflector.go:289] pkg/mod/k8s.io/client-go@v11.0.1-0.20190409021438-1a26190bd76a+incompatible/tools/cache/reflector.go:94: watch of *v1.Secret ended with: too old resource version: 36417 (40656)
log: exiting because of error: log: cannot create log: open /tmp/replicator.replicator-deployment-766c46874f-54bs2.unknownuser.log.WARNING.20200214-223503.1: no such file or directory