Report Generation Fails with "lookup prometheus-server-exp: no such host"
| KB ID: | 4570 |
| Product: | Veeam Kasten for Kubernetes |
| Published: | 2024-06-14 |
| Last Modified: | 2024-06-14 |
Oops! Something went wrong.
Please, try again later.
Challenge
After enabling Veeam Kasten for Kubernetes reports, reports are not generated.
When running an on-demand report policy, the following error message is observed within the executor logs:
"message":"Post "http://prometheus-server-exp:80/k10/prometheus/api/v1/query\": dial tcp: lookup prometheus-server-exp on 192.0.0.10:53: no such host"
cause":{"message":"Failure in subordinate phase","function":"kasten.io/k10/kio/exec/phases/phase.(*queueAndWaitChildrenPhase).processGroup","linenumber":196,"file":"kasten.io/k10/kio/exec/phases/phase/queue_and_wait_children.go:196","fields":[{"name":"FailedSubPhases","value":[{"Phase":"","Err":{"cause":{"cause":{"cause":{"cause":{"cause":{"message":"Post "http://prometheus-server-exp:80/k10/prometheus/api/v1/query\": dial tcp: lookup prometheus-server-exp on 192.0.0.10:53: no such host"},"fields":[{"name":"query","value":"sum(round(increase(action_backup_skipped_overall{cluster=""}[24h])))"}]
Cause
This issue occurs when the Prometheus server hostname cannot be resolved via DNS, which prevents Veeam Kasten for Kubernetes pods (i.e., Executor pods) from capturing the Prometheus metrics. Since the Prometheus instance metrics cannot be collected, the Veeam Kasten for Kubernetes reports, which depend on those metrics, cannot be generated.
This may also occur when the DNS server is not running.
lookup prometheus-server-exp on 192.0.0.10:53: no such host
Solution
Cluster DNS Validation
Validate the cluster DNS by following this DNS troubleshooting guide.
The following commands can be used to help to check DNS functionality within the cluster:
- DNS debugging example pod YAML from kubernetes documentation:
- exec into the pod to run the nslookup command with the name <svcName.NamespaceName>:
Example of an issue with the cluster DNS:
> kubectl -n default exec -i -t dnsutils -- nslookup prometheus-server-exp.kasten-io Server: 10.245.0.10 Address: 10.245.0.10#53 ** server can't find prometheus-server-exp.kasten-io: NXDOMAIN command terminated with exit code 1
Troubleshoot DNS Service Functionality
The next steps are to check if the DNS service is running, whether endpoints are exposed, and restart the DNS deployment.
The commands below demonstrate troubleshooting with coredns as that is the built-in DNS service that comes with a Kubernetes cluster. If you are using a different DNS service, the troubleshooting procedure should be similar, check documentation to be sure.
- Check if DNS pods are running filtering by label kube-dns:
- Check if DNS endpoints are discovered:
- Try restarting DNS services (rollout):
If this KB article did not resolve your issue or you need further assistance with Veeam software, please create a Veeam Support Case.
To submit feedback regarding this article, please click this link: Send Article Feedback
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.
