ZAP On Kubernetes/Istio: Fix UI & Scan Connection Refused
Deploying OWASP ZAP (Zed Attack Proxy) in a containerized environment like Kubernetes, especially when integrated with a service mesh like Istio, can introduce complexities. This article addresses common issues encountered when running ZAP 2.16 as a Docker container on Kubernetes behind Istio, specifically focusing on the inability to access the ZAP UI and the occurrence of "Connection refused" errors during remote scans. Let's dive deep into how to resolve these problems and ensure your ZAP instance is fully functional.
Understanding the Problem: ZAP, Kubernetes, and Istio
When you're facing issues with ZAP UI access and remote scan failures, it's crucial to understand how each component—ZAP, Kubernetes, and Istio—interacts. ZAP, in this context, acts as a dynamic application security testing (DAST) tool, scanning your web applications for vulnerabilities. Kubernetes provides the orchestration, managing the ZAP container, while Istio enhances the environment with features like traffic management, security, and observability. When these components don't align correctly, you'll likely encounter connection issues.
Key Configuration Details
- ZAP Version: 2.16
- Deployment: Docker container in Kubernetes
- Ingress: Istio Gateway with TLS termination
- ZAP Mode: Daemon (configured with specific parameters)
- Target URL for API calls:
https://zap-nonprod.mydomain.com/
Common Symptoms
The primary indicators of this issue include:
- Inability to Access ZAP UI: Attempting to access the ZAP UI via Istio Ingress fails.
- Connection Refused Errors: Remote scans consistently fail with a "Connection refused" error, hindering your ability to perform security assessments.
Diving into the Logs
Log analysis is critical for diagnosing the root cause. A typical log snippet might look like this:
92 [ZAP-daemon] INFO org.zaproxy.addon.network.ExtensionNetwork - ZAP is now listening on 0.0.0.0:8080
...
Connect to http://zap-nonprod.mydomain.com:80 [...] failed: Connection refused
This log suggests that while ZAP is listening on port 8080 inside the pod, the internal ZAP client is attempting to connect to port 80, which may not be exposed or properly routed. This discrepancy is a key area to investigate.
Diagnosing the Root Cause
Several factors can contribute to these issues. Here's a breakdown of the most common culprits:
- Port Mismatch: The most frequent cause is a mismatch between the port ZAP is listening on (8080) and the port to which Istio is routing traffic (potentially 80 or 443).
- Istio Routing Configuration: Incorrectly configured Istio routing rules can prevent traffic from reaching the ZAP pod on the correct port.
- TLS Termination: While TLS termination at the Istio Gateway is best practice, misconfigurations can lead to issues if ZAP is not correctly configured to handle HTTP traffic after TLS termination.
- ZAP Configuration: Incorrect ZAP configurations, especially related to API addresses and CORS, can also contribute to connectivity problems.
Step-by-Step Solutions
Now, let's walk through the solutions to resolve these issues:
1. Verifying and Correcting Port Configuration
Ensure that ZAP is configured to listen on the correct port and that Istio is routing traffic to that port. Here’s how:
-
ZAP Configuration: Double-check your ZAP configuration to confirm it's listening on port 8080 (or your intended port).
zap.sh -daemon -host 0.0.0.0 -port 8080 ... -
Istio Gateway Configuration: Verify that your Istio Gateway and VirtualService are configured to route traffic to the ZAP service on port 8080. Here’s an example:
apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: zap-gateway spec: selector: istio: ingressgateway servers: - port: number: 443 name: https protocol: HTTPS hosts: - "zap-nonprod.mydomain.com" tls: mode: SIMPLE credentialName: zap-tls --- apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: zap-virtualservice spec: hosts: - "zap-nonprod.mydomain.com" gateways: - zap-gateway http: - match: - uri: prefix: "/" route: - destination: host: dop-zap-service port: number: 8080In this example, the VirtualService routes all traffic from
zap-nonprod.mydomain.comto thedop-zap-serviceon port 8080.
2. Adjusting Istio Routing
If the port configuration is correct, the issue might be with the routing rules. Ensure that Istio is correctly routing traffic to the ZAP pod. This involves checking the VirtualService and Service definitions.
-
VirtualService: As shown above, the VirtualService should correctly map the incoming host and path to the ZAP service.
-
Service Definition: The Kubernetes Service definition should target the ZAP pod and expose port 8080.
apiVersion: v1 kind: Service metadata: name: dop-zap-service spec: selector: app: dop-zap-service ports: - protocol: TCP port: 8080 targetPort: 8080
3. Handling TLS Termination
When Istio handles TLS termination, ZAP receives HTTP traffic. Ensure that ZAP is configured to handle this. The provided ZAP configuration includes:
-config api.addrs.addr.scheme=https
-config api.addrs.addr.port=443
These settings can cause issues because ZAP is being told the traffic is coming in via HTTPS on port 443 when it is actually receiving HTTP on port 8080. Remove or modify these settings to align with the actual traffic flow.
Consider adjusting these settings based on your actual traffic flow:
-config api.addrs.addr.scheme=http
-config api.addrs.addr.port=8080
4. Reviewing ZAP Configuration
Ensure that your ZAP configuration aligns with your environment. Key areas to check include:
- API Addresses: Verify that the API addresses are correctly configured to reflect the actual scheme (HTTP/HTTPS) and port.
- CORS Configuration: The provided configuration includes
api.cors.enabled=trueandapi.cors.allowOrigin=*. While this allows all origins, ensure it aligns with your security requirements. For production environments, consider restricting the allowed origins.
5. Kubernetes Network Policies
Kubernetes Network Policies can restrict traffic flow between pods. Ensure that there are no Network Policies preventing traffic from reaching the ZAP pod from other pods or services within the cluster. Review any existing Network Policies to confirm they allow traffic to port 8080 on the ZAP pod.
Practical Steps for Implementation
- Apply the Corrected Configurations: After adjusting the Istio and ZAP configurations, apply the changes to your Kubernetes cluster.
- Restart ZAP Pod: Restart the ZAP pod to ensure the new configuration is loaded.
- Test UI Access: Attempt to access the ZAP UI via the Istio Ingress.
- Run a Remote Scan: Initiate a remote scan to verify that the "Connection refused" error is resolved.
- Monitor Logs: Continuously monitor the ZAP logs and Istio logs to identify any further issues.
Example Scenario
Let's consider a scenario where the Istio VirtualService was misconfigured, routing traffic to port 80 instead of 8080. After updating the VirtualService to correctly route traffic to port 8080, restarting the ZAP pod, and applying the new configuration, the UI becomes accessible, and remote scans complete successfully.
Conclusion
Troubleshooting ZAP deployments on Kubernetes with Istio requires a methodical approach. By verifying port configurations, adjusting Istio routing, handling TLS termination correctly, and reviewing ZAP configurations, you can resolve common issues like inaccessible UIs and "Connection refused" errors. Regularly monitoring logs and testing configurations are essential for maintaining a stable and secure ZAP environment. By following the steps outlined in this article, you'll be well-equipped to tackle these challenges and ensure your ZAP instance is performing optimally.
For further reading on Istio configuration, refer to the official Istio documentation. This should help solidify your understanding and aid in future troubleshooting.