Fleet: Verify Git Commits For Enhanced Security

In the world of DevOps and cloud-native deployments, security is paramount. When managing your deployments with tools like Rancher and Fleet, ensuring the integrity and authenticity of your code changes is a critical step in preventing security breaches. One significant area where this assurance can be bolstered is through the verification of Git commit signatures. This feature request delves into the need for Fleet to support the verification of Git commit signatures, offering a robust defense against compromised Git servers and ensuring that only trusted code makes its way into your deployments. Let's explore why this is important and how it can be implemented.

The Problem: A Gap in Git Commit Verification

Currently, Fleet, a powerful GitOps tool for Kubernetes, lacks a straightforward method to verify commit signatures. This absence poses a security risk, especially in scenarios where your Git server might be compromised. Imagine a malicious actor gaining access to your Git repository; they could potentially alter commit history, inject harmful code, or misrepresent the origin of changes without immediate detection. While individual teams might implement their own local verification processes, this approach is not scalable, consistent, or easily managed across an organization. The dream scenario is to have Fleet natively handle this verification, providing a unified and secure mechanism for all deployments managed by it. My team, for instance, already automates commit verification locally, and extending this capability to Fleet would significantly enhance our security posture and simplify our deployment workflows. This is not just about convenience; it’s about building a more resilient and trustworthy deployment pipeline.

The lack of integrated commit signature verification means that Fleet, by default, trusts the commits it pulls from a Git repository. This trust is placed on the assumption that the Git server itself is secure and has not been tampered with. However, in a real-world scenario, a compromised Git server can lead to serious security vulnerabilities. An attacker could potentially forge commits, making it appear as though changes originated from a trusted source when they actually came from a malicious actor. This could allow them to deploy backdoors, introduce malware, or disrupt operations. By implementing commit signature verification, Fleet would be able to cryptographically confirm that the commits were indeed authored by the expected individuals and that the commit history has not been altered since it was signed. This adds a crucial layer of assurance, ensuring the integrity of the code being deployed to your Kubernetes clusters. Without this feature, organizations relying solely on Fleet for GitOps are exposed to risks that could be mitigated with a simple, yet powerful, security enhancement.

This verification process typically involves using tools like GPG (GNU Privacy Guard) or SSH keys to sign commits. When a developer makes a commit, they sign it with their private key. This signature can then be verified by anyone who has access to the corresponding public key. If the signature is valid and the public key is trusted, it provides strong assurance that the commit is authentic and has not been tampered with. The absence of this verification in Fleet means that these security benefits are not being fully leveraged within the GitOps workflow. The current workaround, as discussed later, highlights the need for a more direct and integrated solution. The goal is to move from a reactive security model to a proactive one, where potential threats are identified and blocked at the source, rather than discovered after they have impacted the system. Embracing commit signature verification within Fleet is a significant step towards achieving this goal, making your GitOps practices more secure and reliable.

The Dream Solution: Native Commit Verification in Fleet

Our ideal solution involves introducing a configuration option within Fleet that enables commit signature verification. This would be accompanied by a mechanism to specify a list of trusted public keys, which could be either GPG or SSH keys. The proposed configuration might look something like this:

verifyCommits: true
gitCommitsTrustedPublicKeys:
  - gpg:96EC02367D5A393663E7E140E133DC84ACC1429D
  - sk-ecdsa-sha2-nistp256@openssh.com AAAAInNrLWVjZHNhLXNoYTItbmlzdHAyNTZAb3BlbnNzaC5jb20AAAAIbmlzdHAyNTYAAABBBInN1j8laTYxXR6BCO2lO1xrWTj7c6wiWQqN//2JGNqM6JiSSGpkotQs6jCXloAEnwhQo3GbGI94KBtCoVgJReAAAAAEc3NoOg==

With this configuration, Fleet would automatically verify the signature of each incoming commit against the provided list of trusted public keys. If a commit's signature is invalid, or if the signing key is not present in the trusted list, Fleet would hard-fail the deployment. This ensures that only commits that have been properly signed and authorized by trusted individuals or systems are deployed. This approach provides a clear and auditable security mechanism, preventing unauthorized or malicious code from entering your production environment.

Implementing this feature would involve several key components within Fleet. First, a new configuration parameter, such as verifyCommits, would be introduced to enable or disable the feature. Second, a mechanism to manage a list of trusted public keys is necessary. This list could be stored as part of the Fleet configuration or referenced from a secure external source. When Fleet fetches new commits, it would use the appropriate tools (like git-gpg-interface or SSH command-line utilities) to verify the commit signatures. The verification process would need to handle different key types (GPG and SSH) and ensure that the public keys are correctly loaded and used. The failure condition would be clearly defined: if verification fails for any reason (e.g., invalid signature, unknown key), the deployment associated with that commit would be halted. This immediate failure prevents potentially compromised code from progressing further in the deployment pipeline, offering a proactive security measure. The output of the verification process should also be logged, providing visibility into why a deployment was failed, which aids in troubleshooting and auditing.

This native integration would drastically simplify the security setup for users. Instead of complex workarounds, administrators could simply enable the feature and provide the necessary public keys. This would democratize the use of commit signature verification, making it accessible to a wider range of users and organizations, regardless of their technical expertise in setting up complex security infrastructure. The configuration should be flexible enough to allow for different trust models, perhaps including options for specifying key expiration or revocation mechanisms in the future. However, the core requirement is the ability to specify a list of trusted keys and fail deployments on verification failure. This dream solution aims to embed security directly into the GitOps workflow, making it an integral part of the deployment process rather than an afterthought.

Alternatives Considered: A Workaround Strategy

Before envisioning a native solution, our team explored alternative methods to achieve commit verification. The current workaround involves maintaining two Git repositories: one hosted remotely and another within the same Kubernetes cluster as the Fleet controller. A deployment process is set up to continuously monitor the remote repository for updates. Upon detecting changes, the process pulls the commits, verifies their signatures locally, and then pushes the verified commits to the local repository. Fleet then reads updates from this local repository.

This setup, while functional, presents several drawbacks. Firstly, it introduces a significant amount of complexity and is often described as an "anti-pattern." It requires managing an extra repository and setting up a custom synchronization and verification pipeline, which can be confusing for new team members and challenging to maintain. Moreover, this synchronization mechanism cannot be managed by Fleet itself, as it’s a prerequisite for Fleet to operate correctly. This means it must be set up and managed manually, adding an operational burden. The duplication of repositories also increases storage requirements and introduces potential synchronization issues.

While this workaround demonstrates that commit verification is achievable with existing tools, it highlights the need for a more integrated and streamlined approach. The manual setup and inherent complexity make it an undesirable long-term solution for many organizations. It is a testament to the importance of commit verification that teams are willing to implement such elaborate workarounds, but it also underscores the demand for a feature that addresses this security concern directly within the platform. The goal of any robust platform should be to simplify complex security measures, making them accessible and manageable for all users. This alternative strategy serves as a strong justification for the development of the proposed native commit verification feature in Fleet.

This alternative strategy, while achieving the desired outcome of verifying commits, comes with a significant overhead. The manual synchronization process requires careful scripting and monitoring to ensure that the local repository remains in sync with the remote one and that the verification step is always executed correctly. Any failure in this pipeline could lead to outdated or unverified code being deployed. Furthermore, managing two repositories doubles the effort required for backups, access control, and general repository maintenance. For larger teams or organizations with many applications managed by Fleet, this approach quickly becomes unmanageable and error-prone. The fact that this workaround involves an external process that feeds Fleet, rather than being part of Fleet, means that Fleet itself has no visibility into the verification process. This lack of transparency can make auditing and compliance more difficult. The complexity also increases the potential for human error during setup or maintenance, which is the very thing that robust security measures aim to prevent. Therefore, while it’s a viable stop-gap, it’s not a sustainable or ideal solution for organizations looking to embed security deeply into their GitOps workflows.

Conclusion: Enhancing Fleet's Security Posture

Implementing Git commit verification in Fleet is not just a nice-to-have feature; it's a crucial enhancement that significantly strengthens the security of your Kubernetes deployments. By enabling Fleet to verify commit signatures, we can proactively defend against compromised Git servers, ensure the integrity of deployed code, and provide greater peace of mind to development teams. The proposed dream solution, with its clear configuration options and native integration, offers a streamlined and effective way to achieve this.

This feature would empower organizations to build more secure and trustworthy GitOps workflows, aligning with best practices in the DevOps landscape. It moves the responsibility of verification from manual, complex workarounds to an automated, integrated process within Fleet itself. We strongly advocate for the inclusion of this feature to bolster the security capabilities of Fleet and provide users with a robust mechanism to protect their deployments.

For those looking to dive deeper into Git security best practices and understand the underlying technologies like GPG and SSH key management, exploring resources from trusted sources can be invaluable. Understanding how to generate, manage, and distribute your public keys is fundamental to leveraging commit verification effectively.

• For comprehensive information on GPG, you can refer to the official GNU Privacy Guard documentation.
• To learn more about SSH key management and usage, the OpenSSH project website offers extensive resources.

These external links provide a wealth of knowledge that can complement the implementation and use of commit verification within tools like Fleet, ensuring your development practices are both efficient and secure.