At StepSecurity, our mission is to bolster the security of CI/CD pipelines by providing real-time monitoring and network visibility. StepSecurity Harden-Runner community tier is used by more than 4,500 open-source projects. In addition, several enterprise customers use Harden-Runner to protect against runtime CI/CD security attacks.
This blog post details a recent incident where Harden-Runner successfully identified unexpected network activity across multiple customers, highlighting the importance of runtime security monitoring in CI/CD.
The Anomaly Detection Event
Starting November 8, 2024, 6:32 PM UTC, Harden-Runner's anomaly detection feature issued alerts about new outbound network connections from GitHub Actions workflows across multiple customers. These connections were directed to the domain api.ipify.org, which was not part of the established baseline for any of the affected workflows. The widespread nature of these alerts indicated a systemic incident impacting multiple runs across several StepSecurity customers.
Investigation and Findings
Our team promptly initiated an investigation to understand the scope and cause of the anomaly. Here's what we discovered:
- Affected Workflows: Our investigation revealed that the unexpected network calls were originating exclusively from GitHub-hosted runners. This was a crucial discovery, as none of our customers using self-hosted runners were impacted. This differentiation helped us narrow our focus to GitHub's infrastructure, streamlining our investigative efforts.
- Destination Domain: The domain api.ipify.org is a service that returns the public IP address of the caller—a functionality not typically required by the workflows in question.
- Responsible Process: Harden-Runner's process mapping identified that a process named provjobd was making these outbound calls.
- Lack of Documentation: Searches for provjobd and references to api.ipify.org in official GitHub documentation yielded no results, suggesting this was not a publicly documented change.
Communication with GitHub Support
Understanding the potential implications of this systemic anomaly, we reached out to GitHub Support for clarification. We are grateful for their prompt and informative response, which provided the following insights:
- Internal Tool: The provjobd process is an internal tool used temporarily by GitHub to collect diagnostic metrics for Actions runners.
- Expected Behavior: The network calls to api.ipify.org were expected during this temporary period and were not indicative of malicious activity.
- Resolution: GitHub assured us that the process was temporary and that they did not anticipate rolling it out again.
We appreciate GitHub Support's transparency and assistance in resolving this matter swiftly.
Resolution and Customer Communication
We took the following steps while investigating this issue:
- Customer Notifications: We promptly informed all affected enterprise customers about the incident, sharing our initial findings to alleviate any concerns.
- Focus on Self-Hosted Runners: The fact that self-hosted runners were not impacted allowed us to reassure customers using those runners and helped us pinpoint the issue as related to GitHub-hosted infrastructure.
- Transparency with the Community: For our community-tier users, we are sharing this blog post to provide transparency and insights into the incident.
Demonstrating Harden-Runner's Capabilities
This incident underscores the effectiveness of Harden-Runner's anomaly detection feature:
- System-Wide Detection: Harden-Runner was able to detect anomalies across multiple customers, highlighting its ability to identify systemic issues.
- Real-Time Alerts: The tool immediately flagged the unexpected outbound traffic, enabling rapid investigation.
- Process-Level Visibility: Mapping network calls to specific processes (such as provjobd) was crucial in identifying the source of the anomaly.
- Baseline Monitoring: By establishing a baseline of normal network activity, Harden-Runner can detect deviations that may indicate security issues.
Conclusion
Security in CI/CD pipelines is paramount, and incidents like this highlight the necessity of vigilant, real-time monitoring. We are grateful to GitHub Support for their transparency and prompt assistance, which helped us quickly resolve the issue. StepSecurity Harden-Runner continues to provide robust security features, empowering both open-source projects and enterprise customers to operate securely. If you have any questions or need assistance with securing your CI/CD pipelines, please do not hesitate to contact us.
