Skip to content

Incidents & Lessons Learned

This page documents security incidents and mistakes that occurred during development, along with the corrective actions taken and lessons learned. Transparency about mistakes is a core part of a healthy security culture.

Incident 001 - Sensitive Files Committed to main

Date: 10/04/2026 Severity: Medium Repository: aliQuest-frontend Discovered by: Security team during CI/CD setup

What Happened

During the initial SonarQube setup, the following files were accidentally committed and pushed directly to the main branch:

File Risk
.scannerwork/.sonar_lock SonarQube internal file - should never be tracked
.scannerwork/report-task.txt SonarQube internal file - should never be tracked
sonar-project.properties Contains a SonarQube token in plaintext

The sonar-project.properties file was created locally to configure the SonarQube scanner and was not added to .gitignore before committing. This resulted in a valid SonarQube analysis token being exposed in the git history.

Impact

  • The exposed token could have been used by anyone with access to the repository to submit analyses to the SonarQube instance or retrieve project data.
  • The token was a Global Analysis Token, meaning it had access to all projects on the SonarQube instance.

Corrective Actions Taken

  1. Token revoked immediately and new one generated
  2. Files removed from git tracking: git rm -r --cached .scannerwork && git rm --cached sonar-project.properties
  3. Added to .gitignore
  4. Corrective commit pushed to main

Lessons Learned

  • Always add tool-specific config files to .gitignore before running the tool for the first time
  • Secrets should never be stored in local config files - use environment variables or GitHub Actions Secrets instead
  • Gitleaks is now integrated into the PR pipeline to catch future incidents before they reach the repo

Incident 002 - Kubernetes Manifests Broke Traefik and Frontend Deployment

Date: 04/05/2026 Severity: High Repository: dev-git-ops Discovered by: Jabbar Baloghlan after PR merge

What Happened

A PR containing security hardening changes (security headers middleware, pod security context, network policies) was merged into the devgitops repo without prior local testing. The changes caused:

  • Traefik ingress controller to malfunction
  • Frontend deployment to fail
  • dev.aliquest.me to become inaccessible

The root causes were metadata mismatches, cross-namespace issues, and networking conflicts between the new middleware and existing Traefik configuration.

Impact

  • dev.aliquest.me was inaccessible for a period of time
  • Jabbar had to manually diagnose and fix the cluster configuration
  • Wasted time on incident recovery instead of development

Corrective Actions Taken

  1. Jabbar manually diagnosed and fixed the cluster configuration
  2. Stable configuration restored
  3. Security headers were eventually applied correctly after fixes

Lessons Learned

  • Never push Kubernetes infrastructure changes without testing locally first
  • Use kubectl apply --dry-run=client and --dry-run=server to validate manifests before pushing
  • Set up a local k3d cluster for testing infra changes before they hit the real cluster
  • The correct workflow is: local dry-run -> push to PR -> review -> merge -> ArgoCD sync

Prevention Going Forward

  • All K8s manifest changes must pass kubectl apply -k ... --dry-run=client before being pushed
  • A local k3d environment is being set up for pre-merge testing
  • Communicate with Jabbar before pushing any changes that touch Traefik or cross-namespace resources