Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
version: 2
updates:
# GitHub Actions version updates
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
day: "monday"
time: "09:00"
labels:
- "dependencies"
- "github-actions"
commit-message:
prefix: "ci"
include: "scope"
open-pull-requests-limit: 5
reviewers:
- "cferman"
# Group all GitHub Actions updates into a single PR
groups:
github-actions:
patterns:
- "*"
update-types:
- "minor"
- "patch"

# Pre-commit hooks updates
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "weekly"
day: "monday"
time: "09:00"
labels:
- "dependencies"
- "python"
commit-message:
prefix: "chore"
include: "scope"
open-pull-requests-limit: 3
# Only update if there are requirements files or pre-commit config changes
ignore:
# Ignore patch updates for stability unless security-related
- dependency-name: "*"
update-types: ["version-update:semver-patch"]
2 changes: 1 addition & 1 deletion .github/workflows/pr-validation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ jobs:
uses: actions/checkout@v4

- name: Run ShellCheck
uses: ludeeus/action-shellcheck@master
uses: ludeeus/action-shellcheck@2.0.0
with:
severity: error
ignore_paths: node_modules vendor
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/shell-script-testing.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ jobs:
uses: actions/checkout@v4

- name: Run ShellCheck
uses: ludeeus/action-shellcheck@master
uses: ludeeus/action-shellcheck@2.0.0
with:
severity: warning
ignore_paths: |
Expand Down
4 changes: 2 additions & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -108,15 +108,15 @@ repos:

- id: openshift-manifest-validate
name: Validate OpenShift manifests
entry: scripts/hooks/validate-openshift-manifests.sh
entry: tests/hooks/validate-openshift-manifests.sh
language: script
files: \.(yaml|yml)$
pass_filenames: true
require_serial: true

- id: script-executable
name: Ensure scripts are executable
entry: scripts/hooks/check-script-permissions.sh
entry: tests/hooks/check-script-permissions.sh
language: script
files: ^scripts/.*\.sh$
pass_filenames: true
Expand Down
34 changes: 18 additions & 16 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,10 +49,12 @@ See `docs/architecture.md` for complete details.
├── scripts/ # Operational automation scripts
│ ├── lib/ # Shared libraries (logging, scaling)
│ ├── scale-aap-*.sh # AAP scaling (OpenShift)
│ ├── dr-*.sh # DR orchestration and testing
│ ├── validate-*.sh # Validation and integrity checks
│ ├── start/stop-*.sh # AAP cluster management (RHEL)
│ └── efm-*.sh # EFM integration hooks
├── openshift/dr-testing/ # DR testing CronJob manifests
├── tests/ # Testing and CI infrastructure
│ ├── scripts/ # Test and validation scripts
│ ├── hooks/ # Pre-commit and CI hooks
│ └── openshift/ # DR testing CronJob manifests
└── reports/ # Deployment validation reports
```

Expand Down Expand Up @@ -97,25 +99,25 @@ kustomize build aap-deploy/openshift/

**Local CI checks:**
```bash
./scripts/run-ci-checks-locally.sh
./tests/scripts/run-ci-checks-locally.sh
```

### DR Testing

**Run automated DR failover test:**
```bash
./scripts/dr-failover-test.sh --dc1-context <dc1> --dc2-context <dc2>
./tests/scripts/dr-failover-test.sh --dc1-context <dc1> --dc2-context <dc2>
```

**Measure RTO/RPO:**
```bash
./scripts/measure-rto-rpo.sh --dc1-context <dc1> --dc2-context <dc2>
./tests/scripts/measure-rto-rpo.sh --dc1-context <dc1> --dc2-context <dc2>
```

**Validate AAP data integrity:**
```bash
./scripts/validate-aap-data.sh create-baseline <context>
./scripts/validate-aap-data.sh validate <context>
./tests/scripts/validate-aap-data.sh create-baseline <context>
./tests/scripts/validate-aap-data.sh validate <context>
```

### AAP Cluster Management (OpenShift)
Expand Down Expand Up @@ -247,15 +249,15 @@ See `.cursor/skills/ansible-redhat-cop-practices/` for complete guidelines.
- **Split-brain prevention:** Fencing logic ensures only one primary active

**DR test workflow:**
1. Create baseline: `./scripts/validate-aap-data.sh create-baseline <dc1-context>`
2. Trigger failover: `./scripts/dr-failover-test.sh --dc1-context <dc1> --dc2-context <dc2>`
3. Measure metrics: `./scripts/measure-rto-rpo.sh` (tracks timestamps, calculates RTO/RPO)
4. Validate data: `./scripts/validate-aap-data.sh validate <dc2-context>`
5. Generate report: `./scripts/generate-dr-report.sh --latest`
1. Create baseline: `./tests/scripts/validate-aap-data.sh create-baseline <dc1-context>`
2. Trigger failover: `./tests/scripts/dr-failover-test.sh --dc1-context <dc1> --dc2-context <dc2>`
3. Measure metrics: `./tests/scripts/measure-rto-rpo.sh` (tracks timestamps, calculates RTO/RPO)
4. Validate data: `./tests/scripts/validate-aap-data.sh validate <dc2-context>`
5. Generate report: `./tests/scripts/generate-dr-report.sh --latest`

**Automated testing:**
- OpenShift CronJob: `openshift/dr-testing/cronjob-dr-test.yaml`
- Results stored in PVC: `openshift/dr-testing/pvc-test-results.yaml`
- OpenShift CronJob: `tests/openshift/dr-testing/cronjob-dr-test.yaml`
- Results stored in PVC: `tests/openshift/dr-testing/pvc-test-results.yaml`

See `docs/dr-testing-guide.md` for complete framework.

Expand All @@ -268,7 +270,7 @@ GitHub Actions workflows (`.github/workflows/`):

**Pre-commit hooks** (`.pre-commit-config.yaml`):
- Trailing whitespace, YAML syntax, ShellCheck, markdownlint, secret detection
- Custom hooks: `hooks/check-script-permissions.sh`, `hooks/validate-openshift-manifests.sh`
- Custom hooks: `tests/hooks/check-script-permissions.sh`, `tests/hooks/validate-openshift-manifests.sh`

## Common Tasks

Expand Down
3 changes: 2 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,8 @@ Before contributing, ensure you have:
- `/docs/` for general documentation
- `/aap-deploy/` for AAP deployment docs
- `/db-deploy/` for database deployment docs
- `/scripts/` for script documentation
- `/scripts/` for operational script documentation
- `/tests/` for testing and CI documentation

### Formatting

Expand Down
10 changes: 6 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,10 +95,12 @@ EDB_Testing/
├── scripts/ # Operational automation scripts
│ ├── lib/ # Shared libraries (logging, scaling)
│ ├── scale-aap-*.sh # AAP scaling scripts
│ ├── dr-*.sh # DR orchestration
│ └── validate-*.sh # Validation scripts
├── openshift/ # OpenShift-specific resources
│ └── dr-testing/ # DR testing CronJob
│ ├── efm-*.sh # EFM integration hooks
│ └── start/stop-*.sh # AAP cluster management (RHEL)
├── tests/ # Testing and CI infrastructure
│ ├── scripts/ # Test and validation scripts
│ ├── hooks/ # Pre-commit and CI hooks
│ └── openshift/ # DR testing CronJob manifests
└── .github/ # CI/CD workflows
└── workflows/ # GitHub Actions
```
Expand Down
14 changes: 7 additions & 7 deletions docs/INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ Choose based on your requirements:
- Scaling AAP up/down: See [scale-aap-up.sh](../scripts/scale-aap-up.sh), [scale-aap-down.sh](../scripts/scale-aap-down.sh)
- Monitoring replication: See [monitor-efm-scripts.sh](../scripts/monitor-efm-scripts.sh)
- DR failover: See [efm-orchestrated-failover.sh](../scripts/efm-orchestrated-failover.sh)
- Data validation: See [validate-aap-data.sh](../scripts/validate-aap-data.sh)
- Data validation: See [validate-aap-data.sh](../tests/scripts/validate-aap-data.sh)

---

Expand All @@ -144,10 +144,10 @@ Choose based on your requirements:
| **[efm-orchestrated-failover.sh](../scripts/efm-orchestrated-failover.sh)** | Full DR failover orchestration | Called by EFM (post-promotion) |
| **[efm-aap-failover-wrapper.sh](../scripts/efm-aap-failover-wrapper.sh)** | EFM integration hook | Called by EFM with failover context |
| **[monitor-efm-scripts.sh](../scripts/monitor-efm-scripts.sh)** | Monitor EFM failover events | `./monitor-efm-scripts.sh` (CronJob) |
| **[dr-failover-test.sh](../scripts/dr-failover-test.sh)** | Automated DR testing framework | See [DR Testing Guide](dr-testing-guide.md) |
| **[validate-aap-data.sh](../scripts/validate-aap-data.sh)** | AAP data integrity validation | `./validate-aap-data.sh <dc1\|dc2>` |
| **[measure-rto-rpo.sh](../scripts/measure-rto-rpo.sh)** | RTO/RPO measurement with milestones | `./measure-rto-rpo.sh start <test-id>` |
| **[generate-dr-report.sh](../scripts/generate-dr-report.sh)** | DR test report generation | `./generate-dr-report.sh <test-id>` |
| **[dr-failover-test.sh](../tests/scripts/dr-failover-test.sh)** | Automated DR testing framework | See [DR Testing Guide](dr-testing-guide.md) |
| **[validate-aap-data.sh](../tests/scripts/validate-aap-data.sh)** | AAP data integrity validation | `./validate-aap-data.sh <dc1\|dc2>` |
| **[measure-rto-rpo.sh](../tests/scripts/measure-rto-rpo.sh)** | RTO/RPO measurement with milestones | `./measure-rto-rpo.sh start <test-id>` |
| **[generate-dr-report.sh](../tests/scripts/generate-dr-report.sh)** | DR test report generation | `./generate-dr-report.sh <test-id>` |

**Script Documentation:**
- **[Scripts README](../scripts/README.md)** ⭐ - Quick reference for all scripts
Expand Down Expand Up @@ -175,7 +175,7 @@ Choose based on your requirements:
**Testing:**
- [Component Testing Results](component-testing-results.md) - Script validation (macOS/CRC)
- [AAP Deployment Validation](aap-deployment-validation-crc.md) - End-to-end validation
- [run-ci-checks-locally.sh](../scripts/run-ci-checks-locally.sh) - Run CI checks before pushing
- [run-ci-checks-locally.sh](../tests/scripts/run-ci-checks-locally.sh) - Run CI checks before pushing

---

Expand All @@ -199,7 +199,7 @@ Choose based on your requirements:
**Security considerations:**

- [Pre-commit Secret Detection](../.pre-commit-config.yaml#L89-L98) - `detect-secrets` integration
- [RBAC Configuration](../openshift/dr-testing/serviceaccount.yaml) - DR testing ServiceAccount
- [RBAC Configuration](../tests/openshift/dr-testing/serviceaccount.yaml) - DR testing ServiceAccount
- [EFM Security](enterprisefailovermanager.md#security) - EFM permissions and VIP management

**Planned Documentation:**
Expand Down
4 changes: 2 additions & 2 deletions docs/aap-containerized-quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -291,7 +291,7 @@ curl -k https://aap.example.com/api/v2/ping/
./scripts/manual-failback-dc1.sh

# Measure RTO/RPO
./scripts/measure-rto-rpo.sh
./tests/scripts/measure-rto-rpo.sh
```

---
Expand Down Expand Up @@ -412,7 +412,7 @@ sudo dnf install -y postgres_exporter node_exporter

```bash
# Add to crontab for quarterly testing
0 2 * * 6 /path/to/scripts/dr-failover-test.sh quarterly-$(date +%Y-Q%q)
0 2 * * 6 /path/to/tests/scripts/dr-failover-test.sh quarterly-$(date +%Y-Q%q)
```

### Document Your Deployment
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -482,7 +482,7 @@ script.post.promotion=/usr/edb/efm-4.x/bin/efm-orchestrated-failover.sh %h %s %a
- [`scale-aap-up.sh`](../scripts/scale-aap-up.sh) - Scale AAP to operational state in target datacenter
- [`scale-aap-down.sh`](../scripts/scale-aap-down.sh) - Scale AAP to zero in inactive datacenter
- [`efm-orchestrated-failover.sh`](../scripts/efm-orchestrated-failover.sh) - Full DR failover orchestration
- [`validate-aap-data.sh`](../scripts/validate-aap-data.sh) - Post-failover data validation
- [`validate-aap-data.sh`](../tests/scripts/validate-aap-data.sh) - Post-failover data validation
- [`monitor-efm-scripts.sh`](../scripts/monitor-efm-scripts.sh) - EFM integration monitoring

**Runbook:**
Expand Down
2 changes: 1 addition & 1 deletion docs/component-testing-results.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ Requires multi-cluster environment or access to remote cluster for full failover

### Fix #1: macOS Date Compatibility

**File:** `/scripts/measure-rto-rpo.sh`
**File:** `/tests/scripts/measure-rto-rpo.sh`

**Changes:**

Expand Down
18 changes: 9 additions & 9 deletions docs/dr-replication-implementation-status.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Following the replication architecture validation (score: 7.1/10), this document
- `/scripts/scale-aap-up.sh` - Added database role validation

**Files Created:**
- `/scripts/test-split-brain-prevention.sh` - Automated test script
- `/tests/scripts/test-split-brain-prevention.sh` - Automated test script
- `/docs/split-brain-prevention.md` - Comprehensive documentation

### Changes Made
Expand Down Expand Up @@ -77,15 +77,15 @@ fi

#### 2. Test Script

Created `/scripts/test-split-brain-prevention.sh` with 4 test cases:
Created `/tests/scripts/test-split-brain-prevention.sh` with 4 test cases:
1. Database role detection verification
2. Safety code presence validation
3. Replica scenario simulation (manual test)
4. Dry-run validation

**Usage:**
```bash
./scripts/test-split-brain-prevention.sh <cluster-context>
./tests/scripts/test-split-brain-prevention.sh <cluster-context>
```

#### 3. Documentation
Expand Down Expand Up @@ -171,7 +171,7 @@ The split-brain check is now active in:
**Objective:** Execute comprehensive failover testing to validate documented RTO/RPO targets

**Deliverables:**
1. `/scripts/dr-failover-test.sh` - Automated failover drill script
1. `/tests/scripts/dr-failover-test.sh` - Automated failover drill script
2. `/docs/failover-test-results.md` - Test report template
3. Quarterly testing schedule
4. Measured actual RTO/RPO values
Expand Down Expand Up @@ -199,7 +199,7 @@ The split-brain check is now active in:
- Measure time to AAP availability

3. **Validation:**
- Run `/scripts/validate-aap-data.sh` (to be created)
- Run `/tests/scripts/validate-aap-data.sh` (to be created)
- Verify no data loss
- Confirm AAP job execution

Expand Down Expand Up @@ -380,12 +380,12 @@ spec:
- Send notifications to relevant teams

2. **Begin GAP-REP-002 Implementation:**
- Create `/scripts/dr-failover-test.sh`
- Create `/scripts/validate-aap-data.sh`
- Create `/tests/scripts/dr-failover-test.sh`
- Create `/tests/scripts/validate-aap-data.sh`
- Document test procedures

3. **Validate Split-Brain Prevention:**
- Execute `/scripts/test-split-brain-prevention.sh`
- Execute `/tests/scripts/test-split-brain-prevention.sh`
- Document results
- Add to weekly health check

Expand Down Expand Up @@ -414,7 +414,7 @@ spec:
- **Baseline Validation:** `/reports/dr-replication-validation-report.md`
- **Split-Brain Documentation:** `/docs/split-brain-prevention.md`
- **Scale AAP Script:** `/scripts/scale-aap-up.sh`
- **Test Script:** `/scripts/test-split-brain-prevention.sh`
- **Test Script:** `/tests/scripts/test-split-brain-prevention.sh`
- **DR Scenarios:** `/docs/dr-scenarios.md`
- **EFM Integration:** `/docs/enterprisefailovermanager.md`

Expand Down
6 changes: 3 additions & 3 deletions docs/dr-testing-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ Result: ✅ PASSED
| **ServiceAccount** | RBAC permissions for test execution | - |
| **PVC** | Persistent storage for test results | 5Gi storage |

**Location:** `/openshift/dr-testing/`
**Location:** `/tests/openshift/dr-testing/`

### 3. Test Phases

Expand Down Expand Up @@ -358,7 +358,7 @@ Recovery Point Objective (RPO):

**1. Configure cluster contexts:**

Edit `/openshift/dr-testing/kustomization.yaml`:
Edit `/tests/openshift/dr-testing/kustomization.yaml`:

```yaml
configMapGenerator:
Expand All @@ -379,7 +379,7 @@ oc create secret generic dr-test-kubeconfig \
**3. Deploy CronJob:**

```bash
cd openshift/dr-testing
cd tests/openshift/dr-testing
oc apply -k .
```

Expand Down
Loading
Loading