Playbook C · radical shadow-run / data-diff first
C · Production-as-Oracle
Skip test design almost entirely. Convert a production backup taken at the start of a closed period, replay that period’s real business events (postings, documents, batch chains) on the converted system, and diff the resulting period-end state against what production actually produced. Production is the oracle and AI is used only for triage, KID matching and residual tests.
3.8×
first deployment vs 4,820 PD baseline estimate
7.8×
mature / productised estimate
6.5× / 12.9×
ceiling at very large scale (first / mature) estimate
~11 months
QA calendar for a two-step IBM i path estimate
When to offer it
- Pure technical/brownfield conversion with stable processes (typical for IBM i shops).
- Large programs or repeat engagements: this is the only playbook with a ceiling clearly above 10×.
- Customers whose auditors accept period-level reconciliation as primary evidence.
Timeline
04812162024283236404448
0 · Feasibility spike + go/no-go
1 · Replay harness + diff platform
2 · Unicode step: full diff gate
3 · Shadow periods per mock
4 · Residual tests, UAT, dress rehearsal
Calendar gains are smaller than effort gains: the technical steps (Unicode conversion, DMO runs, downtime tests) set the pace. Baseline QA calendar for the same path: ~14–16 months estimate.
Phases & activities
0 · Feasibility spike + go/no-go wk 0–8
- Replay one closed week of FI/MM/SD postings on a converted sandbox
- Measure the replayable share of document volume; go/no-go at ≥85%, otherwise fall back to Playbook B
- Seeded-defect evals on the harness
- Data-protection assessment (production data in test)
1 · Replay harness + diff platform wk 6–18
- Event extraction from production (documents, change docs, IDocs, job logs) in original order
- Replay via BAPI/IDoc/API with time-travel handling (posting dates, exchange rates, number ranges)
- Service virtualisation for outbound interfaces
- State-diff platform: DTV/LTVF/Kyano + HANA SQL, with KID normaliser
2 · Unicode step: full diff gate wk 12–18
- Content checksums before/after Unicode conversion; byte diffs on interface files
- Collation tests generated from real selection variants
3 · Shadow periods per mock wk 18–38
- Mock 1: replay a normal month-end; Mock 2: quarter-end; Mock 3: year-end close
- Period-end diffs: balances, open items, stock/valuation, MRP results, outputs
- AI triage of diffs; KID matching; root-cause hypotheses
4 · Residual tests, UAT, dress rehearsal wk 34–48
- Agentic residual tests: Fiori UX, authorisations, forms/output, live interfaces, performance
- UAT on approved differences and new UX
- Dress rehearsal with full replay + diff in downtime window; sign-off
Tools
- Event extraction and replay harness (ecociel asset; Basis Technologies’ Testimony is an earlier example of the record-and-replay idea)
- Service virtualisation for interfaces
- SAP DTV/LTVF, SNP Kyano Validate or Natuvion ARC for state diffs
- Agentic UI testing for residuals (ECT/Tosca, Testsigma, Sofy, Eggplant)
- SAP Cloud ALM for defects and evidence
Roles
- Replay architect (ecociel)
- ABAP/data engineers (2)
- Basis team for repeated period-start conversions
- Functional leads: KID approval and diff decisions
- Finance controllers: period-close comparison sign-off
- Independent QA / external auditor liaison
Prerequisites
- Production backups at period boundaries available for mock conversions
- Sufficient HANA capacity to convert and replay whole periods
- Legal clearance to use production data in isolated test landscapes
- Executive sponsor willing to replace scripted UAT with diff evidence
Risks & mitigations
- Low replayable share (dialog-heavy or interface-driven processes). Mitigation: feasibility gate; fall back to B for non-replayable parts.
- Sequence/timing effects (number ranges, availability checks, MRP) cause false diffs. Mitigation: replay-specific KIDs, ordered replay, tolerance rules.
- Infrastructure cost and time per replay. Mitigation: replay selected periods, not all; scale HANA temporarily.
- Production data exposure. Mitigation: isolated landscape, masking for anything leaving it, no external LLM on raw data.
- Organisational acceptance. Mitigation: show seeded-defect detection results to auditors early.
Effort build-up per bucket estimate
Baseline = the traditional reference program (4,820 PD, 6 validation cycles, see model). “Mature” = second or later deployment with a reusable engine and rule libraries.
| Bucket | Baseline | First | Mature | Reasoning |
|---|---|---|---|---|
| Replay harness + state-diff platform (new) | 350 | 150 | Replay a closed production period (postings via BAPI/IDoc/API, batch-job chains) on a converted copy of the period-start backup, with service virtualisation for interfaces; full state diff vs. real production period-end. First build 350 PD; productised ~150. | |
| Scoping & test design | 800 | 30 | 20 | No test design in the classic sense: choose periods (month-end, quarter/year-end), define exclusions and replay order. |
| Test data & environments | 600 | 72 | 50 | The test data is production itself (period-start backup converted in each mock). ~12 PD/cycle for masking, isolation and housekeeping. |
| Test execution | 1,200 | 260 | 110 | Replay oversight 10 PD/cycle (60). Residual targeted tests for what replay cannot see (UI/Fiori, authorisations, output/forms, live interfaces, performance): 200 PD first, 70 PD mature using agentic UI testing. |
| Data validation & reconciliation | 720 | 120 | 75 | Table- and report-level diffs (DTV/LTVF/Kyano + HANA SQL) of the replayed period-end state vs. real production; 60 PD setup + 10 PD per cycle. |
| Intended-differences (KID) management (new) | 120 | 70 | Same simplification-driven normalisation as Playbook B, plus replay-specific rules (number ranges, timestamps, exchange-rate dates). | |
| User acceptance testing | 600 | 80 | 40 | Acceptance of differences and new UX only. Requires the business and auditors to accept production-as-oracle evidence. |
| Defect / diff triage | 600 | 150 | 50 | First replays are noisy (timing, number ranges, sequence effects); mature classifiers and rule libraries collapse this. |
| Documentation, evidence & sign-off | 300 | 30 | 20 | Period-level reconciliation reports are the evidence. |
| Harness evals (seeded defects) (new) | 50 | 30 | Inject known defects into a copy and prove the diff catches them. | |
| Total | 4,820 | 1,262 | 615 | Factor: first 3.8× (−74%), mature 7.8× (−87%). Fixed part first/mature: 520/240 PD. |
Assumptions
- ≥ 85% of document volume replayable at API/BAPI/IDoc level.
- Harness build ~350 PD first time, ~150 PD reused.
- Mature residual testing uses agentic UI tooling (shared with B).
- Same 6 validation cycles; three of them carry period replays.
Confidence
Low-medium (±50%). The idea has precedent (record-and-replay, report diffs). Replaying whole periods through a converted S/4 system has not been publicly documented at this scale. The feasibility spike exists to replace this estimate with a measured replayable share.