Playbook B · AI-agentic differential testing
B · Equivalence Engine
Treat the conversion as a functional-equivalence problem. Agents turn real production documents into executable scenarios, run them against a copy of ECC and the converted S/4 system, and diff the outcomes (documents, postings, statuses, outputs) through a normaliser that knows the intended differences. Humans approve differences and sign off.
4.0×
first deployment vs 4,820 PD baseline estimate
6.5×
mature / productised estimate
6.3× / 9.2×
ceiling at very large scale (first / mature) estimate
~10 months
QA calendar for a two-step IBM i path estimate
When to offer it
- Brownfield/system conversion where “same as before” really is the requirement.
- Medium-to-large programs, or ecociel’s second+ customer (engine reuse).
- Customers open to AI with a strong data-protection setup.
Timeline
048121620242832364044
0 · Pilot + engine evals
1 · Engine build-out
2 · Unicode step: full diff gate
3 · Differential mock cycles
4 · Focused UAT + dress rehearsal
Calendar gains are smaller than effort gains: the technical steps (Unicode conversion, DMO runs, downtime tests) set the pace. Baseline QA calendar for the same path: ~14–16 months estimate.
Phases & activities
0 · Pilot + engine evals wk 0–6
- One end-to-end process (e.g. order-to-cash) on ECC copy vs S/4 sandbox
- Seeded-defect evals: inject ~30 known defects, target ≥95% detection, ≤5% false alarms
- Data-protection assessment and hosting decision for LLM use
1 · Engine build-out wk 4–14
- Scenario miner: STAD/ST03N, SCMON/SUSG, CDHDR/CDPOS, document flow → process variants → representative documents
- Generator: scenarios → executable tests (BAPI/API first, ECT/Tosca or agentic UI where needed)
- Differential runner + deterministic diff normaliser; KID rule library from the simplification list
- Evidence generator (per scenario: inputs, both outputs, diff, verdict, KID reference)
2 · Unicode step: full diff gate wk 10–16
- Full table checksums and DTV before/after Unicode conversion (expected: identical content)
- Byte-level diff of outbound interface files
- Collation-sensitive report and range tests generated automatically
3 · Differential mock cycles wk 16–36
- Run the full scenario catalogue per mock on both systems
- AI clusters diffs, matches KIDs, drafts root-cause hypotheses; humans decide
- Re-run evals each cycle to guard against engine drift
- Residual tests: Fiori UX, authorisations, forms/output, live interfaces
4 · Focused UAT + dress rehearsal wk 34–44
- UAT limited to approved differences, new UX and confidence walkthroughs
- Dress rehearsal: full differential run + DTV in downtime
- Evidence pack and sign-off by process owners and independent QA
Tools
- SAP Cloud ALM (backbone, defects, evidence)
- Usage/document extraction: STAD/ST03N, SCMON/SUSG, change documents, document flow
- Execution: BAPI/OData/IDoc harness; SAP ECT/Tosca or agentic UI (Testsigma, Sofy, Eggplant) for residual
- Diff: DTV/LTVF or Kyano Validate + custom HANA SQL normaliser
- LLM platform with Swiss/EU hosting (e.g. SAP AI Core / BTP or private deployment)
- ecociel KID register + seeded-defect eval kit
Roles
- Engine lead / AI engineer (ecociel)
- Data/ABAP engineer for extraction and replay
- Functional leads approve KIDs per module
- Process owners: acceptance and sign-off
- Independent QA: reviews eval results and evidence samples
- Key users: ~10–15 for focused UAT
Prerequisites
- Read access to production usage and document data (or a recent masked copy)
- Two parallel environments per cycle: ECC copy and converted S/4 copy from the same snapshot
- Agreed KID approval workflow
- Evals passed before any reduction of manual testing
Risks & mitigations
- Diff noise from intended differences swamps triage. Mitigation: invest in the KID normaliser early; measure noise ratio per cycle.
- Coverage gaps for rare/annual processes. Mitigation: mine 13+ months, force-include period-end variants.
- LLM error in generated tests. Mitigation: tests are deterministic artefacts, verified by evals; LLMs never judge pass/fail alone.
- Auditor scepticism. Mitigation: involve auditors in eval design; evidence per scenario.
- Data protection. Mitigation: masking before LLM exposure; Swiss/EU hosting.
Effort build-up per bucket estimate
Baseline = the traditional reference program (4,820 PD, 6 validation cycles, see model). “Mature” = second or later deployment with a reusable engine and rule libraries.
| Bucket | Baseline | First | Mature | Reasoning |
|---|---|---|---|---|
| Equivalence-engine setup (new) | 300 | 120 | Pipelines from usage/document data (STAD/ST03N, SCMON/SUSG, CDHDR/CDPOS, document flow), scenario miner, test generator, differential runner, diff normaliser, evidence generator. First build 300 PD; productised re-use ~120 PD per customer. | |
| Scoping & test design | 800 | 60 | 40 | Scenarios are mined from real production documents (one or more per process variant) and generated by agents. Humans only review the scenario catalogue and approve coverage. |
| Test data & environments | 600 | 90 | 60 | Same masked snapshot on both sides; refresh automated. 15 PD/cycle (mature 10). |
| Test execution | 1,200 | 148 | 108 | Differential runs (ECC copy vs converted S/4 sandbox) need ~8 PD oversight per cycle (48). Residual tests the engine cannot judge by diff (new Fiori UX, authorisations, forms/output, live interfaces): 100 PD (mature 60 with agentic UI testing). |
| Data validation & reconciliation | 720 | 112 | 90 | DTV/LTVF + engine state diffs; 40 PD setup + 12 PD per cycle. |
| Intended-differences (KID) management (new) | 120 | 90 | Simplification-list driven normalisation rules (e.g. BSEG/FAGLFLEXA→ACDOCA, KONV→PRCD_ELEMENTS, MKPF/MSEG→MATDOC, KNA1/LFA1→BP) plus business approval. Library reuse in mature state. | |
| User acceptance testing | 600 | 150 | 100 | Focused on intended differences, new UX and business confidence walkthroughs, not on re-proving unchanged behaviour. |
| Defect / diff triage | 600 | 150 | 80 | AI clusters diffs, maps them to KIDs, proposes root-cause hypotheses; humans decide. Early cycles are noisy, hence 150 first time. |
| Documentation, evidence & sign-off | 300 | 40 | 25 | Evidence packs generated from run logs and diffs; humans review and sign. |
| Engine evals (seeded defects) (new) | 50 | 30 | Mutation testing of the engine itself (see ecociel Evals concept): proves detection rate before anyone relies on it. | |
| Total | 4,820 | 1,220 | 743 | Factor: first 4.0× (−75%), mature 6.5× (−85%). Fixed part first/mature: 450/220 PD. |
Assumptions
- ≥ 85% of transaction volume is reachable through BAPI/API-level generation; the rest is residual.
- Engine build ~300 PD first time, ~120 PD reused.
- Diff triage is noisy in the first cycle and settles by cycle 3.
- Same 6 validation cycles as baseline.
Confidence
Medium (±40%). The building blocks exist (DTV/LTVF, AI test generation, agentic execution). The integrated engine is new and the factor depends on how well KID normalisation works. Run the pilot with evals before committing to a factor in a contract.