RAR Deployment Runbook

Drew deployed the full RAR codebase to staging on Jan 28, tweaked Lightning pages on Jan 29, then went idle. Three days of no changes. Here's where things stand.

What's Complete

What's Blocked or Missing

Record Counts — Full Inventory

ALR Resolution Sources

How each ALR junction was resolved. Top sources across all 176,042 records:

PL Brand Resolution Sources

How the brand (Account) was determined for each Physical Location:

P0 — Must Fix Before Production

Production deployment is impossible until every P0 is resolved. Salesforce will reject the deploy at 7% coverage.

P1 — Should Fix Before Production

These don't block deployment mechanics but will make production data incomplete or misleading.

P2 — Nice to Have

Remaining Failures (2 — Non-RAR)

Resolved Failures (9 RAR Tests — All Fixed)

These failures were identified on Feb 1 and resolved via production class fix (DomainMaintenanceScheduler) and test data corrections. Full test run on Feb 2 confirms all passing.

Current Numbers (Refreshed Feb 2, 2026)

Remaining Coverage Gaps (RAR Classes)

These 8 classes represent the remaining ~854 uncovered RAR lines. Improving DomainAdminController and PhysicalLocationMergeBatch alone would push RAR coverage past 75%.

How to Verify Coverage

-- Check refreshed coverage
SELECT ApexClassOrTrigger.Name, NumLinesCovered, NumLinesUncovered
FROM ApexCodeCoverageAggregate
WHERE ApexClassOrTrigger.Name LIKE '%PhysicalLocation%'
   OR ApexClassOrTrigger.Name LIKE '%AccountResolution%'
   OR ApexClassOrTrigger.Name LIKE '%Domain%'
ORDER BY NumLinesUncovered DESC

Maintaining Coverage Above 75%

Org-wide coverage now passes the deploy gate at 78.4%. To maintain this and improve RAR-specific coverage past 75%:

• Priority targets: Write tests for zero-coverage controller/batch classes listed above. DomainDuplicateScannerController (421 lines at 0%) alone would push RAR coverage past 75%.
• Deployment validation: Use --test-level RunLocalTests with --code-coverage to ensure aggregate stays fresh. Stale aggregates caused the original 0%/7% misreading.

Nuclear Option

If legacy tests can't be fixed in time: use @isTest(isParallel=false) annotations to prevent interference, or exclude legacy test classes from the deployment validation. The 2 failing ConvertLeadApexTest methods are outside RAR scope — coordinate with Kevin to fix or quarantine before production.

Staging has data. Production has nothing. This is a first-time deployment — no RAR objects, fields, classes, or triggers exist in production. The sequence below assumes all P0 items are resolved.

Pre-Deploy (T-14 to T-1)

• P0s resolved: Code coverage ≥75%, 0 failing tests, merge strategy decided, UAT/CAB signed
• Fresh Full Copy sandbox refresh (latest production data)
• Re-run full test suite on refreshed sandbox — confirm 0 failures
• Export production baseline record counts (Account, NSBA, NSS, Venue)
• Test destructiveChanges.xml rollback on sandbox — confirm clean removal
• Translate sandbox Merge_Target__c IDs to production IDs using Website_Root__c as join key
• Prepare Merge_Target CSV with production Account IDs
• Confirm Boomi sync pause window with Kevin
• Select deployment window (recommended: Friday 6 PM ET → Sunday noon)
• Final go/no-go call: Drew + Kevin + Nick

Phase 0: Prerequisites

Phase 1: Metadata Deployment

Phase 2: Flow Deactivation

Phase 3–8: Data Migration

Phase 9: Validation

Phase 10–11: Go-Live

Current State

Merge Readiness of the 17,144 Pending

Option A: Reparenting (Recommended)

Use LocationReparentingService. Both accounts survive. ALR junctions are moved from source to target. Source account gets Merge_Status__c = 'Consolidated Out'. Fully reversible — reparented ALRs are tagged with Resolution_Source__c = 'LocationReparentingService' and have Previous_Account__c for rollback.

• Run PREVIEW mode first → sample 50 accounts
• Execute in batches of 500
• Run AccountHierarchyBatchJob after
• Run UltimateParentBatchJob after

Option B: Database.merge() (NOT Recommended)

Permanently deletes the losing account. Irreversible. On 17K accounts with 94% having no confidence score, this is unacceptable risk. The only valid use case: 4 accounts flagged Auto-Merge with high confidence.

Suggested Approach

Tiered strategy based on available confidence data:

Integrity Checks (Staging Baselines)

-- 1. Orphan PLs (expect: 0 — all PLs have Account via Master-Detail)
SELECT COUNT() FROM Physical_Location__c WHERE Account__c = null

-- 2. Duplicate domains (expect: 0 rows)
SELECT Clean_Domain__c, COUNT(Id) cnt
FROM Domain__c GROUP BY Clean_Domain__c HAVING COUNT(Id) > 1

-- 3. Child accounts missing ultimate parent (expect: 0)
SELECT COUNT() FROM Account
WHERE Top_Level_Parent__c = null AND ParentId != null

-- 4. PLs without any ALR junction (expect: 0)
SELECT COUNT() FROM Physical_Location__c
WHERE Id NOT IN (SELECT Physical_Location__c FROM Account_Location_Relationship__c)

-- 5. Unlinked NSBAs (staging: 59 of 59,482)
SELECT COUNT() FROM NetSuite_Billing_Account__c WHERE Physical_Location__c = null

-- 6. Unlinked NS Subscriptions (staging: 76 of 83,486)
SELECT COUNT() FROM NetSuite_Subscription__c WHERE Physical_Location__c = null

Record Counts (Staging Baselines)

-- Total PLs (staging: 78,610)
SELECT COUNT() FROM Physical_Location__c

-- Total Domains (staging: 30,213)
SELECT COUNT() FROM Domain__c

-- ALR by type (staging: Brand 62,042 / CC 75,040 / Operator 38,960)
SELECT Relationship_Type__c, COUNT(Id) cnt
FROM Account_Location_Relationship__c GROUP BY Relationship_Type__c

-- Domain match status (staging: Matched 30,128 / Manual 83 / No Match 1)
SELECT Match_Status__c, COUNT(Id) cnt
FROM Domain__c GROUP BY Match_Status__c

-- Account merge state
SELECT Merge_Status__c, COUNT(Id) cnt
FROM Account GROUP BY Merge_Status__c

-- Brand validation (staging: Validated 65,395 / Pending 13,215)
SELECT Brand_Validation_Status__c, COUNT(Id) cnt
FROM Physical_Location__c GROUP BY Brand_Validation_Status__c

ARR Reconciliation

-- NS Subscription ARR by status (Active staging: $27,152,726.93)
SELECT Status__c, SUM(ARR__c) arr, COUNT(Id) cnt
FROM NetSuite_Subscription__c GROUP BY Status__c

-- PL ARR (staging: ALL NULL — this is the P1-1 gap)
SELECT COUNT() FROM Physical_Location__c WHERE ARR__c != null

-- PL Subscription ARR rollup (staging: 48,835 with ARR > 0)
SELECT COUNT() FROM Physical_Location__c WHERE Subscription_ARR__c > 0

-- Top 20 brands by ARR
SELECT Account__r.Name,
       SUM(Physical_Location__r.Subscription_ARR__c) totalARR,
       COUNT(Physical_Location__c) locations
FROM Account_Location_Relationship__c
WHERE Relationship_Type__c = 'Brand' AND Is_Active__c = true
GROUP BY Account__r.Name
ORDER BY SUM(Physical_Location__r.Subscription_ARR__c) DESC LIMIT 20

Linkage Health

-- NSBA → PL linkage (staging: 59,423 of 59,482 = 99.9%)
SELECT COUNT() FROM NetSuite_Billing_Account__c WHERE Physical_Location__c != null

-- NS Subscription → PL (staging: 83,410 of 83,486 = 99.9%)
SELECT COUNT() FROM NetSuite_Subscription__c WHERE Physical_Location__c != null

-- Venue → PL (staging: 101,929 of 118,473 = 86.0%)
SELECT COUNT() FROM Venue__c WHERE Physical_Location__c != null

-- NSBA → PL linkage rate (staging: 66.1%)
SELECT COUNT() FROM Physical_Location__c
WHERE Id IN (SELECT Physical_Location__c FROM NetSuite_Billing_Account__c)

-- Account_Hierarchy (staging: 0 — empty)
SELECT COUNT() FROM Account_Hierarchy__c

Code Coverage

-- RAR class coverage (expect: populated after full test run)
SELECT ApexClassOrTrigger.Name, NumLinesCovered, NumLinesUncovered
FROM ApexCodeCoverageAggregate
WHERE ApexClassOrTrigger.Name LIKE '%PhysicalLocation%'
   OR ApexClassOrTrigger.Name LIKE '%AccountResolution%'
   OR ApexClassOrTrigger.Name LIKE '%ALR%'
   OR ApexClassOrTrigger.Name LIKE '%Domain%'
ORDER BY NumLinesUncovered DESC

-- Org-wide aggregate (staging: 933 covered / 11,843 uncovered = 7%)
SELECT SUM(NumLinesCovered) covered, SUM(NumLinesUncovered) uncovered
FROM ApexCodeCoverageAggregate

Post-Go-Live Monitoring

-- Exceptions today
SELECT CreatedDate, Message__c FROM Custom_Exception__c
WHERE CreatedDate = TODAY ORDER BY CreatedDate DESC

-- New PLs created today (should see new ones after Boomi resumes)
SELECT COUNT() FROM Physical_Location__c WHERE CreatedDate = TODAY

-- New ALRs today
SELECT Relationship_Type__c, COUNT(Id) cnt
FROM Account_Location_Relationship__c WHERE CreatedDate = TODAY
GROUP BY Relationship_Type__c

-- Kill switch status
SELECT All_Automations__c, Triggers__c, Batch_Jobs__c,
       Scheduled_Jobs__c, Queueable_Jobs__c
FROM Automation_Settings__c

Two Automation_Settings__c records exist. Master (org-level) has everything ON. Drew's user-level override has most things OFF — this was intentional during data migration to prevent triggers from interfering with batch loads.

Master Record (Org Default) — All ON

All 17 individual toggles are also ON at master level.

User Record (Drew) — Mostly OFF

Before Production Go-Live

Drew's user-level overrides should be deleted or set to all ON. In production, the master record should control everything. No user-level overrides unless needed for a specific integration user (e.g., Boomi).

Emergency Kill (Anonymous Apex)

Automation_Settings__c s = Automation_Settings__c.getOrgDefaults();
s.All_Automations__c = false;
upsert s;

How Disabled Batches Behave

When a batch's kill switch is FALSE, its start() method returns Database.getQueryLocator('SELECT Id FROM Physical_Location__c WHERE Id = null') — an empty result set. The batch completes immediately with 0 records processed. No exceptions thrown.

Currently Running

Jobs That Should Be Scheduled (Not Yet)

Schedule Commands (for production go-live)

// Domain validation — daily 7:00 UTC
System.schedule('DomainValidation_Daily', '0 0 7 * * ?',
    new DomainValidationScheduler());

// Ultimate parent — daily 8:00 UTC
System.schedule('UltimateParent_Daily', '0 0 8 * * ?',
    new UltimateParentBatchJob());

// PL enrichment — nightly 9:00 UTC
System.schedule('PLEnrichment_Nightly', '0 0 9 * * ?',
    new PhysicalLocationEnrichmentBatch());

// Account hierarchy — weekly Sunday 10:00 UTC
System.schedule('AccountHierarchy_Weekly', '0 0 10 ? * SUN',
    new AccountHierarchyBatchJob());

Metadata Inventory

Emergency Kill Switch

Fastest possible response. Disables all RAR automation in under 60 seconds:

// Kill everything
Automation_Settings__c s = Automation_Settings__c.getOrgDefaults();
s.All_Automations__c = false;
upsert s;

// Or via CLI:
echo "Automation_Settings__c s = Automation_Settings__c.getOrgDefaults(); \
s.All_Automations__c = false; upsert s;" | sf apex run --target-org RBProduction

Per-Phase Rollback

Full Rollback (4–6 hours)

Execute in reverse order:

1. All_Automations__c = FALSE (master kill)
2. Reactivate Account flows (recorded versions)
3. Resume Boomi sync
4. Delete data: Physical_Location__c (cascades ALR), Domain__c, Account_Hierarchy__c
5. Null RAR fields: Website_Root__c, Brand_Parent__c, Top_Level_Parent__c, Merge_Target__c, Merged_Into__c, Merge_Status__c on Account. Physical_Location__c on NSBA, NSS, NSSL, Venue.
6. Deploy destructiveChanges.xml
7. Restore Permission Sets if modified
8. Verify org is in pre-deployment state

Data Recovery Sources

Escalation Path

1. Drew Lambert — code/data issues (first responder)
2. Kevin Grothe — org admin, flow/config, Boomi issues
3. Nick Bolton — business decision authority
4. Aron Kuehnemann — executive escalation

Notification Matrix

Post-Deployment Hypercare