03 · Data

A record layout says how it was stored. A spec says what it means.

Strong data modeling comes first in the Standard, and that holds in a migration. Copybooks and tables are good evidence of the data's shape, but they often say little about its meaning, its owner or its sensitivity.

Flat files and VSAM · at Tidewater

One copybook becomes less than you'd expect.

Anchor Core's payment-due file is a fixed layout. The Standard already has a Scheduled Payment record type, so this translation shows the mapping's real work: most of it isn't about types.

PMTDUE copybookBefore

       01  PMT-DUE-REC.
           05  PMT-BANK-ID      PIC X(04).
           05  PMT-LOAN-NO      PIC X(12).
           05  PMT-LOAN-CLASS   PIC X(01).
           05  PMT-DUE-DATE     PIC 9(08).
           05  PMT-AMT-DUE      PIC S9(7)V99 COMP-3.
           05  PMT-STATUS       PIC X(01).
           05  PMT-REF-NO       PIC X(10).
           05  PMT-ACH-TRACE    REDEFINES PMT-REF-NO
                                PIC X(10).

domains/payments/scheduled-payment.mdAfter

# Scheduled Payment
Code: SPM
Domain: payments
Retention: seven years after the loan closes

## Fields
| Field       | Type                  | Required | Class        |
|-------------|-----------------------|----------|--------------|
| loan number | key of borrowing.loan | yes      | internal     |
| due date    | date                  | yes      | internal     |
| amount      | money                 | yes      | confidential |
| status      | status                | yes      | internal     |

Source field	Becomes	Why
`PMT-BANK-ID`	The tenant, not a field	The platform carries it, and row level security enforces it. A person confirms that each record belongs to one bank.
`PMT-LOAN-NO`	loan number	A business key into another domain, as DATA-028 requires
`PMT-LOAN-CLASS`	loan class, on Loan	A fact about the loan, copied here for the batch job's convenience
`PMT-DUE-DATE`	due date	`99999999` meant "deferred" on 1,204 records, so that's an open question, not a date
`PMT-AMT-DUE`	amount, `money`, confidential	The file was never protected as confidential, but the classification comes from what the data is
`PMT-STATUS`	status	`E R L C` become `expected`, `received`, `late`, `cancelled`
`PMT-REF-NO`, `PMT-ACH-TRACE`	Dropped here	One field with two meanings. Check reference and ACH trace are separate facts, and both belong on Payment

Db2 · at Tidewater

Relational doesn't mean already specified.

Db2 feels close to the target, so it gets under-read. TWDB.LOAN looked straightforward and held three surprises.

TWDB.LOANBefore

CREATE TABLE TWDB.LOAN (
  BANK_ID   CHAR(4)  NOT NULL,
  LOAN_NO   CHAR(12) NOT NULL,
  LOAN_STAT CHAR(1)  NOT NULL,  -- A W C X
  ...
);

SELECT LOAN_STAT, COUNT(*) FROM TWDB.LOAN GROUP BY LOAN_STAT;
-- A 18,402   W 611   C 40,117   X 288   H 41

Value profileStatus H

The comment lists four statuses, and production has five. The 41 H loans were set by hand during a dispute in 2003. A steward decides what they are now.

Stored procedureSP_CLOSE_LOAN

It refuses to close a loan with a balance, and nothing else says so. LON-001: A loan closes only when its reported balance is zero.

Sort orderEBCDIC vs ASCII

EBCDIC sorts letters before digits, and ASCII sorts digits first. Harborside's loan numbers mix both, so its list changes order unless a rule says otherwise. LON-002: Loan lists sort by loan number with letters before digits.

Dialects and traps

Check these on every layout.

Concern	z/OS	IBM i	What to specify
Record layouts	COBOL copybooks, VSAM, sequential datasets	DDS physical and logical files, sometimes program-described	Fields with types from the Standard's catalog
Multiple meanings	`REDEFINES`, multiple record types in one file	Format selectors, multi-format logical files	Separately named fields
Decimal storage	Packed (`COMP-3`), zoned	Packed, zoned	`decimal` or `money`; never floating point
Dates	Text, packed, Julian, two-digit years with windows	Same, plus `CYYMMDD`	`date`, with the conversion stated and tested
Special values	`LOW-VALUES`, `HIGH-VALUES`, spaces, all nines	Zeros, `LOVAL`, `HIVAL`	A stated meaning: often an absent value or a status
Time of change	Effective dates kept in the data	Same	One time axis, as DATA-023 requires; "as of" dates are ordinary fields

Data migration strategy · at Tidewater

Does the future system need the past?

That's the question for each record type, and the architects planning the migration answer it. If the answer is no, the record comes in as a transferred record, and its past stays in the archive. If it's yes, whatever that history depends on has to come too. Payment history can't be explained without the balances that go with it.

Record type	Future need	Decision	How it comes in
Scheduled Payment	Only what's still expected	Transferred record	Current state, with one `transferred` history entry
Loan	Current terms and status	Transferred record	Same; balance as last reported
Payment	Banks dispute the last 24 months	History kept	Loaded as Payment records, with `payment date` as an ordinary field
Balance as reported	Explains those payments	History kept, because Payment needs it	Loaded as records, for the same 24 months

Loaded history doesn't rewrite the record

A record's history entries always show when the new system recorded something. So past events come in as records, with their dates in ordinary fields, and they never pose as history entries from years ago. The balances are copies of what Anchor Core reported, and its ledger stays authoritative.

Conversion

Converted data goes in by the front door.

Existing records aren't exempt from DATA-025. Conversion is a process in the application that writes the data, and every row goes through the module, so validation, encryption and history all apply. Bulk SQL would be faster, but it would skip all three.

Decide what converts first. Tidewater has 59,459 loans, and only 18,402 are active. Closed loans stay in the readable archive, and the steward decides on the 41 with status H.

Each bank's load runs in shadow first, on real data. A production load that goes wrong is fixed forward, because the database is shared.

apps/payments/processes/convert-scheduled-payments.md

# Convert Scheduled Payments
Code: CSP
Application: payments

## Stages
1. Receive the extract (automated)
2. Parse and validate (sub-process, in chunks)
3. Review exceptions (human, skill: data stewardship)
4. Commit (sub-process, in chunks)
5. Reconcile (automated)

## Who can start it
The Payments data steward, one bank at a time.

## Volume
| Items              | Expected per run | Decrypts per item | At once |
|--------------------|------------------|-------------------|---------|
| scheduled payments | 240,000          | 1                 | 50      |

## Rules
- CSP-001: A bank's load matches the extract by record count and by total amount.
- CSP-002: A row that fails validation is never dropped or repaired silently.
- CSP-003: A loaded record starts no follow-up, notice or other work.
- CSP-004: No scheduled payment is loaded twice.

Rules

MIG-009 to MIG-013, MIG-043, MIG-045

MIG-009
Every source field maps to a typed, classified field or is recorded as dropped with a reason.
MIG-010
A source field holding more than one meaning becomes separately named fields, each with its own type and class.
MIG-011
Special values such as all nines, spaces, or zero dates are given stated meanings and never carried forward as magic values.
MIG-012
Every record type's tenant boundary is identified from the data and confirmed by a person, never assumed.
MIG-013
Converted data reaches a domain only through a validated commit step, and each load reconciles by record count and control total.

MIG-043
Which records convert is a recorded decision for each record type, and records that don't convert stay in a readable archive.

MIG-045
The data migration strategy records, for each record type, whether it comes in as a transferred record or keeps its history, based on whether the future system needs that history, and it preserves whatever that history depends on.