Skip to main content

Data model and migrations

Data model​

A case type's entities are declared in model files, not Java: one file per entity at src/main/jcm/model/<Entity>.yml. The model has no enums (a ref field over a reference list takes their place), embedded parts, many-to-many references or inheritance in this release, and cannot drop a whole entity; an entity that needs one is a Jmix entity in Java with a changelog of its own, as the app's User is. These are build input, not classpath configuration, and no override folder or studio publish changes them. The library jcm-model (com.jcm:jcm-model, in the BOM; snakeyaml only, no Spring and no Jmix) reads and checks them, records the schema they need (schema.json, below) and works out what changes from one to the next. The com.jcm.app plugin's generateJcmModel writes the entities and captions from them at build time (below), its jcmChangelog writes the next changelog when the model changes, and its checkJcmModel (under check) holds the model, schema.json and the written changelogs together. A new app's starter case type is a model file: jcm new writes src/main/jcm/model/<Name>.yml (below), its schema.json and 020-jcm-create-<name>.xml from templates, byte for byte what jcmChangelog writes for that file, with jcm { appPackage } in build.gradle, and no entity Java or captions:

entity: Request # a name whose table is a reserved word adds table: <NAME>_FILE (Case: CASE_FILE)
instance-name: title
fields:
title: {type: text, length: 200, required: true}
receivedOn: {type: date}
urgent: {type: boolean, default: false}
urgencyReason: {type: text, length: 1000, label: Why it is urgent}
entity: Party # class and entity name; the file must be Party.yml; the table defaults to PARTY
instance-name: name # optional; must name one of the fields
parent: {name: appeal, entity: Appeal} # makes this a child: required, cascade on delete
fields:
name: {type: text, length: 100, required: true, label: Name}
kind: {type: ref, list: ref.party-kind}
notes: {type: long-text}

On the case entity, children: names its compositions (attribute: entity, kept in file order); each child's file names this entity as its parent:

entity: Appeal
instance-name: caseTitle
fields:
caseTitle: {type: text, length: 200, required: true, label: Case title}
receivedOn: {type: date}
summary: {type: long-text}
children:
parties: Party
hearings: Hearing
decisions: Decision

The file's keys are entity (required), table, instance-name, parent, fields, children, audit, removed, extends and unique.

TypeJava and columnNotes
textString, varchar(length)length defaults to 255, at most 4000 (use long-text above); index: true up to length 400
long-text@Lob String, clob
dateLocalDate, date
date-timeLocalDateTime, datetime
booleanBoolean, booleanrequires a default:; the column is not null
integerInteger, int
decimalBigDecimal, decimal(precision, scale)defaults to 19, 2; effective precision 1–38, scale 0–precision, checked after defaults
refRefItem, @RefListCode(list)a reference-data lookup; list is required, and every save holds the field to that list's items (Reference data)
userthe app's User
referenceanother model entity named by entity:
documentUUID, @DocumentRef(types)one of the record's documents at one revision; optional types: [CODE, ...]; needs jcm-attachments (Attachments, Document revision fields)
  • Every field takes type (required), required, label (defaults to the name as words: receivedOn is "Received on"), column (for a reserved word, such as value), index: true, changed: true and audit: false (keep no history of the field's changes; see Field history; audit: true is refused, being the default). Only text takes length, only decimal precision and scale, only boolean default, only ref list, only reference entity and only document types.
  • Names follow the demo's changelogs: UPPER_SNAKE tables and columns (InboundDocument is INBOUND_DOCUMENT, receivedOn is RECEIVED_ON); a ref, user, reference or document field's column ends in _ID (kind is KIND_ID), and so does the parent's (APPEAL_ID); IDX_<table>_ON_<attribute>, FK_<table>_ON_<attribute>, PK_<table> (IDX_PARTY_ON_APPEAL, named after the attribute, not the column), and UK_<table>_ON_<attribute>[_<attribute>] for a unique: entry (UK_MEMBER_RECORD_ON_MEMBER_ID_SOURCE_SYSTEM).
  • References, ref and user fields and parents always get an index and a foreign key; index: true on one is allowed and changes nothing, index: false is refused. A document field always gets an index and never a foreign key: a document's row is not a model table.
  • Every entity has a UUID id and an Integer version; audit: true adds createdBy, createdDate, lastModifiedBy and lastModifiedDate. audit on the entity is about those stamps only; field history is on for every field regardless, and a field leaves it with audit: false on the field itself.
  • unique: is a list of attribute lists, unique: [[memberId], [memberId, sourceSystem]]: the values of each entry's attributes together are on one row only, held by a unique index over their columns in the entry's order. An entry names one to three fields of type text, integer, date, ref or reference, each required: true (a key that may be empty is no key, and SQL Server's unique index holds one null only). The entry's width is at most 1,700 bytes, SQL Server's limit for an index key: a text counts its length (varchar(n) holds n bytes under the UTF-8 collation), an integer 4, a date 3, a ref or reference 16. The generated entity carries @Index(name = "UK_...", columnList = "MEMBER_ID, SOURCE_SYSTEM", unique = true). PostgreSQL and HSQLDB compare the key exactly and SQL Server by its collation (ignoring case), so AB1 and ab1 are two keys to the first two and one to the third; adding an entry over rows that exist counts them ignoring case on every database (below).
  • A removed field is listed under removed: [status]; a length cut, another decimal precision or scale, or a field becoming required carries changed: true on the field. Those are the only type changes made (text to text, decimal to decimal); any other is refused even when marked (below).

Every problem in every file is reported at once, each as <file>:<line>: and a sentence, sorted by file and line. A file that cannot be read (not YAML, no entity:, an entity name that is not a name) is reported in itself only: the other files are not told that its entity is missing. These are refused:

  • Appeal.yml:4: unknown key 'colour'; the keys are entity, table, instance-name, parent, fields, children, audit, removed, extends, unique
  • Appeal.yml:3: field 'caseTitle' has the unknown key 'size'; the keys are type, length, precision, scale, required, label, column, index, list, entity, default, changed, audit, types
  • Appeal.yml:3: field 'caseTitle' has audit: true, which is the default; write audit: false to keep no history of the field, or leave the key out
  • Appeal.yml:4: the key 'caseTitle' appears twice (any key written twice)
  • Appeal.yml:1: has no entity:; give it entity: Appeal, and Party.yml:1: is not a model file; it holds entity: and fields: as a mapping; a file that is not YAML: <file>:<line>: is not valid YAML: <what the reader found>
  • Apeal.yml:1: holds entity Appeal, so it must be named Appeal.yml
  • Party.yml:6: field 'kind' has the unknown type 'lookup'; the types are text, long-text, date, date-time, boolean, integer, decimal, ref, user, reference, document (has no type: when it has none)
  • Case.yml:5: field 'title' is text and does not take 'list'
  • Case.yml:5: field 'urgent' is boolean and needs default: true or default: false
  • Review.yml:4: field 'report' has types that are not a list of document type codes, as types: [REPORT] (a single word, an empty list, a mapping), field 'report' names the document type 'REPORT' twice, field 'report' is document and always has an index; drop index: false, a required one on an entity with no parent: (Attachments, Document revision fields), and on an entity with extends:, UserProfile.yml:4: field 'badge' is document, which names one of a case's documents; UserProfile extends user and belongs to no case
  • Party.yml:6: field 'kind' is ref and always has an index; drop index: false
  • Note.yml:3: field 'body' is long-text, which SQL Server cannot index; drop index: true (Liquibase's clob is varchar(max) there, which no index key can hold)
  • Note.yml:4: field 'more' has length 4001; text holds at most 4000 characters: use long-text for a new field; for a field that exists, cut it to 4000 with changed: true, or move it to long-text in stages (add a long-text field, copy the values, remove this one) (above 8000 Liquibase makes the column varchar(max) on SQL Server)
  • Note.yml:5: field 'title' has length 401 and index: true; SQL Server indexes text up to 400 characters: shorten it to 400 (with changed: true for a field that exists), or drop index: true from a new field (SQL Server's nonclustered index key holds 1,700 bytes; 400 characters of four UTF-8 bytes stay under it)

Upgrading an app whose model files predate these limits: checkJcmModel now refuses a text longer than 4000 and an indexed text longer than 400. For a field that already exists, cut it with changed: true (length: 4000, or length: 400 for the indexed one), a guarded narrowing that fails, leaving every row as it was, while a value is longer; or, for text that holds longer values, move it in stages: add a long-text field, copy the values, remove the old field. Changing it to long-text in place and dropping its index are both refused.

  • Party.yml:6: field 'kind' is ref and needs list: with the reference list's code, and field 'previous' is reference and needs entity: with the entity's name
  • A value of the wrong kind: Case.yml:5: field 'title' has length 'long'; it takes a whole number above 0, in plain digits (numbers are decimal digits only: 0x10, 1_000, +5 and 1.5 are refused, 012 is 12), field 'title' has required 'maybe'; it takes true or false, audit is 'often'; it takes true or false, field 'amount' has scale 6 above its precision 4; a YAML null or an empty value where a word belongs: Appeal.yml:4: field 'caseTitle' has label '~'; it takes a word or phrase (and instance-name is '~', ...)
  • Names Java cannot use: field 'case-title' is not a name a field can have; use letters and digits, starting with a lower-case letter (and the same for a child, a parent's name and an entity, which starts upper-case), field 'class' is a word Java keeps; name the field otherwise
  • Case.yml:5: 'version' is every entity's own attribute; name the field otherwise (and id); with audit: true, 'createdBy' is an attribute audit: true adds; name the field otherwise (likewise createdDate, lastModifiedBy, lastModifiedDate). A parent or a child named so is refused the same way, ending name the parent otherwise or name the child otherwise
  • Hearing.yml:2: parent has the unknown key 'kind'; the keys are name, entity, and parent needs name: and entity:, as {name: appeal, entity: Appeal}
  • Appeal.yml:8: child 'parties' has the name of field 'parties'; name one of them otherwise (a field, the parent and the children share one set of names)
  • Appeal.yml:2: instance-name 'title' is no field of Appeal; the fields are caseTitle, receivedOn
  • Hearing.yml:7: field 'venue' names the entity 'Room', which no model file declares (and the same for a child or a parent: child 'parties' names the entity 'Party', ..., parent names the entity 'Appeal', ...)
  • Appeal.yml:9: child 'parties' is Party, whose parent is not Appeal; give Party.yml parent: {name: appeal, entity: Appeal}
  • Appeal.yml:9: child 'others' is Party, which child 'parties' already holds
  • Party.yml:3: Party names Appeal as its parent, but Appeal.yml lists it under no children
  • Node.yml:2: Node cannot be its own parent; for a tree of one entity, give it a field of type: reference, entity: Node (its parent column is not null, so no first row could exist, and SQL Server refuses a key that cascades onto its own table)
  • Party.yml:7: field 'kindId' has the column KIND_ID, which field 'kind' already has (and which parent 'appeal' already has, which every entity has for ID and VERSION, which audit: true adds); two entities with one table: entity Foo has the table PARTY, which Party.yml already has
  • Party.yml:6: field 'value' would have the column VALUE, a reserved word in SQL; give it a column of its own with column:; a column: that is itself reserved: field 'order' has the column POSITION, a reserved word in SQL; choose another; Order.yml:1: entity Order would have the table ORDER, a reserved word in SQL; give it a table of its own with table: (table ORDER is a reserved word in SQL; choose another for a table:). The reserved words are those of HSQLDB, PostgreSQL and SQL Server together, so a model that passes runs on all three.
  • A column: or table: that is not UPPER_SNAKE: field 'x' has the column 'amount', which is not a name a column can have; use upper-case letters, digits and _, starting with a letter
  • A generated name above 63 characters, which PostgreSQL would cut: Party.yml:5: field '<f>' makes the index name IDX_PARTY_ON_<F>, 64 characters; PostgreSQL keeps 63. Shorten the field's name or give the entity a shorter table:; for a column, ... makes the column name <C>, 64 characters; PostgreSQL keeps 63. Shorten the field's name or give it a shorter column:; for a foreign key, makes the foreign key name; for the table or primary key, entity <E> makes the primary key name PK_<T>, 65 characters; PostgreSQL keeps 63. Give it a shorter table:
  • Hearing.yml:9: 'room' is under removed: and under fields:; keep one
  • A unique: that is not a list of lists of names: MemberRecord.yml:9: unique: holds a list of attribute lists, as [[memberId], [memberId, sourceSystem]]
  • MemberRecord.yml:9: the unique entry [colour] names 'colour', which is no field of MemberRecord; the fields are memberId, sourceSystem, lastName (the parent, a child and the audit attributes are no fields)
  • MemberRecord.yml:9: the unique entry [balance] names 'balance', which is decimal; a unique attribute is text, integer, date, ref or reference
  • MemberRecord.yml:9: the unique entry [lastName] names 'lastName', which is not required; a key that may be empty is no key: give the field required: true (with changed: true for a field that exists)
  • MemberRecord.yml:9: the unique entry [memberId, sourceSystem, bornOn, badge] names 4 attributes; an entry takes one to 3, a unique entry names one to 3 attributes; this one names none for [], and the unique entry [badge, badge] names 'badge' twice
  • MemberRecord.yml:9: the unique entry [memberId] is written twice; keep one (the same attributes in another order are another entry, with an index of its own)
  • MemberRecord.yml:9: the unique entry [wide, badge, employer] is 1710 bytes wide (wide 1690, badge 4, employer 16); SQL Server holds an index key of at most 1700 bytes: shorten a text attribute or take one out of the entry
  • MemberRecord.yml:9: the unique entry [memberId, sourceSystem] names 'sourceSystem', which is under removed:; take it out of the entry, or drop the entry
  • An entry whose index name is above 63 characters: the unique entry [...] makes the index name UK_..., 86 characters; PostgreSQL keeps 63. Shorten the attributes' names or give the entity a shorter table:

Generated entities and captions​

The com.jcm.app plugin takes the app's package in build.gradle:

jcm {
appPackage = 'com.acme' // required once src/main/jcm/model holds a model file
// modelFolder = file('src/main/jcm/model') the default
// userTable = 'USER_' the default; the table a user field's key points at
// changelogFolder = file('src/main/resources/com/acme/liquibase/changelog') the default
}

generateJcmModel reads the model files and writes, under build/generated/jcm/:

  • java/<app package>/entity/<Entity>.java: one Jmix entity per file, in <appPackage>.entity under the file's name, so code and tests that use it by type compile as they would against a hand-written class. Its first line says which model file it comes from. Annotations follow the demo's hand-written entities: UUID id and Integer version, the fields in file order (@NotNull and nullable = false for a required field and every boolean, @Lob for long-text, a UUID with @DocumentRef, its types and @PropertyDatatype("jcm_documentRef") for document, @RefListCode with a lazy @ManyToOne to RefItem for ref, a lazy @ManyToOne to the app's User class, in the same package, for user, and to the entity for reference; a boolean starts at its default), the audit attributes with Spring Data's @CreatedBy, @CreatedDate, @LastModifiedBy and @LastModifiedDate, each child as @Composition @OneToMany(mappedBy = <the child's parent name>), and the parent as @OnDeleteInverse(DeletePolicy.CASCADE) with a required lazy @ManyToOne; the @Table lists the parent's index and then each indexed field's, named as above; and one getter and setter per attribute.
  • resources/jcm-model/messages.properties: the captions, <app package>.entity/<Entity>=<the name as words>, .id=ID, .version=Version, each field's label, each child and the parent as their names in words, and the audit attributes as Created by, Created date, Last modified by and Last modified date. jcm-core adds this bundle last, so a line in the app's own messages.properties still overrides a caption.
  • resources/jcm/model/<Entity>.yml, a copy of each model file, and resources/jcm-model/schema.json, a copy of the snapshot when there is one: the model as the app reads it at run time, fixed when it is built (Where files are read from) and held against the entities at startup (What stops startup, The entity model).

With the java plugin applied, the two folders are source and resource folders of main, so compileJava (and Jmix's entity enhancing) and processResources run generateJcmModel first, and the IDE sees the classes once it has synced the Gradle project. These files are never edited and never committed: change the model file and build again. A model file removed takes its class away on the next build. With no model file the task writes nothing.

The task fails, before writing anything, with:

  • set jcm { appPackage = '...' } in build.gradle: the entities of src/main/jcm/model go into <appPackage>.entity
  • the model's own refusals above, each with its file and line, under The model files in src/main/jcm/model are refused:
  • Party is declared in src/main/jcm/model/Party.yml and written in src/main/java/com/acme/entity/Party.java; delete the Java class, for a hand-written class in <appPackage>.entity with a model entity's name (the slip when an entity moves to a model file), instead of the compiler's duplicate-class error.
  • Hearing.yml:7: field 'officer' is user, so the generated class takes the app's User from com.acme.entity; no source folder of main has com/acme/entity/User.java, when a model has a user field and the app's User is elsewhere.
  • Index.yml:1: entity Index has the name of a type the generated classes use (jakarta.persistence.Index); name the entity otherwise, for an entity named like a type the classes import or name: User (the app's), RefItem, RefListCode, DocumentRef, PropertyDatatype, the Jmix and JPA annotations and enums they use (Entity, Table, Column, Index, Id, Version, Lob, JoinColumn, ManyToOne, OneToMany, FetchType, NotNull, JmixEntity, JmixGeneratedValue, InstanceName, Composition, OnDeleteInverse, DeletePolicy, CreatedBy, CreatedDate, LastModifiedBy, LastModifiedDate), List, UUID, BigDecimal, LocalDate, LocalDateTime, String, Integer, Boolean and Object.

The schema snapshot and its changes​

src/main/jcm/model/schema.json records the schema the app's changelogs make for its model, so the next change can be worked out from it: one entry per entity (in entity order) with its table, primary key, columns (name, attribute, Liquibase type, nullable, default), indexes and foreign keys, and the changelogs jcmChangelog wrote with the SHA-256 of each. It is written with two-space indents and keys in a fixed order, so the same model always gives the same bytes; it is committed, never edited by hand. A file that is not such a snapshot is refused with its line: schema.json:5: unknown key 'colour'; the keys are format, tables, changelogs, schema.json:1: has no 'changelogs', schema.json:4: the key 'tables' appears twice, schema.json:1: has format 2; this JCM reads format 1, so a newer JCM wrote it, a value of the wrong kind (column OFFICER_ID of CASE_RECORD has nullable 'true'; it takes true or false), a type it does not hold (column VERSION of CASE_RECORD has the type 'bigint'; schema.json holds the types ${uuid.type}, int, varchar(<length>), clob, date, datetime, boolean, decimal(<precision>, <scale>)), a checksum that is not 64 lower-case hex digits, a column, entity or table listed twice, a boolean column without "true" or "false" as its default, and an index or key on a column the table does not have.

An index on one column is {"name": ..., "column": ...}, with "unique": true after them when it is unique (an extension's owner, a unique: entry on one attribute). The index of a unique: entry on several attributes has "columns": ["MEMBER_ID", "SOURCE_SYSTEM"] in place of column, in the entry's order; a columns that is not a list of two or more of the table's columns, each once, is refused. A schema.json written before unique: existed holds one-column indexes only: it is read as it is, and written back byte for byte.

Columns are ID ${uuid.type} and VERSION int (both not null), the fields in file order, the audit columns (CREATED_BY varchar(255), CREATED_DATE datetime, LAST_MODIFIED_BY varchar(255), LAST_MODIFIED_DATE datetime, all nullable) and last the parent column (not null). A field is nullable unless it is required: true; a boolean is never null and has its default. A ref field's key goes to JCM_REF_ITEM, a user field's to the user table (USER_), a reference's to its entity's table, and a parent's to the parent's table with cascade on delete.

The model is compared with the snapshot entity by entity and field by field:

  • Made without a mark: a new entity's table (a table after the new tables it references), a new field that is not required (or a boolean, which has its default) with its index and key, a longer text, a field no longer required, index: true added to a field, and a unique: entry added or taken out. An entry added to an entity whose table exists is written as a change that can fail on the rows (below); one on a field that is new on such a table is refused as the new required field is. An entry whose attributes change is the old entry taken out and the new one added.
  • A wider decimal (at least the old scale and at least the old integer digits) still needs changed: true, as every decimal change does, but holds every value, so it is written with the additive changes, not as one that can fail.
  • Made only when the field carries changed: true: a shorter text, another decimal precision or scale, and an existing field becoming required (once every row has a value). Only text-to-text and decimal-to-decimal type changes are supported. Cross-family conversions, including text to long-text or integer, are refused before either changelog or snapshot is written, even with the mark. Use a staged conversion: add a new nullable field, explicitly convert and verify its values with application-specific code or SQL, switch consumers, then remove the old field. Changing existing rows cannot repair an unsupported conversion.
  • A new required field without a supported default is refused even with changed: true: add it nullable, backfill every row, then make it required with the mark. A supported default (currently boolean) permits direct addition.
  • Made only when the field is listed under removed:: dropping its column, with its foreign key and index (the four audit attributes too, once audit: true is gone).
  • A mark the snapshot already agrees with (a changed: true on a field that no longer changes, or a removed: entry whose column is gone) changes nothing in that run. Remove it once jcmChangelog has written its changelog: checkJcmModel warns about each one, naming the field, because a changed: true left on a field would let a later cut or type change on it through without a fresh decision. Field order is no change.

Everything else is refused, every refusal at once, and when any is refused no change is made:

  • Hearing.room is in schema.json but not in Hearing.yml; list it under removed: [room] to drop its column, or put the field back
  • Party.name changes from text(100) to long-text; this conversion is not supported on every database; use a staged conversion: add a new nullable field, explicitly convert and verify its values, then switch to it and remove the old field
  • Party.name is cut from 100 to 50 characters; mark the field changed: true to alter its column
  • Appeal.receivedOn becomes required; mark the field changed: true once every row has a value
  • Appeal.priority is new and required, but the APPEAL table already exists and its rows have no value; add it without required: true, fill it, then make it required with changed: true
  • Party.kind changes from ref to text; a field that joins another table cannot change type: remove it and add a new field (also when marked; and for ref to user, or a reference to another entity)
  • Appeal.urgent changes from boolean to text(255); a boolean field cannot change type: remove it and add a new field
  • Appeal.urgent changes its default from false to true; jcmChangelog does not change a default in this release: put default: false back
  • Hearing.room loses its index; jcmChangelog does not drop an index in this release: put index: true back
  • Hearing.room changes its column from ROOM to HEARING_ROOM; a field keeps its column: put column: ROOM back, or remove the field and add a new one
  • Hearing.venue has the column ROOM, which schema.json gives Hearing.room; a column keeps its field: give venue a column of its own, or keep the name room (a field renamed, or a new field on a dropped field's column)
  • Party's table changes from PARTY to CASE_PARTY; jcmChangelog does not rename a table in this release: put table: PARTY back
  • Hearing's parent changes from appeal (Appeal) to none; jcmChangelog does not change an entity's parent in this release: put the parent back as it was (and from none, or to another entity)
  • Party has the audit columns in schema.json, but Party.yml has no audit: true; put audit: true back, or list createdBy, createdDate, lastModifiedBy and lastModifiedDate under removed: to drop their columns (turning audit on adds the four columns). On an entity that never had audit: true, fields of those names are plain fields and are dropped like any other.
  • Decision is in schema.json but no model file declares it; jcmChangelog does not drop tables in this release: keep Decision.yml
  • Alpha and Beta are new and reference each other, so neither table can be created first; add them without one of those references, run jcmChangelog, or jcm changelog, then add it

The changes are named in a few words for the changelog's file: create-hearing-and-party, add-hearing-notes, add-hearing-fields-and-party-email, alter-party-name, drop-hearing-status; by kind (create, add, alter, drop) and entity, several fields of one entity as <entity>-fields, and more than three parts as the first three and -and-more.

Changelogs: jcmChangelog, --baseline and checkJcmModel​

./gradlew jcmChangelog (jcm changelog with only Docker) compares the model files with schema.json (with no schema.json, with an empty schema, so every table is created) and writes:

  • the next changelog, <changelogFolder>/<NNN>-jcm-<summary>.xml. NNN is the highest three-digit number that starts a file name in the folder, plus 10 (010 in an empty folder; 050 after the demo's 010-040); a name without one is ignored, and past 999 the task stops. <summary> is the changes in a few words, as above. The folder is jcm { changelogFolder }, by default src/main/resources/<appPackage as path>/liquibase/changelog, which the app's includeAll runs in name order; it must lie inside a resource folder of main, and the file's logicalFilePath is its path there (com/acme/liquibase/changelog/020-jcm-create-case.xml).
  • the new schema.json, the model's snapshot with the file and its checksum added to changelogs: the SHA-256 of the file's UTF-8 text with every CRLF and CR turned into LF, so a checkout that rewrites line ends (Git's core.autocrlf on Windows) holds the same file, as Liquibase's own checksum does. A script that writes these files (the scaffold) hashes the same way.

The changelog is written under <name>.tmp (not a changelog to includeAll), then schema.json, and only then does the file take its name, so a failure never leaves a written changelog that schema.json does not list.

It logs jcmChangelog wrote <changelog> and src/main/jcm/model/schema.json. With nothing to change it writes nothing and says The model and schema.json agree; no changelog written. Fields moved to another order are no change to the database: schema.json is rewritten in the model's order and no changelog is written (The model and schema.json agree but for the order of fields; schema.json rewritten, no changelog written). A refused change fails the task with every sentence above under jcmChangelog refuses the model in src/main/jcm/model:, and nothing is written. After it has written, it warns about each mark the model no longer needs (below): remove them then.

The changelog is laid out as the demo's hand-written ones, author jcm, changeSet ids from 1:

  • one changeSet per new table: createTable with PK_<TABLE>, a not-null column's nullable="false", a boolean's defaultValueBoolean, then the parent's index and key (onDelete="CASCADE"), then each indexed field's index and, for a field that joins another table, its key;
  • one per added field: addColumn, with its index and key;
  • one per added index, and for a field whose column changes (a longer text, a wider decimal, no longer required): the column's dropForeignKeyConstraint and dropIndex in a changeSet of their own when it has them, the modifyDataType (followed by addNotNullConstraint for a required column) or dropNotNullConstraint, and the index and key created again in a changeSet of their own. SQL Server's ALTER COLUMN fails on a column an index or a foreign key holds and leaves the column nullable, so the same file does this on every database;
  • one per removed field: dropForeignKeyConstraint, dropIndex, dropColumn, as the demo's 040 does;
  • a unique: entry on a new table is a createIndex unique="true" with its columns, after the table's other indexes, in the table's changeSet; an entry taken out is one dropIndex, in a changeSet of its own before every other change to tables that exist (so before the column of a removed field it named). A key column that is altered (made longer, or narrowed with the mark) has the indexes of the entries that name it dropped and created again with its own index, in the same changeSets;
  • last, the changes marked changed: true that can fail on the rows a database holds (a shorter text, a decimal with fewer fractional or integer digits, a field becoming required), each in changeSets of their own (with the index and key inside them as below), under a comment saying so.
  • a narrowing (a shorter text, a decimal losing digits) is guarded against data loss on PostgreSQL, SQL Server and HSQLDB only (below), so its modifyDataType is written for dbms="postgresql,mssql" and, through a guard domain, for dbms="hsqldb", never for a negated list; a changeSet before the column's first one, for dbms="!postgresql,!mssql,!hsqldb", halts the deploy on any other database with a precondition (onFail="HALT"): NOTE.AMOUNT changes from decimal(12, 4) to decimal(12, 2), a change JCM makes only where it is guarded against losing data. JCM supports PostgreSQL, SQL Server and HSQLDB; run the application on one of those. Changelogs written before this (none of the demo's, the sample's or the scaffold's narrows) stay as they are.
  • a narrowing, or a field made required whose column has an index or a foreign key (a ref, user or reference field always has both): on PostgreSQL and SQL Server (dbms="postgresql,mssql") SQL Server's validating SELECT (for a narrowing), the dropForeignKeyConstraint and dropIndex, the alteration and the index and key created again are one changeSet, which rolls back whole when a row cannot take the change, so the index and key are never missing; HSQLDB commits each DDL statement, so there (dbms="hsqldb") the column is altered with its key and index in place, which HSQLDB allows, and a failure changes nothing; any other database halts as above (NOTE.OWNER_ID becomes required, which JCM makes only where a failure keeps its foreign key. ..., ... keeps its index. for a plain indexed field). A field made required with no index or key is one addNotNullConstraint on every database.
  • last of all, each unique: entry added to a table that exists, under the comment A unique: entry in the model: fails while rows hold an empty or repeated key. Fix those rows and run the changelogs again; never edit this file. The rows that would break the index are counted before it is created: a row with a key column null, and rows with the same key, text compared in lower case on every database (AB1 and ab1 are one key). When any is found the migration stops with MEMBER_RECORD holds rows whose MEMBER_ID is empty or not unique ignoring case; find them with: SELECT LOWER(MEMBER_ID), COUNT(*) FROM MEMBER_RECORD GROUP BY LOWER(MEMBER_ID) HAVING COUNT(*) > 1 (for several attributes whose MEMBER_ID and SOURCE_SYSTEM are empty or not unique together ignoring case, with every column in the query; ignoring case and LOWER() only where an attribute is text), no index is made, and the same changelog finishes once the rows are fixed. On PostgreSQL and SQL Server (dbms="postgresql,mssql") the count and the createIndex are one changeSet under a table lock (LOCK TABLE <table> IN SHARE MODE; WITH (TABLOCKX, HOLDLOCK)), so no row arrives between them. HSQLDB commits each DDL statement and a failed CREATE UNIQUE INDEX changes nothing there, so its changeSet (dbms="hsqldb") counts in a sqlCheck precondition (onFail="HALT") and creates the index alone. Any other database halts at a changeSet before both: MEMBER_RECORD gains the unique index UK_MEMBER_RECORD_ON_MEMBER_ID, which JCM creates only where it can first check the rows for an empty or repeated key. JCM supports PostgreSQL, SQL Server and HSQLDB; run the application on one of those. A key attribute made required in the same change (required: true, changed: true) is altered by the step above first, so rows with no value stop the migration there.
  • a type change the writer does not know (anything but text to text or decimal to decimal, which SchemaDiff never asks for) stops jcmChangelog rather than being written unguarded.

A file jcmChangelog wrote is never edited: Liquibase refuses a changeSet that has run and has changed, and checkJcmModel refuses it before then. A marked narrowing never silently truncates text or rounds decimals. Decimal widening requires both at least the old scale and at least the old integer capacity (precision minus scale); all other decimal changes validate exact representability first. If a marked change fails on the rows (a value too long, fractional digits or integer range the new decimal cannot hold, an empty value in a field made required), every changeSet before it has run and is recorded; explicitly correct those rows (shorten, round or fill them, in the database or with a changelog of the app's own numbered before it) and start the app again: Liquibase runs the rest of the file. Once a database has run part of a file, the rest must run there. Narrowing protection also covers concurrent writes: PostgreSQL retains a validating CHECK constraint until after ALTER; HSQLDB alters through a temporary target domain whose constraints validate original values inside ALTER; SQL Server uses a top-level validating SELECT to start its JDBC-managed implicit transaction and holds an exclusive table lock through ALTER. An IF EXISTS subquery alone does not start that transaction and cannot hold the lock between validation and ALTER, even with JDBC autocommit off. Temporary checks/domains have separate recorded creation and cleanup steps, so retry needs no manual cleanup. Do not remove these guards or edit a partially applied changelog. While no database has run it, the change can be given up: restore the model files and schema.json from version control and delete the file.

./gradlew jcmChangelog --baseline (jcm changelog --baseline) records the model as the first schema.json and writes no changelog, for an app whose own changelogs already create its tables (the demo's 010-040): the model must then describe those tables exactly, which the run-time schema check proves. Once schema.json exists it is refused: schema.json already exists; --baseline records a first snapshot only. Run jcmChangelog without it. It is refused too while the changelog folder holds a file jcmChangelog wrote (below): recording the model then would drop those files' checksums.

checkJcmModel, which check runs, fails with:

  • src/main/jcm/model has no schema.json; run ./gradlew jcmChangelog, or jcmChangelog --baseline when the app's own changelogs already create its tables (jcm changelog, or jcm changelog --baseline, with only Docker), for model files without a snapshot;
  • src/main/jcm/model and schema.json differ: Hearing.notes is new; run ./gradlew jcmChangelog, or jcm changelog whenever jcmChangelog would write something, each difference named (Note is new, Hearing.room changes type, Hearing.room becomes required, Hearing.room gains an index, Hearing.officer is listed under removed:, MemberRecord gains the unique index UK_MEMBER_RECORD_ON_MEMBER_ID, MemberRecord loses the unique index ..., Party.name is no longer required, Hearing's fields are in another order);
  • src/main/jcm/model and schema.json differ, and jcmChangelog refuses the difference: and every refusal;
  • 050-jcm-add-hearing-notes.xml was written by jcmChangelog and has changed since; a changelog that has run must not change: put it back and change the model instead, when a listed file's SHA-256 differs;
  • 050-jcm-add-hearing-notes.xml is listed in schema.json but is not in src/main/resources/com/jcm/demo/liquibase/changelog;
  • 050-jcm-add-hearing-notes.xml in src/main/resources/com/jcm/demo/liquibase/changelog was written by jcmChangelog, but schema.json does not list it, so schema.json is out of step with the written changelogs: it was deleted or replaced, or two branches each wrote a changelog. Restore the schema.json that lists it from version control, or merge both branches' schema.json so it lists every written changelog; running jcmChangelog first would write that change again, for a file in the changelog folder that is named NNN-jcm-*.xml or carries the comment jcmChangelog writes into every file, and that schema.json does not list. Without schema.json this comes instead of the missing-snapshot sentence.

jcmChangelog refuses the same three changelog problems before it writes. When the model and schema.json agree, checkJcmModel warns about each mark that changes nothing: Hearing.yml:6: Hearing.room is marked changed: true, but schema.json already holds it as the model has it; remove the mark, so a later change to it is decided afresh, and Hearing.yml: Hearing.status is listed under removed:, but schema.json has no column for it any more; take it out of removed:. With no model file and no schema.json (an app on Java entities) it checks nothing. Both tasks need the changelog folder only when they write or check a changelog; without appPackage or changelogFolder that is refused: set jcm { appPackage = '...' } in build.gradle, or jcm { changelogFolder = file('...') }: jcmChangelog writes into src/main/resources/<appPackage as path>/liquibase/changelog, and a folder outside the resource folders with jcm { changelogFolder } is db, outside the resource folders of main, so the app would not find its changelogs; put it under src/main/resources.

Worked example​

demo-hearings/src/main/jcm/model/Party.yml
entity: Party
instance-name: name
parent: {name: appeal, entity: Appeal}
fields:
name: {type: text, length: 100, required: true}
kind: {type: ref, list: ref.party-kind}
email: {type: text, length: 254}
representative: {type: reference, entity: PersonProfile}