Data model and migrations
Data model
A case type's entities are declared in model files, not Java: one file per entity at
src/main/jcm/model/<Entity>.yml. The model has no enums (a ref field over a reference list takes their place),
embedded parts, many-to-many references or inheritance in this release, and cannot drop a whole entity; an entity
that needs one is a Jmix entity in Java with a changelog of its own, as the app's User is. These are build input, not classpath configuration, and no override folder or
studio publish changes them. The library jcm-model (com.jcm:jcm-model, in the BOM; snakeyaml only, no Spring and
no Jmix) reads and checks them, records the schema they need (schema.json, below) and works out what changes from
one to the next. The com.jcm.app plugin's generateJcmModel writes the entities and captions from them at
build time (below), its jcmChangelog writes the next changelog when the model changes, and its checkJcmModel
(under check) holds the model, schema.json and the written changelogs together. A new app's starter case type
is a model file: jcm new writes src/main/jcm/model/<Name>.yml (below), its schema.json and
020-jcm-create-<name>.xml from templates, byte for byte what jcmChangelog writes for that file, with
jcm { appPackage } in build.gradle, and no entity Java or captions:
entity: Request # a name whose table is a reserved word adds table: <NAME>_FILE (Case: CASE_FILE)
instance-name: title
fields:
title: {type: text, length: 200, required: true}
receivedOn: {type: date}
urgent: {type: boolean, default: false}
urgencyReason: {type: text, length: 1000, label: Why it is urgent}
entity: Party # class and entity name; the file must be Party.yml; the table defaults to PARTY
instance-name: name # optional; must name one of the fields
parent: {name: appeal, entity: Appeal} # makes this a child: required, cascade on delete
fields:
name: {type: text, length: 100, required: true, label: Name}
kind: {type: ref, list: ref.party-kind}
notes: {type: long-text}
On the case entity, children: names its compositions (attribute: entity, kept in file order); each child's file
names this entity as its parent:
entity: Appeal
instance-name: caseTitle
fields:
caseTitle: {type: text, length: 200, required: true, label: Case title}
receivedOn: {type: date}
summary: {type: long-text}
children:
parties: Party
hearings: Hearing
decisions: Decision
The file's keys are entity (required), table, instance-name, parent, fields, children, audit,
removed, extends and unique.
| Type | Java and column | Notes |
|---|---|---|
text | String, varchar(length) | length defaults to 255, at most 4000 (use long-text above); index: true up to length 400 |
long-text | @Lob String, clob | |
date | LocalDate, date | |
date-time | LocalDateTime, datetime | |
boolean | Boolean, boolean | requires a default:; the column is not null |
integer | Integer, int | |
decimal | BigDecimal, decimal(precision, scale) | defaults to 19, 2; effective precision 1–38, scale 0–precision, checked after defaults |
ref | RefItem, @RefListCode(list) | a reference-data lookup; list is required, and every save holds the field to that list's items (Reference data) |
user | the app's User | |
reference | another model entity named by entity: | |
document | UUID, @DocumentRef(types) | one of the record's documents at one revision; optional types: [CODE, ...]; needs jcm-attachments (Attachments, Document revision fields) |
- Every field takes
type(required),required,label(defaults to the name as words:receivedOnis "Received on"),column(for a reserved word, such asvalue),index: true,changed: trueandaudit: false(keep no history of the field's changes; see Field history;audit: trueis refused, being the default). Onlytexttakeslength, onlydecimalprecisionandscale, onlybooleandefault, onlyreflist, onlyreferenceentityand onlydocumenttypes. - Names follow the demo's changelogs:
UPPER_SNAKEtables and columns (InboundDocumentisINBOUND_DOCUMENT,receivedOnisRECEIVED_ON); aref,user,referenceordocumentfield's column ends in_ID(kindisKIND_ID), and so does the parent's (APPEAL_ID);IDX_<table>_ON_<attribute>,FK_<table>_ON_<attribute>,PK_<table>(IDX_PARTY_ON_APPEAL, named after the attribute, not the column), andUK_<table>_ON_<attribute>[_<attribute>]for aunique:entry (UK_MEMBER_RECORD_ON_MEMBER_ID_SOURCE_SYSTEM). - References,
refanduserfields and parents always get an index and a foreign key;index: trueon one is allowed and changes nothing,index: falseis refused. Adocumentfield always gets an index and never a foreign key: a document's row is not a model table. - Every entity has a UUID
idand an Integerversion;audit: trueaddscreatedBy,createdDate,lastModifiedByandlastModifiedDate.auditon the entity is about those stamps only; field history is on for every field regardless, and a field leaves it withaudit: falseon the field itself. unique:is a list of attribute lists,unique: [[memberId], [memberId, sourceSystem]]: the values of each entry's attributes together are on one row only, held by a unique index over their columns in the entry's order. An entry names one to three fields of typetext,integer,date,reforreference, eachrequired: true(a key that may be empty is no key, and SQL Server's unique index holds one null only). The entry's width is at most 1,700 bytes, SQL Server's limit for an index key: atextcounts itslength(varchar(n)holds n bytes under the UTF-8 collation), aninteger4, adate3, areforreference16. The generated entity carries@Index(name = "UK_...", columnList = "MEMBER_ID, SOURCE_SYSTEM", unique = true). PostgreSQL and HSQLDB compare the key exactly and SQL Server by its collation (ignoring case), soAB1andab1are two keys to the first two and one to the third; adding an entry over rows that exist counts them ignoring case on every database (below).- A removed field is listed under
removed: [status]; a length cut, another decimal precision or scale, or a field becoming required carrieschanged: trueon the field. Those are the only type changes made (text to text, decimal to decimal); any other is refused even when marked (below).
Every problem in every file is reported at once, each as <file>:<line>: and a sentence, sorted by file and line.
A file that cannot be read (not YAML, no entity:, an entity name that is not a name) is reported in itself only:
the other files are not told that its entity is missing. These are refused:
Appeal.yml:4: unknown key 'colour'; the keys are entity, table, instance-name, parent, fields, children, audit, removed, extends, uniqueAppeal.yml:3: field 'caseTitle' has the unknown key 'size'; the keys are type, length, precision, scale, required, label, column, index, list, entity, default, changed, audit, typesAppeal.yml:3: field 'caseTitle' has audit: true, which is the default; write audit: false to keep no history of the field, or leave the key outAppeal.yml:4: the key 'caseTitle' appears twice(any key written twice)Appeal.yml:1: has no entity:; give it entity: Appeal, andParty.yml:1: is not a model file; it holds entity: and fields: as a mapping; a file that is not YAML:<file>:<line>: is not valid YAML: <what the reader found>Apeal.yml:1: holds entity Appeal, so it must be named Appeal.ymlParty.yml:6: field 'kind' has the unknown type 'lookup'; the types are text, long-text, date, date-time, boolean, integer, decimal, ref, user, reference, document(has no type:when it has none)Case.yml:5: field 'title' is text and does not take 'list'Case.yml:5: field 'urgent' is boolean and needs default: true or default: falseReview.yml:4: field 'report' has types that are not a list of document type codes, as types: [REPORT](a single word, an empty list, a mapping),field 'report' names the document type 'REPORT' twice,field 'report' is document and always has an index; drop index: false, a required one on an entity with noparent:(Attachments, Document revision fields), and on an entity withextends:,UserProfile.yml:4: field 'badge' is document, which names one of a case's documents; UserProfile extends user and belongs to no caseParty.yml:6: field 'kind' is ref and always has an index; drop index: falseNote.yml:3: field 'body' is long-text, which SQL Server cannot index; drop index: true(Liquibase'sclobisvarchar(max)there, which no index key can hold)Note.yml:4: field 'more' has length 4001; text holds at most 4000 characters: use long-text for a new field; for a field that exists, cut it to 4000 with changed: true, or move it to long-text in stages (add a long-text field, copy the values, remove this one)(above 8000 Liquibase makes the columnvarchar(max)on SQL Server)Note.yml:5: field 'title' has length 401 and index: true; SQL Server indexes text up to 400 characters: shorten it to 400 (with changed: true for a field that exists), or drop index: true from a new field(SQL Server's nonclustered index key holds 1,700 bytes; 400 characters of four UTF-8 bytes stay under it)
Upgrading an app whose model files predate these limits: checkJcmModel now refuses a text longer than 4000
and an indexed text longer than 400. For a field that already exists, cut it with changed: true (length: 4000,
or length: 400 for the indexed one), a guarded narrowing that fails, leaving every row as it was, while a value is
longer; or, for text that holds longer values, move it in stages: add a long-text field, copy the values, remove
the old field. Changing it to long-text in place and dropping its index are both refused.
Party.yml:6: field 'kind' is ref and needs list: with the reference list's code, andfield 'previous' is reference and needs entity: with the entity's name- A value of the wrong kind:
Case.yml:5: field 'title' has length 'long'; it takes a whole number above 0, in plain digits(numbers are decimal digits only:0x10,1_000,+5and1.5are refused,012is 12),field 'title' has required 'maybe'; it takes true or false,audit is 'often'; it takes true or false,field 'amount' has scale 6 above its precision 4; a YAML null or an empty value where a word belongs:Appeal.yml:4: field 'caseTitle' has label '~'; it takes a word or phrase(andinstance-name is '~', ...) - Names Java cannot use:
field 'case-title' is not a name a field can have; use letters and digits, starting with a lower-case letter(and the same for a child, a parent's name and an entity, which starts upper-case),field 'class' is a word Java keeps; name the field otherwise Case.yml:5: 'version' is every entity's own attribute; name the field otherwise(andid); withaudit: true,'createdBy' is an attribute audit: true adds; name the field otherwise(likewisecreatedDate,lastModifiedBy,lastModifiedDate). A parent or a child named so is refused the same way, endingname the parent otherwiseorname the child otherwiseHearing.yml:2: parent has the unknown key 'kind'; the keys are name, entity, andparent needs name: and entity:, as {name: appeal, entity: Appeal}Appeal.yml:8: child 'parties' has the name of field 'parties'; name one of them otherwise(a field, the parent and the children share one set of names)Appeal.yml:2: instance-name 'title' is no field of Appeal; the fields are caseTitle, receivedOnHearing.yml:7: field 'venue' names the entity 'Room', which no model file declares(and the same for a child or a parent:child 'parties' names the entity 'Party', ...,parent names the entity 'Appeal', ...)Appeal.yml:9: child 'parties' is Party, whose parent is not Appeal; give Party.yml parent: {name: appeal, entity: Appeal}Appeal.yml:9: child 'others' is Party, which child 'parties' already holdsParty.yml:3: Party names Appeal as its parent, but Appeal.yml lists it under no childrenNode.yml:2: Node cannot be its own parent; for a tree of one entity, give it a field of type: reference, entity: Node(its parent column is not null, so no first row could exist, and SQL Server refuses a key that cascades onto its own table)Party.yml:7: field 'kindId' has the column KIND_ID, which field 'kind' already has(andwhich parent 'appeal' already has,which every entity hasforIDandVERSION,which audit: true adds); two entities with one table:entity Foo has the table PARTY, which Party.yml already hasParty.yml:6: field 'value' would have the column VALUE, a reserved word in SQL; give it a column of its own with column:; acolumn:that is itself reserved:field 'order' has the column POSITION, a reserved word in SQL; choose another;Order.yml:1: entity Order would have the table ORDER, a reserved word in SQL; give it a table of its own with table:(table ORDER is a reserved word in SQL; choose anotherfor atable:). The reserved words are those of HSQLDB, PostgreSQL and SQL Server together, so a model that passes runs on all three.- A
column:ortable:that is notUPPER_SNAKE:field 'x' has the column 'amount', which is not a name a column can have; use upper-case letters, digits and _, starting with a letter - A generated name above 63 characters, which PostgreSQL would cut:
Party.yml:5: field '<f>' makes the index name IDX_PARTY_ON_<F>, 64 characters; PostgreSQL keeps 63. Shorten the field's name or give the entity a shorter table:; for a column,... makes the column name <C>, 64 characters; PostgreSQL keeps 63. Shorten the field's name or give it a shorter column:; for a foreign key,makes the foreign key name; for the table or primary key,entity <E> makes the primary key name PK_<T>, 65 characters; PostgreSQL keeps 63. Give it a shorter table: Hearing.yml:9: 'room' is under removed: and under fields:; keep one- A
unique:that is not a list of lists of names:MemberRecord.yml:9: unique: holds a list of attribute lists, as [[memberId], [memberId, sourceSystem]] MemberRecord.yml:9: the unique entry [colour] names 'colour', which is no field of MemberRecord; the fields are memberId, sourceSystem, lastName(the parent, a child and the audit attributes are no fields)MemberRecord.yml:9: the unique entry [balance] names 'balance', which is decimal; a unique attribute is text, integer, date, ref or referenceMemberRecord.yml:9: the unique entry [lastName] names 'lastName', which is not required; a key that may be empty is no key: give the field required: true (with changed: true for a field that exists)MemberRecord.yml:9: the unique entry [memberId, sourceSystem, bornOn, badge] names 4 attributes; an entry takes one to 3,a unique entry names one to 3 attributes; this one names nonefor[], andthe unique entry [badge, badge] names 'badge' twiceMemberRecord.yml:9: the unique entry [memberId] is written twice; keep one(the same attributes in another order are another entry, with an index of its own)MemberRecord.yml:9: the unique entry [wide, badge, employer] is 1710 bytes wide (wide 1690, badge 4, employer 16); SQL Server holds an index key of at most 1700 bytes: shorten a text attribute or take one out of the entryMemberRecord.yml:9: the unique entry [memberId, sourceSystem] names 'sourceSystem', which is under removed:; take it out of the entry, or drop the entry- An entry whose index name is above 63 characters:
the unique entry [...] makes the index name UK_..., 86 characters; PostgreSQL keeps 63. Shorten the attributes' names or give the entity a shorter table:
Generated entities and captions
The com.jcm.app plugin takes the app's package in build.gradle:
jcm {
appPackage = 'com.acme' // required once src/main/jcm/model holds a model file
// modelFolder = file('src/main/jcm/model') the default
// userTable = 'USER_' the default; the table a user field's key points at
// changelogFolder = file('src/main/resources/com/acme/liquibase/changelog') the default
}
generateJcmModel reads the model files and writes, under build/generated/jcm/:
java/<app package>/entity/<Entity>.java: one Jmix entity per file, in<appPackage>.entityunder the file's name, so code and tests that use it by type compile as they would against a hand-written class. Its first line says which model file it comes from. Annotations follow the demo's hand-written entities: UUIDidand Integerversion, the fields in file order (@NotNullandnullable = falsefor a required field and every boolean,@Lobforlong-text, aUUIDwith@DocumentRef, itstypesand@PropertyDatatype("jcm_documentRef")fordocument,@RefListCodewith a lazy@ManyToOnetoRefItemforref, a lazy@ManyToOneto the app'sUserclass, in the same package, foruser, and to the entity forreference; a boolean starts at its default), the audit attributes with Spring Data's@CreatedBy,@CreatedDate,@LastModifiedByand@LastModifiedDate, each child as@Composition @OneToMany(mappedBy = <the child's parent name>), and the parent as@OnDeleteInverse(DeletePolicy.CASCADE)with a required lazy@ManyToOne; the@Tablelists the parent's index and then each indexed field's, named as above; and one getter and setter per attribute.resources/jcm-model/messages.properties: the captions,<app package>.entity/<Entity>=<the name as words>,.id=ID,.version=Version, each field'slabel, each child and the parent as their names in words, and the audit attributes as Created by, Created date, Last modified by and Last modified date. jcm-core adds this bundle last, so a line in the app's ownmessages.propertiesstill overrides a caption.resources/jcm/model/<Entity>.yml, a copy of each model file, andresources/jcm-model/schema.json, a copy of the snapshot when there is one: the model as the app reads it at run time, fixed when it is built (Where files are read from) and held against the entities at startup (What stops startup, The entity model).
With the java plugin applied, the two folders are source and resource folders of main, so compileJava (and
Jmix's entity enhancing) and processResources run generateJcmModel first, and the IDE sees the classes once it
has synced the Gradle project. These files are never edited and never committed: change the model file and build
again. A model file removed takes its class away on the next build. With no model file the task writes nothing.
The task fails, before writing anything, with:
set jcm { appPackage = '...' } in build.gradle: the entities of src/main/jcm/model go into <appPackage>.entity- the model's own refusals above, each with its file and line, under
The model files in src/main/jcm/model are refused: Party is declared in src/main/jcm/model/Party.yml and written in src/main/java/com/acme/entity/Party.java; delete the Java class, for a hand-written class in<appPackage>.entitywith a model entity's name (the slip when an entity moves to a model file), instead of the compiler's duplicate-class error.Hearing.yml:7: field 'officer' is user, so the generated class takes the app's User from com.acme.entity; no source folder of main has com/acme/entity/User.java, when a model has auserfield and the app'sUseris elsewhere.Index.yml:1: entity Index has the name of a type the generated classes use (jakarta.persistence.Index); name the entity otherwise, for an entity named like a type the classes import or name:User(the app's),RefItem,RefListCode,DocumentRef,PropertyDatatype, the Jmix and JPA annotations and enums they use (Entity,Table,Column,Index,Id,Version,Lob,JoinColumn,ManyToOne,OneToMany,FetchType,NotNull,JmixEntity,JmixGeneratedValue,InstanceName,Composition,OnDeleteInverse,DeletePolicy,CreatedBy,CreatedDate,LastModifiedBy,LastModifiedDate),List,UUID,BigDecimal,LocalDate,LocalDateTime,String,Integer,BooleanandObject.
The schema snapshot and its changes
src/main/jcm/model/schema.json records the schema the app's changelogs make for its model, so the next change can
be worked out from it: one entry per entity (in entity order) with its table, primary key, columns (name, attribute,
Liquibase type, nullable, default), indexes and foreign keys, and the changelogs jcmChangelog wrote with the SHA-256
of each. It is written with two-space indents and keys in a fixed order, so the same model always gives the same
bytes; it is committed, never edited by hand. A file that is not such a snapshot is refused with its line:
schema.json:5: unknown key 'colour'; the keys are format, tables, changelogs, schema.json:1: has no 'changelogs',
schema.json:4: the key 'tables' appears twice, schema.json:1: has format 2; this JCM reads format 1, so a newer JCM wrote it, a value of the wrong kind (column OFFICER_ID of CASE_RECORD has nullable 'true'; it takes true or false),
a type it does not hold (column VERSION of CASE_RECORD has the type 'bigint'; schema.json holds the types ${uuid.type}, int, varchar(<length>), clob, date, datetime, boolean, decimal(<precision>, <scale>)), a checksum that
is not 64 lower-case hex digits, a column, entity or table listed twice, a boolean column without "true" or
"false" as its default, and an index or key on a column the table does not have.
An index on one column is {"name": ..., "column": ...}, with "unique": true after them when it is unique (an
extension's owner, a unique: entry on one attribute). The index of a unique: entry on several attributes has
"columns": ["MEMBER_ID", "SOURCE_SYSTEM"] in place of column, in the entry's order; a columns that is not a
list of two or more of the table's columns, each once, is refused. A schema.json written before unique: existed
holds one-column indexes only: it is read as it is, and written back byte for byte.
Columns are ID ${uuid.type} and VERSION int (both not null), the fields in file order, the audit columns
(CREATED_BY varchar(255), CREATED_DATE datetime, LAST_MODIFIED_BY varchar(255), LAST_MODIFIED_DATE datetime,
all nullable) and last the parent column (not null). A field is nullable unless it is required: true; a boolean is
never null and has its default. A ref field's key goes to JCM_REF_ITEM, a user field's to the user table
(USER_), a reference's to its entity's table, and a parent's to the parent's table with cascade on delete.
The model is compared with the snapshot entity by entity and field by field:
- Made without a mark: a new entity's table (a table after the new tables it references), a new field that is not
required (or a boolean, which has its default) with its index and key, a longer
text, a field no longer required,index: trueadded to a field, and aunique:entry added or taken out. An entry added to an entity whose table exists is written as a change that can fail on the rows (below); one on a field that is new on such a table is refused as the new required field is. An entry whose attributes change is the old entry taken out and the new one added. - A wider decimal (at least the old scale and at least the old integer digits) still needs
changed: true, as every decimal change does, but holds every value, so it is written with the additive changes, not as one that can fail. - Made only when the field carries
changed: true: a shortertext, another decimal precision or scale, and an existing field becoming required (once every row has a value). Only text-to-text and decimal-to-decimal type changes are supported. Cross-family conversions, including text to long-text or integer, are refused before either changelog or snapshot is written, even with the mark. Use a staged conversion: add a new nullable field, explicitly convert and verify its values with application-specific code or SQL, switch consumers, then remove the old field. Changing existing rows cannot repair an unsupported conversion. - A new required field without a supported default is refused even with
changed: true: add it nullable, backfill every row, then make it required with the mark. A supported default (currently boolean) permits direct addition. - Made only when the field is listed under
removed:: dropping its column, with its foreign key and index (the four audit attributes too, onceaudit: trueis gone). - A mark the snapshot already agrees with (a
changed: trueon a field that no longer changes, or aremoved:entry whose column is gone) changes nothing in that run. Remove it oncejcmChangeloghas written its changelog:checkJcmModelwarns about each one, naming the field, because achanged: trueleft on a field would let a later cut or type change on it through without a fresh decision. Field order is no change.
Everything else is refused, every refusal at once, and when any is refused no change is made:
Hearing.room is in schema.json but not in Hearing.yml; list it under removed: [room] to drop its column, or put the field backParty.name changes from text(100) to long-text; this conversion is not supported on every database; use a staged conversion: add a new nullable field, explicitly convert and verify its values, then switch to it and remove the old fieldParty.name is cut from 100 to 50 characters; mark the field changed: true to alter its columnAppeal.receivedOn becomes required; mark the field changed: true once every row has a valueAppeal.priority is new and required, but the APPEAL table already exists and its rows have no value; add it without required: true, fill it, then make it required with changed: trueParty.kind changes from ref to text; a field that joins another table cannot change type: remove it and add a new field(also when marked; and forreftouser, or a reference to another entity)Appeal.urgent changes from boolean to text(255); a boolean field cannot change type: remove it and add a new fieldAppeal.urgent changes its default from false to true; jcmChangelog does not change a default in this release: put default: false backHearing.room loses its index; jcmChangelog does not drop an index in this release: put index: true backHearing.room changes its column from ROOM to HEARING_ROOM; a field keeps its column: put column: ROOM back, or remove the field and add a new oneHearing.venue has the column ROOM, which schema.json gives Hearing.room; a column keeps its field: give venue a column of its own, or keep the name room(a field renamed, or a new field on a dropped field's column)Party's table changes from PARTY to CASE_PARTY; jcmChangelog does not rename a table in this release: put table: PARTY backHearing's parent changes from appeal (Appeal) to none; jcmChangelog does not change an entity's parent in this release: put the parent back as it was(and from none, or to another entity)Party has the audit columns in schema.json, but Party.yml has no audit: true; put audit: true back, or list createdBy, createdDate, lastModifiedBy and lastModifiedDate under removed: to drop their columns(turning audit on adds the four columns). On an entity that never hadaudit: true, fields of those names are plain fields and are dropped like any other.Decision is in schema.json but no model file declares it; jcmChangelog does not drop tables in this release: keep Decision.ymlAlpha and Beta are new and reference each other, so neither table can be created first; add them without one of those references, run jcmChangelog, or jcm changelog, then add it
The changes are named in a few words for the changelog's file: create-hearing-and-party, add-hearing-notes,
add-hearing-fields-and-party-email, alter-party-name, drop-hearing-status; by kind (create, add, alter, drop)
and entity, several fields of one entity as <entity>-fields, and more than three parts as the first three and
-and-more.
Changelogs: jcmChangelog, --baseline and checkJcmModel
./gradlew jcmChangelog (jcm changelog with only Docker) compares the model files with schema.json (with no schema.json, with an empty schema,
so every table is created) and writes:
- the next changelog,
<changelogFolder>/<NNN>-jcm-<summary>.xml.NNNis the highest three-digit number that starts a file name in the folder, plus 10 (010in an empty folder;050after the demo's010-040); a name without one is ignored, and past999the task stops.<summary>is the changes in a few words, as above. The folder isjcm { changelogFolder }, by defaultsrc/main/resources/<appPackage as path>/liquibase/changelog, which the app'sincludeAllruns in name order; it must lie inside a resource folder ofmain, and the file'slogicalFilePathis its path there (com/acme/liquibase/changelog/020-jcm-create-case.xml). - the new
schema.json, the model's snapshot with the file and its checksum added tochangelogs: the SHA-256 of the file's UTF-8 text with every CRLF and CR turned into LF, so a checkout that rewrites line ends (Git'score.autocrlfon Windows) holds the same file, as Liquibase's own checksum does. A script that writes these files (the scaffold) hashes the same way.
The changelog is written under <name>.tmp (not a changelog to includeAll), then schema.json, and only then does
the file take its name, so a failure never leaves a written changelog that schema.json does not list.
It logs jcmChangelog wrote <changelog> and src/main/jcm/model/schema.json. With nothing to change it writes nothing
and says The model and schema.json agree; no changelog written. Fields moved to another order are no change to the
database: schema.json is rewritten in the model's order and no changelog is written (The model and schema.json agree but for the order of fields; schema.json rewritten, no changelog written). A refused change fails the task
with every sentence above under jcmChangelog refuses the model in src/main/jcm/model:, and nothing is written.
After it has written, it warns about each mark the model no longer needs (below): remove them then.
The changelog is laid out as the demo's hand-written ones, author jcm, changeSet ids from 1:
- one changeSet per new table:
createTablewithPK_<TABLE>, a not-null column'snullable="false", a boolean'sdefaultValueBoolean, then the parent's index and key (onDelete="CASCADE"), then each indexed field's index and, for a field that joins another table, its key; - one per added field:
addColumn, with its index and key; - one per added index, and for a field whose column changes (a longer text, a wider decimal, no longer required): the column's
dropForeignKeyConstraintanddropIndexin a changeSet of their own when it has them, themodifyDataType(followed byaddNotNullConstraintfor a required column) ordropNotNullConstraint, and the index and key created again in a changeSet of their own. SQL Server'sALTER COLUMNfails on a column an index or a foreign key holds and leaves the column nullable, so the same file does this on every database; - one per removed field:
dropForeignKeyConstraint,dropIndex,dropColumn, as the demo's 040 does; - a
unique:entry on a new table is acreateIndex unique="true"with its columns, after the table's other indexes, in the table's changeSet; an entry taken out is onedropIndex, in a changeSet of its own before every other change to tables that exist (so before the column of a removed field it named). A key column that is altered (made longer, or narrowed with the mark) has the indexes of the entries that name it dropped and created again with its own index, in the same changeSets; - last, the changes marked
changed: truethat can fail on the rows a database holds (a shorter text, a decimal with fewer fractional or integer digits, a field becoming required), each in changeSets of their own (with the index and key inside them as below), under a comment saying so. - a narrowing (a shorter text, a decimal losing digits) is guarded against data loss on PostgreSQL, SQL Server and
HSQLDB only (below), so its
modifyDataTypeis written fordbms="postgresql,mssql"and, through a guard domain, fordbms="hsqldb", never for a negated list; a changeSet before the column's first one, fordbms="!postgresql,!mssql,!hsqldb", halts the deploy on any other database with a precondition (onFail="HALT"):NOTE.AMOUNT changes from decimal(12, 4) to decimal(12, 2), a change JCM makes only where it is guarded against losing data. JCM supports PostgreSQL, SQL Server and HSQLDB; run the application on one of those.Changelogs written before this (none of the demo's, the sample's or the scaffold's narrows) stay as they are. - a narrowing, or a field made required whose column has an index or a foreign key (a
ref,userorreferencefield always has both): on PostgreSQL and SQL Server (dbms="postgresql,mssql") SQL Server's validating SELECT (for a narrowing), thedropForeignKeyConstraintanddropIndex, the alteration and the index and key created again are one changeSet, which rolls back whole when a row cannot take the change, so the index and key are never missing; HSQLDB commits each DDL statement, so there (dbms="hsqldb") the column is altered with its key and index in place, which HSQLDB allows, and a failure changes nothing; any other database halts as above (NOTE.OWNER_ID becomes required, which JCM makes only where a failure keeps its foreign key. ...,... keeps its index.for a plain indexed field). A field made required with no index or key is oneaddNotNullConstrainton every database. - last of all, each
unique:entry added to a table that exists, under the commentA unique: entry in the model: fails while rows hold an empty or repeated key. Fix those rows and run the changelogs again; never edit this file.The rows that would break the index are counted before it is created: a row with a key column null, and rows with the same key, text compared in lower case on every database (AB1andab1are one key). When any is found the migration stops withMEMBER_RECORD holds rows whose MEMBER_ID is empty or not unique ignoring case; find them with: SELECT LOWER(MEMBER_ID), COUNT(*) FROM MEMBER_RECORD GROUP BY LOWER(MEMBER_ID) HAVING COUNT(*) > 1(for several attributeswhose MEMBER_ID and SOURCE_SYSTEM are empty or not unique together ignoring case, with every column in the query;ignoring caseandLOWER()only where an attribute is text), no index is made, and the same changelog finishes once the rows are fixed. On PostgreSQL and SQL Server (dbms="postgresql,mssql") the count and thecreateIndexare one changeSet under a table lock (LOCK TABLE <table> IN SHARE MODE;WITH (TABLOCKX, HOLDLOCK)), so no row arrives between them. HSQLDB commits each DDL statement and a failedCREATE UNIQUE INDEXchanges nothing there, so its changeSet (dbms="hsqldb") counts in asqlCheckprecondition (onFail="HALT") and creates the index alone. Any other database halts at a changeSet before both:MEMBER_RECORD gains the unique index UK_MEMBER_RECORD_ON_MEMBER_ID, which JCM creates only where it can first check the rows for an empty or repeated key. JCM supports PostgreSQL, SQL Server and HSQLDB; run the application on one of those.A key attribute made required in the same change (required: true, changed: true) is altered by the step above first, so rows with no value stop the migration there. - a type change the writer does not know (anything but text to text or decimal to decimal, which
SchemaDiffnever asks for) stopsjcmChangelograther than being written unguarded.
A file jcmChangelog wrote is never edited: Liquibase refuses a changeSet that has run and has changed, and
checkJcmModel refuses it before then. A marked narrowing never silently truncates text or rounds decimals.
Decimal widening requires both at least the old scale and at least the old integer capacity (precision minus
scale); all other decimal changes validate exact representability first. If a marked change fails on the rows
(a value too long, fractional digits or integer range the new decimal cannot hold, an empty value in a field made
required), every changeSet before it has run and is recorded; explicitly correct those rows (shorten, round or
fill them, in the database or with a changelog of the app's own numbered before it) and start the app again: Liquibase runs the rest of the file. Once a database has run part of a file, the rest
must run there. Narrowing protection also covers concurrent writes: PostgreSQL retains a validating CHECK
constraint until after ALTER; HSQLDB alters through a temporary target domain whose constraints validate original
values inside ALTER; SQL Server uses a top-level validating SELECT to start its JDBC-managed implicit transaction
and holds an exclusive table lock through ALTER. An IF EXISTS subquery alone does not start that transaction and
cannot hold the lock between validation and ALTER, even with JDBC autocommit off.
Temporary checks/domains have separate recorded creation and cleanup steps, so retry needs no manual cleanup.
Do not remove these guards or edit a partially applied changelog. While no database has run it, the change can be given up: restore the model files and schema.json
from version control and delete the file.
./gradlew jcmChangelog --baseline (jcm changelog --baseline) records the model as the first schema.json and writes no changelog, for an app
whose own changelogs already create its tables (the demo's 010-040): the model must then describe those tables
exactly, which the run-time schema check proves. Once schema.json exists it is refused: schema.json already exists; --baseline records a first snapshot only. Run jcmChangelog without it. It is refused too while the changelog folder
holds a file jcmChangelog wrote (below): recording the model then would drop those files' checksums.
checkJcmModel, which check runs, fails with:
src/main/jcm/model has no schema.json; run ./gradlew jcmChangelog, or jcmChangelog --baseline when the app's own changelogs already create its tables (jcm changelog, or jcm changelog --baseline, with only Docker), for model files without a snapshot;src/main/jcm/model and schema.json differ: Hearing.notes is new; run ./gradlew jcmChangelog, or jcm changelogwheneverjcmChangelogwould write something, each difference named (Note is new,Hearing.room changes type,Hearing.room becomes required,Hearing.room gains an index,Hearing.officer is listed under removed:,MemberRecord gains the unique index UK_MEMBER_RECORD_ON_MEMBER_ID,MemberRecord loses the unique index ...,Party.name is no longer required,Hearing's fields are in another order);src/main/jcm/model and schema.json differ, and jcmChangelog refuses the difference:and every refusal;050-jcm-add-hearing-notes.xml was written by jcmChangelog and has changed since; a changelog that has run must not change: put it back and change the model instead, when a listed file's SHA-256 differs;050-jcm-add-hearing-notes.xml is listed in schema.json but is not in src/main/resources/com/jcm/demo/liquibase/changelog;050-jcm-add-hearing-notes.xml in src/main/resources/com/jcm/demo/liquibase/changelog was written by jcmChangelog, but schema.json does not list it, so schema.json is out of step with the written changelogs: it was deleted or replaced, or two branches each wrote a changelog. Restore the schema.json that lists it from version control, or merge both branches' schema.json so it lists every written changelog; running jcmChangelog first would write that change again, for a file in the changelog folder that is namedNNN-jcm-*.xmlor carries the commentjcmChangelogwrites into every file, and thatschema.jsondoes not list. Withoutschema.jsonthis comes instead of the missing-snapshot sentence.
jcmChangelog refuses the same three changelog problems before it writes. When the model and schema.json agree,
checkJcmModel warns about each mark that changes nothing: Hearing.yml:6: Hearing.room is marked changed: true, but schema.json already holds it as the model has it; remove the mark, so a later change to it is decided afresh, and
Hearing.yml: Hearing.status is listed under removed:, but schema.json has no column for it any more; take it out of removed:. With no model file and no schema.json (an app on Java entities) it checks nothing. Both tasks need the
changelog folder only when they write or check a changelog; without appPackage or changelogFolder that is refused:
set jcm { appPackage = '...' } in build.gradle, or jcm { changelogFolder = file('...') }: jcmChangelog writes into src/main/resources/<appPackage as path>/liquibase/changelog, and a folder outside the resource folders with jcm { changelogFolder } is db, outside the resource folders of main, so the app would not find its changelogs; put it under src/main/resources.
Worked example
entity: Party
instance-name: name
parent: {name: appeal, entity: Appeal}
fields:
name: {type: text, length: 100, required: true}
kind: {type: ref, list: ref.party-kind}
email: {type: text, length: 254}
representative: {type: reference, entity: PersonProfile}