Tony CHEMIT pushed to branch develop at ultreiaio / ird-observe Commits: df376a3f by Tony Chemit at 2022-09-11T20:04:04+02:00 Update changelog for v.9.0.7 [skip CI] - - - - - 553ce875 by Tony Chemit at 2022-09-11T20:23:37+02:00 update hibernate - - - - - 355d696b by Tony Chemit at 2022-09-11T20:34:39+02:00 update pom - - - - - a9be2e1d by Tony Chemit at 2022-09-11T22:35:16+02:00 Problème lors d'un déplacement de toutes les données d'une marée Senne - Close #2428 - - - - - e6cebd5e by Tony Chemit at 2022-09-11T23:24:38+02:00 Persistence - Improve ReplicatePlan (required types should be replace when possible) - See #2428 - - - - - f2f53e78 by Tony Chemit at 2022-09-12T00:38:25+02:00 Merge branch 'feature/features/fix_replication_sql_2428' into develop - - - - - 9 changed files: - CHANGELOG.md - core/persistence/java/src/main/resources/META-INF/persistence/Observe/TopiaEntitySqlReplicatePlanModel.json - pom.xml - toolkit/persistence/src/main/java/org/nuiton/topia/service/sql/TopiaEntitySqlModelResource.java - toolkit/persistence/src/main/java/org/nuiton/topia/service/sql/internal/consumer/DeletePartialEntityConsumer.java - toolkit/persistence/src/main/java/org/nuiton/topia/service/sql/internal/consumer/DuplicateEntityConsumer.java - toolkit/persistence/src/main/java/org/nuiton/topia/service/sql/internal/consumer/ReplicateEntityWork.java - toolkit/persistence/src/main/java/org/nuiton/topia/service/sql/internal/consumer/ReplicateEntityWorkContext.java - toolkit/templates/src/main/java/org/nuiton/topia/templates/sql/plan/TopiaEntitySqlReplicatePlanBuilder.java Changes: ===================================== CHANGELOG.md ===================================== @@ -1,7 +1,7 @@ # ObServe changelog * Author [Tony Chemit](mailto:dev@tchemit.fr) - * Last generated at 2022-09-02 19:20. + * Last generated at 2022-09-11 20:03. ## Version [9.0.7](https://gitlab.com/ultreiaio/ird-observe/-/milestones/232) @@ -15,15 +15,25 @@ ### Issues * [[Type::Anomalie 2324]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2324) **Ouverture de base impossible selon configuration précédente de l'arbre** (Thanks to Tony CHEMIT) (Reported by Pascal Cauquil) * [[Type::Anomalie 2327]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2327) **Clé de libellé non traduite en anglais sur la page de synchrod e la synchro avancée** (Thanks to Tony CHEMIT) (Reported by Pascal Cauquil) + * [[Type::Anomalie 2331]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2331) **Erreur après une mise à jour simple de référentiel de la base locale** (Thanks to Tony CHEMIT) (Reported by Pascal Cauquil) + * [[Type::Anomalie 2340]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2340) **La taille de la fenêtre de configuration de l'arbre de navigation tronque les boutons du bas si on passe du module PS vers LL** (Thanks to Tony CHEMIT) (Reported by Pascal Cauquil) + * [[Type::Anomalie 2381]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2381) **Connection manager, bases distantes, champs non visibles sous Windows** (Thanks to Tony CHEMIT) (Reported by Pascal Cauquil) + * [[Type::Anomalie 2399]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2399) **Problème SSL - en attente - ne s'est pas reproduit** (Thanks to Pascal Cauquil) (Reported by Pascal Cauquil) * [[Type::Anomalie 2411]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2411) **HResponseErrorException / InvalidAuthenticationTokenException en 9.0.6 en quittant l'application** (Thanks to Tony CHEMIT) (Reported by Pascal Cauquil) * [[Type::Anomalie 2412]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2412) **Certains services sont encore en GET alors qu'ils devraient être en POST** (Thanks to Tony CHEMIT) (Reported by Tony CHEMIT) + * [[Type::Anomalie 2415]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2415) **Erreur lors du rechargement de la source si token expiré** (Thanks to Tony CHEMIT) (Reported by Pascal Cauquil) * [[Type::Anomalie 2416]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2416) **La boîte 'A propos' ne fontionne pas** (Thanks to Tony CHEMIT) (Reported by Pascal Cauquil) * [[Type::Anomalie 2419]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2419) **Erreur de migration de l'espèce AVDTH 8 de CAPT_ELEM vers ps_logbook.catch** (Thanks to Tony CHEMIT) (Reported by Pascal Cauquil) * [[Type::Anomalie 2420]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2420) **Statut de la doc de mapping des espèces AVDTH/ObServe** (Thanks to Tony CHEMIT) (Reported by Pascal Cauquil) * [[Type::Anomalie 2423]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2423) **Erreur calcul des données sur V9** (Thanks to Tony CHEMIT) (Reported by Philippe Sabarros) * [[Type::Anomalie 2427]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2427) **Erreur de type de mesure sur les échantillons AVDTH migrés** (Thanks to Tony CHEMIT) (Reported by Pascal Cauquil) + * [[Type::Anomalie 2428]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2428) **Problème lors d'un déplacement de toutes les données d'une marée Senne** (Thanks to Tony CHEMIT) (Reported by Tony CHEMIT) + * [[Type::Evolution 1248]](https://gitlab.com/ultreiaio/ird-observe/-/issues/1248) **Mise à jour du module cartographie avec les nouvelles données** (Thanks to Pascal Cauquil) (Reported by Tony CHEMIT) * [[Type::Evolution 1751]](https://gitlab.com/ultreiaio/ird-observe/-/issues/1751) **Mettre un sablier lors de l'ouverture si le nœud a beaucoup de données à charger.** (Thanks to Tony CHEMIT) (Reported by Tony CHEMIT) + * [[Type::Evolution 1866]](https://gitlab.com/ultreiaio/ird-observe/-/issues/1866) **Remplir les données depuis AVDTH** (Thanks to Tony CHEMIT) (Reported by Tony CHEMIT) * [[Type::Evolution 2104]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2104) **On pourrait avoir un service de génération de topiaid** (Thanks to Tony CHEMIT) (Reported by Pascal Cauquil) + * [[Type::Evolution 2223]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2223) **Ajouter une action nouvelle marée dans l'arbre de navigation** (Thanks to Tony CHEMIT) (Reported by Tony CHEMIT) + * [[Type::Evolution 2225]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2225) **Bien gérer le repositionnement dans l'arbre de navigation d'une marée suite à une sauvegarde si le critère a été modifié** (Thanks to Tony CHEMIT) (Reported by Tony CHEMIT) * [[Type::Evolution 2232]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2232) **Dans l'UI des profils de connexion, trimmer les espaces** (Thanks to Tony CHEMIT) (Reported by Pascal Cauquil) * [[Type::Evolution 2259]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2259) **Dans l'assistant rapport, ajouter un export CSV** (Thanks to Tony CHEMIT) (Reported by Pascal Cauquil) * [[Type::Evolution 2293]](https://gitlab.com/ultreiaio/ird-observe/-/issues/2293) **Il serait intéressant de permettre l'absence de base déclarée par défaut** (Thanks to Tony CHEMIT) (Reported by Pascal Cauquil) ===================================== core/persistence/java/src/main/resources/META-INF/persistence/Observe/TopiaEntitySqlReplicatePlanModel.json ===================================== @@ -898,6 +898,7 @@ "from": "WHERE sampleActivity.sample %s", "parentColumnName": "sample", "columnNames": "topiaId,topiaCreateDate,topiaVersion,lastUpdateDate,homeId,weightedWeight,activity,sample", + "columnsToDetach": "activity~ps_logbook.activity", "requiredTypes": "ps_logbook.activity" }, { @@ -926,6 +927,7 @@ "from": "WHERE sampleActivity.topiaId %s", "parentColumnName": "sample", "columnNames": "topiaId,topiaCreateDate,topiaVersion,lastUpdateDate,homeId,weightedWeight,activity,sample", + "columnsToDetach": "activity~ps_logbook.activity", "requiredTypes": "ps_logbook.activity", "entryPoint": true } @@ -969,6 +971,7 @@ "from": "WHERE wellPlan.topiaId %s", "parentColumnName": "trip", "columnNames": "topiaId,topiaCreateDate,topiaVersion,trip_idx,lastUpdateDate,homeId,well,weight,activity,species,weightCategory,wellSamplingConformity,wellSamplingStatus,trip", + "columnsToDetach": "activity~ps_logbook.activity", "requiredTypes": "ps_logbook.activity", "entryPoint": true } ===================================== pom.xml ===================================== @@ -23,7 +23,7 @@ <parent> <groupId>io.ultreia.maven</groupId> <artifactId>pom</artifactId> - <version>2022.84</version> + <version>2022.85</version> </parent> <groupId>fr.ird.observe</groupId> <artifactId>ird-observe</artifactId> @@ -119,7 +119,7 @@ <lib.version.h2>1.4.196</lib.version.h2> <!-- FIXME can't use 1.4.197 (date has changed + blob also)--> <!-- FIXME <lib.version.h2>2.1.210</lib.version.h2>--> - <lib.version.hibernate>5.6.10.Final</lib.version.hibernate> + <lib.version.hibernate>5.6.11.Final</lib.version.hibernate> <!-- FIXME <lib.version.hibernate>6.1.0.Final</lib.version.hibernate>--> <lib.version.jaxb-api>2.4.0-b180830.0359</lib.version.jaxb-api> <lib.version.jaxb-runtime>2.4.0-b180830.0438</lib.version.jaxb-runtime> ===================================== toolkit/persistence/src/main/java/org/nuiton/topia/service/sql/TopiaEntitySqlModelResource.java ===================================== @@ -86,10 +86,6 @@ public interface TopiaEntitySqlModelResource { TopiaEntitySqlBlobModel getBlobModel(); -// TopiaMetadataModel getMetaModel(); - -// TopiaMetadataModelPaths getMetaModelPaths(); - Set<ReplicatePartialRequestCallback> getReplicatePartialRequestCallbacks(); Set<ReplicateRequestCallback> getReplicateRequestCallbacks(); ===================================== toolkit/persistence/src/main/java/org/nuiton/topia/service/sql/internal/consumer/DeletePartialEntityConsumer.java ===================================== @@ -42,10 +42,6 @@ import java.util.Set; public class DeletePartialEntityConsumer implements SqlRequestConsumer<DeletePartialEntityRequest> { public static final String DELETE_REVERSE_ASSOCIATION_STATEMENT = "UPDATE %1$s _main SET _main.%2$s = NULL WHERE _main.%2$s %3$s;"; - //fr.ird.data.ps.logbook.Route#1616767950667#0.2733836567509864 30/03/2019 - //fr.ird.data.ps.observation.Activity#1554021069610#0.2462353141793846 - //fr.ird.data.ps.logbook.Activity#1616767975989#0.4873500558653976 - @Override public void consume(DeletePartialEntityRequest request, SqlRequestSetConsumerContext context) { TopiaEntitySqlDeletePlan deletePlan = request.getDeletePlan(); @@ -57,12 +53,12 @@ public class DeletePartialEntityConsumer implements SqlRequestConsumer<DeletePar SqlScriptWriter writer = context.getWriter(); for (TopiaEntitySqlDeletePlanTask task : deletePlan) { if (task.accept(shell)) { - consume(context, writer, ids, task, detachTasksMapping == null ? null : detachTasksMapping.get(task.getGav())); + consume(writer, ids, task, detachTasksMapping == null ? null : detachTasksMapping.get(task.getGav())); } } } - protected void consume(SqlRequestSetConsumerContext context, SqlScriptWriter writer, String ids, TopiaEntitySqlDeletePlanTask task, Map<String, Set<String>> detachTasks) { + protected void consume(SqlScriptWriter writer, String ids, TopiaEntitySqlDeletePlanTask task, Map<String, Set<String>> detachTasks) { String deleteSql = task.getDeleteSql(); String realDeleteSql = task.applyIds(deleteSql, ids); if (detachTasks != null) { ===================================== toolkit/persistence/src/main/java/org/nuiton/topia/service/sql/internal/consumer/DuplicateEntityConsumer.java ===================================== @@ -25,11 +25,9 @@ package org.nuiton.topia.service.sql.internal.consumer; import org.nuiton.topia.service.sql.internal.SqlRequestConsumer; import org.nuiton.topia.service.sql.internal.SqlRequestSetConsumerContext; import org.nuiton.topia.service.sql.internal.request.DuplicateEntityRequest; -import org.nuiton.topia.service.sql.internal.request.ReplicateEntityRequest; import org.nuiton.topia.service.sql.model.TopiaEntitySqlSelectArgument; import org.nuiton.topia.service.sql.plan.replicate.TopiaEntitySqlReplicatePlan; import org.nuiton.topia.service.sql.plan.replicate.TopiaEntitySqlReplicatePlanTask; -import org.nuiton.topia.service.sql.request.ReplicateRequestCallback; import java.util.Set; ===================================== toolkit/persistence/src/main/java/org/nuiton/topia/service/sql/internal/consumer/ReplicateEntityWork.java ===================================== @@ -75,8 +75,7 @@ class ReplicateEntityWork implements SqlWork { this.columnNames = task.getColumnNames(); String selectClause = SqlRequestConsumer.getSelectClause(task.getTableName(), columnNames); this.sql = TopiaEntitySqlReplicatePlanTask.applyIds(task.getSelectSql(), selectClause, ids); - - this.blobsBuilder = context().initBlobsBuilder(task.useBlob(), task.getSchemaAndTableName()); + this.blobsBuilder = context.initBlobsBuilder(task.useBlob(), task.getSchemaAndTableName()); this.blobColumns = blobsBuilder.keySet(); this.insertStatementSql = SqlRequestConsumer.newInsertStatementSql(task.getInsertSql(), columnNames, blobColumns); this.columnsToReplace = task.getColumnsToReplace(); @@ -95,11 +94,16 @@ class ReplicateEntityWork implements SqlWork { this.parentColumnName = parentColumnName; this.entryPoint = task.isEntryPoint(); Set<String> shell = context.getShell(); - // need to detach only when not in shell + // need to detach only when not in shell, otherwise replace this.columnsToDetach = new LinkedHashSet<>(); for (Map.Entry<String, String> entry : task.getColumnsToDetachSimple()) { - if (!shell.contains(entry.getValue())) { - columnsToDetach.add(entry.getKey()); + String otherGav = entry.getValue(); + String columnName = entry.getKey(); + if (!shell.contains(otherGav)) { + columnsToDetach.add(columnName); + } else { + // do replace + columnsToReplace.add(columnName); } } this.recursiveProperty = task.getRecursiveColumnName().orElse(null); @@ -114,10 +118,10 @@ class ReplicateEntityWork implements SqlWork { this.columnNames = task.getColumnNames(); String selectClause = SqlRequestConsumer.getSelectClause(task.getTableName(), columnNames); this.sql = TopiaEntitySqlReplicatePlanTask.applyIds(partialTask.getCopyTask().getSelectSql(), selectClause, ids); - this.blobsBuilder = context().initBlobsBuilder(task.useBlob(), task.getSchemaAndTableName()); + this.blobsBuilder = context.initBlobsBuilder(task.useBlob(), task.getSchemaAndTableName()); this.blobColumns = blobsBuilder.keySet(); this.insertStatementSql = SqlRequestConsumer.newInsertStatementSql(task.getInsertSql(), columnNames, blobColumns); - this.columnsToReplace = task.getColumnsToReplace(); + this.columnsToReplace = new LinkedHashSet<>(task.getColumnsToReplace()); List<String> parentColumnNames = task.getParentColumnName(); String parentColumnName = null; if (parentColumnNames.size() == 1) { @@ -133,11 +137,16 @@ class ReplicateEntityWork implements SqlWork { this.parentColumnName = parentColumnName; this.entryPoint = task.isEntryPoint(); Set<String> shell = context.getShell(); - // need to detach only when not in shell + // need to detach only when not in shell, otherwise replace this.columnsToDetach = new LinkedHashSet<>(); for (Map.Entry<String, String> entry : task.getColumnsToDetachSimple()) { - if (!shell.contains(entry.getValue())) { - columnsToDetach.add(entry.getKey()); + String otherGav = entry.getValue(); + String columnName = entry.getKey(); + if (!shell.contains(otherGav)) { + columnsToDetach.add(columnName); + } else { + // do replace + columnsToReplace.add(columnName); } } this.recursiveProperty = task.getRecursiveColumnName().orElse(null); @@ -168,7 +177,7 @@ class ReplicateEntityWork implements SqlWork { public void execute(Connection connection) throws SQLException { long t0 = TimeLog.getTime(); try (PreparedStatement statement = connection.prepareStatement(sql)) { - statement.setFetchSize(context.context().getReadFetchSize()); + statement.setFetchSize(context.getReadFetchSize()); try (ResultSet resultSet = statement.executeQuery()) { if (recursiveProperty == null) { simpleCopy(statement, resultSet); @@ -199,7 +208,7 @@ class ReplicateEntityWork implements SqlWork { parents.put(key == null ? sentinel : key, topiaId); requestsByTopiaId.put(topiaId, sql); } - List<String> orderedTopiaIds = context().generateOrder(parents, sentinel); + List<String> orderedTopiaIds = context.generateOrder(parents, sentinel); if (orderedTopiaIds.size() != requestsByTopiaId.size()) { throw new IllegalStateException("Mismatch ids size!!!"); } @@ -251,11 +260,6 @@ class ReplicateEntityWork implements SqlWork { String stringValue = (String) columnValue; if (entryPoint && stringValue.equals(oldParentId)) { argumentsBuilder.append(", NULL"); -// if (newParentId.toLowerCase().contains("." + columnName + "#")) { -// argumentsBuilder.append(", '").append(newParentId).append("'"); -// } else { -// argumentsBuilder.append(", NULL"); -// } continue; } argumentsBuilder.append(", '").append(stringValue.replaceAll("'", "''")).append("'"); @@ -270,7 +274,7 @@ class ReplicateEntityWork implements SqlWork { for (String columnName : blobColumns) { Object columnValue = resultSet.getObject(columnName); if (blobColumns.contains(columnName)) { - context().copyBlob(statement, rowTopiaId, columnValue, blobsBuilder.get(columnName)); + context.copyBlob(statement, rowTopiaId, columnValue, blobsBuilder.get(columnName)); } } String arguments = argumentsBuilder.substring(2); ===================================== toolkit/persistence/src/main/java/org/nuiton/topia/service/sql/internal/consumer/ReplicateEntityWorkContext.java ===================================== @@ -22,10 +22,15 @@ package org.nuiton.topia.service.sql.internal.consumer; * #L% */ +import com.google.common.collect.Multimap; +import io.ultreia.java4all.util.sql.BlobsContainer; import io.ultreia.java4all.util.sql.SqlScriptWriter; import org.nuiton.topia.persistence.TopiaIdFactoryForBulkSupport; import org.nuiton.topia.service.sql.internal.SqlRequestSetConsumerContext; +import java.sql.PreparedStatement; +import java.sql.SQLException; +import java.util.List; import java.util.Map; import java.util.Objects; import java.util.Set; @@ -86,4 +91,20 @@ class ReplicateEntityWorkContext { public void writeSql(String sql) { writer.writeSql(sql); } + + public Map<String, BlobsContainer.Builder> initBlobsBuilder(boolean useBlob, String schemaAndTableName) { + return context.initBlobsBuilder(useBlob, schemaAndTableName); + } + + public int getReadFetchSize() { + return context.getReadFetchSize(); + } + + public void copyBlob(PreparedStatement preparedStatement, String topiaId, Object columnValue, BlobsContainer.Builder blobsBuilder) throws SQLException { + context.copyBlob(preparedStatement, topiaId, columnValue, blobsBuilder); + } + + public List<String> generateOrder(Multimap<String, String> parents, String sentinel) { + return context.generateOrder(parents, sentinel); + } } ===================================== toolkit/templates/src/main/java/org/nuiton/topia/templates/sql/plan/TopiaEntitySqlReplicatePlanBuilder.java ===================================== @@ -136,25 +136,24 @@ public class TopiaEntitySqlReplicatePlanBuilder { throw new IllegalArgumentException(String.format("Can't manage with this case: %s", parentColumnNames)); } } +// Set<String> requiredTypes = new LinkedHashSet<>(); Set<String> requiredTypes = new LinkedHashSet<>(); - Set<String> realRequiredTypes = new LinkedHashSet<>(); String literalName = metadataEntity.getType(); if (mandatoryReverseCompositions.containsKey(literalName)) { for (TopiaMetadataComposition composition : mandatoryReverseCompositions.get(literalName)) { String type = composition.getTarget().getType(); if (missingTypes.contains(type)) { - requiredTypes.add(type); - realRequiredTypes.add(composition.getTarget().getSchemaAndTableName()); + requiredTypes.add(composition.getTarget().getSchemaAndTableName()); } } } List<String> columNames = List.copyOf(table.getAuthorizedColumnNames()); Set<String> columnsToDetach = new LinkedHashSet<>(); for (Map.Entry<String, String> entry : metadataEntity.getManyToOneAssociations().entrySet()) { - addSpecialColumns(columnsToReplace, columnsToDetach, requiredTypes, entry); + addSpecialColumns(columnsToReplace, columnsToDetach, entry); } for (Map.Entry<String, String> entry : metadataEntity.getReversedAssociations().entrySet()) { - addSpecialColumns(columnsToReplace, columnsToDetach, requiredTypes, entry); + addSpecialColumns(columnsToReplace, columnsToDetach, entry); } // the parent column is always to replace parentColumnNames.forEach(columnsToReplace::remove); @@ -167,7 +166,7 @@ public class TopiaEntitySqlReplicatePlanBuilder { columNames, columnsToReplace, columnsToDetach, - realRequiredTypes, + requiredTypes, useBlob, mainTable, recursiveColumnName)); @@ -228,12 +227,12 @@ public class TopiaEntitySqlReplicatePlanBuilder { null)); } - private void addSpecialColumns(Set<String> columnsToReplace, Set<String> columnsToDetach, Set<String> requiredTypes, Map.Entry<String, String> entry) { + private void addSpecialColumns(Set<String> columnsToReplace, Set<String> columnsToDetach, Map.Entry<String, String> entry) { String literalName = entry.getValue(); if (literals.contains(literalName)) { // this one need to be replace columnsToReplace.add(entry.getKey()); - } else if (!standaloneLiterals.contains(literalName) && !requiredTypes.contains(literalName)) { + } else if (!standaloneLiterals.contains(literalName)) {// && !requiredTypes.contains(literalName)) { // this one need to be detach int endIndex = literalName.lastIndexOf("_"); String columnDefinition = entry.getKey() + "~" + literalName.substring(0, endIndex) + "." + Introspector.decapitalize(literalName.substring(endIndex + 1)); View it on GitLab: https://gitlab.com/ultreiaio/ird-observe/-/compare/d685e4aa9f7585b1cc15eaf78... -- View it on GitLab: https://gitlab.com/ultreiaio/ird-observe/-/compare/d685e4aa9f7585b1cc15eaf78... You're receiving this email because of your account on gitlab.com.
participants (1)
-
Tony CHEMIT (＠tchemit)