branch develop updated (ce3262e -> a6cabb5)
This is an automated email from the git hooks/post-receive script. New change to branch develop in repository observe. See http://git.codelutin.com/observe.git from ce3262e ajout des dto stub dans le modèle Seine (refs #7458). new cb0ab4d On doit travailler sur une copie lors de la fermetures des contextes applicatives car ils se déenregistre d'eux même du cache new 737af7e Ajout d'un drapeau pour pouvoir créer des bases h2 vides new 428ea83 Masquer une erreur technique lors de la création d'un répertoire temporaire new f731035 récupération du persistence context dans les services new 1facc96 Implantation de l'export du référentiel new 9c7e8d7 Replication des référentiels et données ok \o/ (see #7476) new a6cabb5 Mettre en place le service qui permet de repliquer des données (termine #7476) Merge branch 'feature/7476' into develop The 7 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "adds" were already present in the repository and have only been added to this reference. Detailed log of new commits: commit a6cabb5d17de867a29584a85eaa5ac9e2e80f11e Merge: 428ea83 9c7e8d7 Author: Tony CHEMIT <chemit@codelutin.com> Date: Wed Aug 26 15:46:20 2015 +0200 Mettre en place le service qui permet de repliquer des données (termine #7476) Merge branch 'feature/7476' into develop commit 9c7e8d75426d168644069d5a2ccfc6612e54a737 Author: Tony CHEMIT <chemit@codelutin.com> Date: Wed Aug 26 15:46:14 2015 +0200 Replication des référentiels et données ok \o/ (see #7476) commit 1facc96d0d6a64c022cb1e90f269c033d407b612 Author: Tony CHEMIT <chemit@codelutin.com> Date: Wed Aug 26 14:26:00 2015 +0200 Implantation de l'export du référentiel commit f7310356514404cf7aad3f87ffb97c5063be19eb Author: Tony CHEMIT <chemit@codelutin.com> Date: Wed Aug 26 14:25:18 2015 +0200 récupération du persistence context dans les services commit 428ea83a7f8e183749d74380ee389c633d7e03ff Author: Tony CHEMIT <chemit@codelutin.com> Date: Wed Aug 26 14:18:03 2015 +0200 Masquer une erreur technique lors de la création d'un répertoire temporaire commit 737af7e9114e3330a5da48d9cf00b20ee050eb8b Author: Tony CHEMIT <chemit@codelutin.com> Date: Wed Aug 26 12:45:03 2015 +0200 Ajout d'un drapeau pour pouvoir créer des bases h2 vides commit cb0ab4d6807bcd0ff0316ccb720dfc08d3e74398 Author: Tony CHEMIT <chemit@codelutin.com> Date: Wed Aug 26 12:43:21 2015 +0200 On doit travailler sur une copie lors de la fermetures des contextes applicatives car ils se déenregistre d'eux même du cache Summary of changes: .../ird/observe/entities/longline/TdrTopiaDao.java | 82 +++++++ .../src/main/xmi/observe-common.properties | 6 +- .../services/ObserveServiceApplicationContext.java | 9 +- .../service/DataSourceDumpProducerService.java | 2 +- .../services/service/DataSourceService.java | 4 +- .../dto/DataSourceCreateConfigurationDto.java | 15 +- .../services/ObserveServiceContextTopia.java | 4 +- .../ird/observe/services/ObserveServiceTopia.java | 3 + .../ObserveTopiaApplicationContextFactory.java | 3 +- .../DataSourceDumpProducerServiceTopia.java | 237 ++++++++++++++++++++- .../services/service/DataSourceServiceTopia.java | 11 +- .../DataSourceDumpProducerServiceTopiaTest.java | 51 +++++ .../service/DataSourceServiceTopiaTest.java | 22 +- 13 files changed, 413 insertions(+), 36 deletions(-) create mode 100644 observe-entities/src/main/java/fr/ird/observe/entities/longline/TdrTopiaDao.java create mode 100644 observe-services-topia/src/test/java/fr/ird/observe/services/service/DataSourceDumpProducerServiceTopiaTest.java -- To stop receiving notification emails like this one, please contact codelutin.com SCM administrator <admin+scm@list.forge.codelutin.com>.
This is an automated email from the git hooks/post-receive script. New commit to branch develop in repository observe. See http://git.codelutin.com/observe.git commit cb0ab4d6807bcd0ff0316ccb720dfc08d3e74398 Author: Tony CHEMIT <chemit@codelutin.com> Date: Wed Aug 26 12:43:21 2015 +0200 On doit travailler sur une copie lors de la fermetures des contextes applicatives car ils se déenregistre d'eux même du cache --- .../fr/ird/observe/services/ObserveTopiaApplicationContextFactory.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/observe-services-topia/src/main/java/fr/ird/observe/services/ObserveTopiaApplicationContextFactory.java b/observe-services-topia/src/main/java/fr/ird/observe/services/ObserveTopiaApplicationContextFactory.java index c5cc904..7d9783a 100644 --- a/observe-services-topia/src/main/java/fr/ird/observe/services/ObserveTopiaApplicationContextFactory.java +++ b/observe-services-topia/src/main/java/fr/ird/observe/services/ObserveTopiaApplicationContextFactory.java @@ -13,6 +13,7 @@ import org.apache.commons.logging.LogFactory; import org.nuiton.topia.persistence.TopiaApplicationContext; import java.util.HashMap; +import java.util.LinkedHashSet; import java.util.Map; /** @@ -67,7 +68,7 @@ public class ObserveTopiaApplicationContextFactory { public static void close() { - for (TopiaApplicationContext topiaApplicationContext : TOPIA_APPLICATION_CONTEXT_CACHE.values()) { + for (TopiaApplicationContext topiaApplicationContext : new LinkedHashSet<>(TOPIA_APPLICATION_CONTEXT_CACHE.values())) { if (!topiaApplicationContext.isClosed()) { try { -- To stop receiving notification emails like this one, please contact codelutin.com SCM administrator <admin+scm@list.forge.codelutin.com>.
This is an automated email from the git hooks/post-receive script. New commit to branch develop in repository observe. See http://git.codelutin.com/observe.git commit 737af7e9114e3330a5da48d9cf00b20ee050eb8b Author: Tony CHEMIT <chemit@codelutin.com> Date: Wed Aug 26 12:45:03 2015 +0200 Ajout d'un drapeau pour pouvoir créer des bases h2 vides --- .../dto/DataSourceCreateConfigurationDto.java | 15 ++++++++++++++- .../service/DataSourceServiceTopiaTest.java | 22 ++++------------------ 2 files changed, 18 insertions(+), 19 deletions(-) diff --git a/observe-services-model/src/main/java/fr/ird/observe/services/dto/DataSourceCreateConfigurationDto.java b/observe-services-model/src/main/java/fr/ird/observe/services/dto/DataSourceCreateConfigurationDto.java index 983f7b1..ec129f9 100644 --- a/observe-services-model/src/main/java/fr/ird/observe/services/dto/DataSourceCreateConfigurationDto.java +++ b/observe-services-model/src/main/java/fr/ird/observe/services/dto/DataSourceCreateConfigurationDto.java @@ -32,6 +32,19 @@ public class DataSourceCreateConfigurationDto extends AbstractObserveDto { */ protected String[] optionalImportDataIds; + /** + * Pour autoriser la création d'une base vide (utilisé pour les bases temporaires) + */ + protected boolean canCreateEmptyDatabase; + + public boolean isCanCreateEmptyDatabase() { + return canCreateEmptyDatabase; + } + + public void setCanCreateEmptyDatabase(boolean canCreateEmptyDatabase) { + this.canCreateEmptyDatabase = canCreateEmptyDatabase; + } + public boolean isImportDatabase() { return optionalImportDatabase != null; } @@ -90,7 +103,7 @@ public class DataSourceCreateConfigurationDto extends AbstractObserveDto { } else { - if (!isImportReferential()) { + if (!isCanCreateEmptyDatabase() && !isImportReferential()) { // on n'autorise pas la création d'une base sans référentiel. throw new DataSourceCreateWithNoReferentialImportException(this); diff --git a/observe-services-topia/src/test/java/fr/ird/observe/services/service/DataSourceServiceTopiaTest.java b/observe-services-topia/src/test/java/fr/ird/observe/services/service/DataSourceServiceTopiaTest.java index 4361709..4441b80 100644 --- a/observe-services-topia/src/test/java/fr/ird/observe/services/service/DataSourceServiceTopiaTest.java +++ b/observe-services-topia/src/test/java/fr/ird/observe/services/service/DataSourceServiceTopiaTest.java @@ -59,15 +59,8 @@ public class DataSourceServiceTopiaTest { @Test public void testOpen() throws IOException, IncompatibleDataSourceCreateConfigurationException, DataSourceCreateWithNoReferentialImportException, DatabaseConnexionNotAuthorizedException, DatabaseNotFoundException { - DataSourceCreateConfigurationDto dataSourceCreateConfiguration = new DataSourceCreateConfigurationDto() { - - private static final long serialVersionUID = 1L; - - @Override - public void validateConfiguration() { - // no validation - } - }; + DataSourceCreateConfigurationDto dataSourceCreateConfiguration = new DataSourceCreateConfigurationDto(); + dataSourceCreateConfiguration.setCanCreateEmptyDatabase(true); service.create(dataSourceCreateConfiguration); service.open(); @@ -78,15 +71,8 @@ public class DataSourceServiceTopiaTest { @Test public void testCreateEmptyDataSource() throws IOException, IncompatibleDataSourceCreateConfigurationException, DataSourceCreateWithNoReferentialImportException { - DataSourceCreateConfigurationDto dataSourceCreateConfiguration = new DataSourceCreateConfigurationDto() { - - private static final long serialVersionUID = 1L; - - @Override - public void validateConfiguration() { - // no validation - } - }; + DataSourceCreateConfigurationDto dataSourceCreateConfiguration = new DataSourceCreateConfigurationDto(); + dataSourceCreateConfiguration.setCanCreateEmptyDatabase(true); service.create(dataSourceCreateConfiguration); assertSchemaCreated(); -- To stop receiving notification emails like this one, please contact codelutin.com SCM administrator <admin+scm@list.forge.codelutin.com>.
This is an automated email from the git hooks/post-receive script. New commit to branch develop in repository observe. See http://git.codelutin.com/observe.git commit 428ea83a7f8e183749d74380ee389c633d7e03ff Author: Tony CHEMIT <chemit@codelutin.com> Date: Wed Aug 26 14:18:03 2015 +0200 Masquer une erreur technique lors de la création d'un répertoire temporaire --- .../observe/services/ObserveServiceApplicationContext.java | 9 +++++++-- .../fr/ird/observe/services/service/DataSourceService.java | 4 +--- .../fr/ird/observe/services/ObserveServiceContextTopia.java | 4 ++-- .../ird/observe/services/service/DataSourceServiceTopia.java | 11 ++++++++--- 4 files changed, 18 insertions(+), 10 deletions(-) diff --git a/observe-services-api/src/main/java/fr/ird/observe/services/ObserveServiceApplicationContext.java b/observe-services-api/src/main/java/fr/ird/observe/services/ObserveServiceApplicationContext.java index 49a84ee..5e80a74 100644 --- a/observe-services-api/src/main/java/fr/ird/observe/services/ObserveServiceApplicationContext.java +++ b/observe-services-api/src/main/java/fr/ird/observe/services/ObserveServiceApplicationContext.java @@ -53,9 +53,14 @@ public class ObserveServiceApplicationContext { this.temporaryDirectoryRoot = temporaryDirectoryRoot; } - public File createTemporaryDirectory(String prefix) throws IOException { + public File createTemporaryDirectory(String prefix) { Set<PosixFilePermission> posixFilePermissions = PosixFilePermissions.fromString("rwxr-x---"); FileAttribute<Set<PosixFilePermission>> fileAttribute = PosixFilePermissions.asFileAttribute(posixFilePermissions); - return Files.createTempDirectory(temporaryDirectoryRoot, prefix, fileAttribute).toFile(); + try { + return Files.createTempDirectory(temporaryDirectoryRoot, prefix, fileAttribute).toFile(); + } catch (IOException e) { + //TODO Avoir une exception plus concrete + throw new RuntimeException("Could not create temporary directory with preifx: "+prefix,e); + } } } diff --git a/observe-services-api/src/main/java/fr/ird/observe/services/service/DataSourceService.java b/observe-services-api/src/main/java/fr/ird/observe/services/service/DataSourceService.java index f93dce1..319d81e 100644 --- a/observe-services-api/src/main/java/fr/ird/observe/services/service/DataSourceService.java +++ b/observe-services-api/src/main/java/fr/ird/observe/services/service/DataSourceService.java @@ -6,8 +6,6 @@ import fr.ird.observe.services.dto.DataSourceCreateWithNoReferentialImportExcept import fr.ird.observe.services.dto.IncompatibleDataSourceCreateConfigurationException; import fr.ird.observe.services.spi.NoDataAccess; -import java.io.IOException; - /** * Created on 21/08/15. * @@ -22,7 +20,7 @@ public interface DataSourceService extends ObserveService { boolean canConnect(); @NoDataAccess - void create(DataSourceCreateConfigurationDto dataSourceCreateConfiguration) throws IOException, IncompatibleDataSourceCreateConfigurationException, DataSourceCreateWithNoReferentialImportException; + void create(DataSourceCreateConfigurationDto dataSourceCreateConfiguration) throws IncompatibleDataSourceCreateConfigurationException, DataSourceCreateWithNoReferentialImportException; @NoDataAccess void open() throws DatabaseNotFoundException, DatabaseConnexionNotAuthorizedException; diff --git a/observe-services-topia/src/main/java/fr/ird/observe/services/ObserveServiceContextTopia.java b/observe-services-topia/src/main/java/fr/ird/observe/services/ObserveServiceContextTopia.java index 5bec7f8..c846e89 100644 --- a/observe-services-topia/src/main/java/fr/ird/observe/services/ObserveServiceContextTopia.java +++ b/observe-services-topia/src/main/java/fr/ird/observe/services/ObserveServiceContextTopia.java @@ -7,7 +7,6 @@ import fr.ird.observe.services.configuration.ObserveDataSourceConfigurationTopia import fr.ird.observe.services.dto.constants.ReferentialLocale; import java.io.File; -import java.io.IOException; import java.util.Date; /** @@ -52,7 +51,7 @@ public class ObserveServiceContextTopia implements ObserveServiceContext { } @Override - public File createTemporaryDirectory(String prefix) throws IOException { + public File createTemporaryDirectory(String prefix) { return applicationContext.createTemporaryDirectory(prefix); } @@ -88,4 +87,5 @@ public class ObserveServiceContextTopia implements ObserveServiceContext { topiaPersistenceContext.close(); topiaPersistenceContext = null; } + } diff --git a/observe-services-topia/src/main/java/fr/ird/observe/services/service/DataSourceServiceTopia.java b/observe-services-topia/src/main/java/fr/ird/observe/services/service/DataSourceServiceTopia.java index 25a94d5..52ea0f3 100644 --- a/observe-services-topia/src/main/java/fr/ird/observe/services/service/DataSourceServiceTopia.java +++ b/observe-services-topia/src/main/java/fr/ird/observe/services/service/DataSourceServiceTopia.java @@ -34,7 +34,7 @@ public class DataSourceServiceTopia extends ObserveServiceTopia implements DataS } @Override - public void create(DataSourceCreateConfigurationDto dataSourceCreateConfiguration) throws IOException, IncompatibleDataSourceCreateConfigurationException, DataSourceCreateWithNoReferentialImportException { + public void create(DataSourceCreateConfigurationDto dataSourceCreateConfiguration) throws IncompatibleDataSourceCreateConfigurationException, DataSourceCreateWithNoReferentialImportException { dataSourceCreateConfiguration.validateConfiguration(); @@ -143,10 +143,15 @@ public class DataSourceServiceTopia extends ObserveServiceTopia implements DataS } - protected void executeGzipSqlStatements(ObserveTopiaApplicationContext topiaApplicationContext, String temporaryFilePrefix, byte... content) throws IOException { + protected void executeGzipSqlStatements(ObserveTopiaApplicationContext topiaApplicationContext, String temporaryFilePrefix, byte... content) { File temporaryDirectory = serviceContext.createTemporaryDirectory(temporaryFilePrefix); - topiaApplicationContext.executeGzipSqlStatements(temporaryDirectory, content); + try { + topiaApplicationContext.executeGzipSqlStatements(temporaryDirectory, content); + } catch (IOException e) { + //TODO Avoir une exception concrete + throw new RuntimeException("Could not execute GZipSqlStatements", e); + } } -- To stop receiving notification emails like this one, please contact codelutin.com SCM administrator <admin+scm@list.forge.codelutin.com>.
This is an automated email from the git hooks/post-receive script. New commit to branch develop in repository observe. See http://git.codelutin.com/observe.git commit f7310356514404cf7aad3f87ffb97c5063be19eb Author: Tony CHEMIT <chemit@codelutin.com> Date: Wed Aug 26 14:25:18 2015 +0200 récupération du persistence context dans les services --- .../src/main/java/fr/ird/observe/services/ObserveServiceTopia.java | 3 +++ 1 file changed, 3 insertions(+) diff --git a/observe-services-topia/src/main/java/fr/ird/observe/services/ObserveServiceTopia.java b/observe-services-topia/src/main/java/fr/ird/observe/services/ObserveServiceTopia.java index 76cbb97..263af71 100644 --- a/observe-services-topia/src/main/java/fr/ird/observe/services/ObserveServiceTopia.java +++ b/observe-services-topia/src/main/java/fr/ird/observe/services/ObserveServiceTopia.java @@ -257,4 +257,7 @@ public class ObserveServiceTopia implements ObserveService { return serviceContext.now(); } + public ObserveTopiaPersistenceContext getTopiaPersistenceContext() { + return serviceContext.getTopiaPersistenceContext(); + } } -- To stop receiving notification emails like this one, please contact codelutin.com SCM administrator <admin+scm@list.forge.codelutin.com>.
This is an automated email from the git hooks/post-receive script. New commit to branch develop in repository observe. See http://git.codelutin.com/observe.git commit 1facc96d0d6a64c022cb1e90f269c033d407b612 Author: Tony CHEMIT <chemit@codelutin.com> Date: Wed Aug 26 14:26:00 2015 +0200 Implantation de l'export du référentiel --- .../service/DataSourceDumpProducerService.java | 2 +- .../DataSourceDumpProducerServiceTopia.java | 106 ++++++++++++++++++++- .../DataSourceDumpProducerServiceTopiaTest.java | 52 ++++++++++ 3 files changed, 158 insertions(+), 2 deletions(-) diff --git a/observe-services-api/src/main/java/fr/ird/observe/services/service/DataSourceDumpProducerService.java b/observe-services-api/src/main/java/fr/ird/observe/services/service/DataSourceDumpProducerService.java index 163d414..816dd3d 100644 --- a/observe-services-api/src/main/java/fr/ird/observe/services/service/DataSourceDumpProducerService.java +++ b/observe-services-api/src/main/java/fr/ird/observe/services/service/DataSourceDumpProducerService.java @@ -12,5 +12,5 @@ public interface DataSourceDumpProducerService extends ObserveService{ byte[] getReferentialDump(); byte[] getDataDump(String... importDataIds); - + } diff --git a/observe-services-topia/src/main/java/fr/ird/observe/services/service/DataSourceDumpProducerServiceTopia.java b/observe-services-topia/src/main/java/fr/ird/observe/services/service/DataSourceDumpProducerServiceTopia.java index c4fb15f..62d3df0 100644 --- a/observe-services-topia/src/main/java/fr/ird/observe/services/service/DataSourceDumpProducerServiceTopia.java +++ b/observe-services-topia/src/main/java/fr/ird/observe/services/service/DataSourceDumpProducerServiceTopia.java @@ -1,6 +1,24 @@ package fr.ird.observe.services.service; +import fr.ird.observe.ObserveTopiaApplicationContext; +import fr.ird.observe.ObserveTopiaPersistenceContext; +import fr.ird.observe.entities.Entities; import fr.ird.observe.services.ObserveServiceTopia; +import fr.ird.observe.services.ObserveTopiaApplicationContextFactory; +import fr.ird.observe.services.configuration.ObserveDataSourceConfigurationTopiaH2; +import fr.ird.observe.services.dto.DataSourceCreateConfigurationDto; +import fr.ird.observe.services.dto.DataSourceCreateWithNoReferentialImportException; +import fr.ird.observe.services.dto.IncompatibleDataSourceCreateConfigurationException; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.nuiton.topia.replication.TopiaReplicationService; +import org.nuiton.topia.replication.model.ReplicationModel; +import org.nuiton.util.StringUtil; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; /** * Created on 23/08/15. @@ -9,9 +27,95 @@ import fr.ird.observe.services.ObserveServiceTopia; */ public class DataSourceDumpProducerServiceTopia extends ObserveServiceTopia implements DataSourceDumpProducerService { + /** Logger. */ + private static final Log log = LogFactory.getLog(DataSourceDumpProducerServiceTopia.class); + + public static final String EXPORT_SQL_STATEMENT = "SCRIPT NOPASSWORDS NOSETTINGS TO '%s' CHARSET 'UTF-8' SCHEMA OBSERVE_COMMON, OBSERVE_SEINE, OBSERVE_LONGLINE;"; + @Override public byte[] getReferentialDump() { - return new byte[0]; + + long t0 = now().getTime(); + + ObserveDataSourceConfigurationTopiaH2 temporaryDataSourceConfiguration = createTemporaryDatabase("referentialDump"); + + if (log.isInfoEnabled()) { + log.info(String.format("Referential temporary database created in %s", StringUtil.convertTime(t0, System.nanoTime()))); + } + + ObserveTopiaApplicationContext temporaryTopiaApplicationContext = ObserveTopiaApplicationContextFactory.getOrCreateTopiaApplicationContext(temporaryDataSourceConfiguration); + try { + + t0 = now().getTime(); + + TopiaReplicationService service = temporaryTopiaApplicationContext.getReplicationService(); + ReplicationModel model = service.prepareForAll(Entities.REFERENCE_ENTITIES); + + ObserveTopiaApplicationContext sourceTopiaApplicationContext = serviceContext.getTopiaApplicationContext(); + service.doReplicate(model, sourceTopiaApplicationContext); + + if (log.isInfoEnabled()) { + log.info(String.format("Referential replication done in %s", StringUtil.convertTime(t0, System.nanoTime()))); + } + + } catch (Exception e) { + throw new RuntimeException("Could not replicate referantial", e); + } + + Path dumpFile = exportToFile(temporaryDataSourceConfiguration, temporaryTopiaApplicationContext); + + try { + byte[] content = Files.readAllBytes(dumpFile); + return content; + } catch (IOException e) { + //TODO Avoir une exception concrete + throw new RuntimeException("Could not read file " + dumpFile + " content", e); + } + + } + + protected Path exportToFile(ObserveDataSourceConfigurationTopiaH2 temporaryDataSourceConfiguration, + ObserveTopiaApplicationContext temporaryTopiaApplicationContext) { + + Path dumpFile = temporaryDataSourceConfiguration.getDirectory().toPath().resolve("dump.sql"); + + if (log.isInfoEnabled()) { + log.info("Export sql to file: " + dumpFile); + } + try (ObserveTopiaPersistenceContext temporaryPersistenceContext = temporaryTopiaApplicationContext.newPersistenceContext()) { + + String scriptSqlQuery = String.format(EXPORT_SQL_STATEMENT, dumpFile); + + temporaryPersistenceContext.getSqlSupport().executeSql(scriptSqlQuery); + + } + return dumpFile; + + } + + protected ObserveDataSourceConfigurationTopiaH2 createTemporaryDatabase(String prefix) { + + ObserveDataSourceConfigurationTopiaH2 temporaryDataSourceConfiguration = new ObserveDataSourceConfigurationTopiaH2(); + temporaryDataSourceConfiguration.setDbName("obstuna"); + temporaryDataSourceConfiguration.setLabel(prefix); + temporaryDataSourceConfiguration.setUsername("sa"); + temporaryDataSourceConfiguration.setPassword('s', 'a'); + File temporaryDataSourceDirectory = serviceContext.createTemporaryDirectory(prefix); + temporaryDataSourceConfiguration.setDirectory(temporaryDataSourceDirectory); + + DataSourceService dataSourceService = serviceContext.newService(temporaryDataSourceConfiguration, DataSourceService.class); + + try { + DataSourceCreateConfigurationDto dataSourceCreateConfiguration = new DataSourceCreateConfigurationDto(); + dataSourceCreateConfiguration.setCanCreateEmptyDatabase(true); + dataSourceService.create(dataSourceCreateConfiguration); + } catch (IncompatibleDataSourceCreateConfigurationException | DataSourceCreateWithNoReferentialImportException e) { + //FIXME Avoir une exception plus concrete + throw new RuntimeException("Could not create temporary database: " + prefix, e); + } + + return temporaryDataSourceConfiguration; + } @Override diff --git a/observe-services-topia/src/test/java/fr/ird/observe/services/service/DataSourceDumpProducerServiceTopiaTest.java b/observe-services-topia/src/test/java/fr/ird/observe/services/service/DataSourceDumpProducerServiceTopiaTest.java new file mode 100644 index 0000000..2ab540c --- /dev/null +++ b/observe-services-topia/src/test/java/fr/ird/observe/services/service/DataSourceDumpProducerServiceTopiaTest.java @@ -0,0 +1,52 @@ +package fr.ird.observe.services.service; + +import com.google.common.collect.ImmutableSet; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import java.util.Set; + +/** + * Created on 26/08/15. + * + * @author Tony Chemit - chemit@codelutin.com + */ +public class DataSourceDumpProducerServiceTopiaTest extends AbstractServiceTopiaTest { + + protected DataSourceDumpProducerService service; + + @Override + public Set<String> getTestNamesChangeDataBase() { + return ImmutableSet.of(); + } + + @Override + public String getScriptName() { + return "referentiel"; + } + + @Before + public void setUp() throws Exception { + + super.setUp(); + service = newService(DataSourceDumpProducerService.class); + } + + @Test + public void testGetReferentialDump() throws Exception { + + byte[] referentialDump = service.getReferentialDump(); + Assert.assertNotNull(referentialDump); + + } + + @Test + public void testGetDataDump() throws Exception { + + byte[] referentialDump = service.getDataDump(); + Assert.assertNotNull(referentialDump); + + + } +} \ No newline at end of file -- To stop receiving notification emails like this one, please contact codelutin.com SCM administrator <admin+scm@list.forge.codelutin.com>.
This is an automated email from the git hooks/post-receive script. New commit to branch develop in repository observe. See http://git.codelutin.com/observe.git commit 9c7e8d75426d168644069d5a2ccfc6612e54a737 Author: Tony CHEMIT <chemit@codelutin.com> Date: Wed Aug 26 15:46:14 2015 +0200 Replication des référentiels et données ok \o/ (see #7476) --- .../ird/observe/entities/longline/TdrTopiaDao.java | 82 +++++++++ .../src/main/xmi/observe-common.properties | 6 +- .../DataSourceDumpProducerServiceTopia.java | 183 ++++++++++++++++++--- .../DataSourceDumpProducerServiceTopiaTest.java | 3 +- 4 files changed, 242 insertions(+), 32 deletions(-) diff --git a/observe-entities/src/main/java/fr/ird/observe/entities/longline/TdrTopiaDao.java b/observe-entities/src/main/java/fr/ird/observe/entities/longline/TdrTopiaDao.java new file mode 100644 index 0000000..b8e0036 --- /dev/null +++ b/observe-entities/src/main/java/fr/ird/observe/entities/longline/TdrTopiaDao.java @@ -0,0 +1,82 @@ +package fr.ird.observe.entities.longline; + +import com.google.common.collect.ArrayListMultimap; +import com.google.common.collect.Multimap; +import org.nuiton.topia.persistence.support.TopiaSqlQuery; + +import java.sql.Connection; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.util.List; +import java.util.Map; +import java.util.Set; + +public class TdrTopiaDao extends AbstractTdrTopiaDao<Tdr> { + + + public Multimap<String, String> getTdrIdsBySeineIds(TripLongline tripLongline) { + + Set<String> setLonglineIds = TripLonglines.getSetIdsWithTdr(tripLongline); + + Multimap<String, String> result = ArrayListMultimap.create(); + + GetTdrIdsQuery sqlQuery = new GetTdrIdsQuery(); + + for (String setLonglineId : setLonglineIds) { + + sqlQuery.setSetId(setLonglineId); + + List<String> tdrIds = topiaSqlSupport.findMultipleResult(sqlQuery); + + result.putAll(setLonglineId, tdrIds); + + } + return result; + + } + + public void applyTdrAssociationFix(Multimap<String, String> tdrAssociation) { + + String request = "\nUPDATE OBSERVE_LONGLINE.TDR SET SET = '%s' WHERE topiaid = '%s';"; + + StringBuilder builder = new StringBuilder(); + + if (tdrAssociation != null) { + for (Map.Entry<String, String> entry : tdrAssociation.entries()) { + String setLonglineId = entry.getKey(); + String tdrId = entry.getValue(); + builder.append(String.format(request, setLonglineId, tdrId)); + } + } + + topiaSqlSupport.executeSql(builder.toString()); + + } + + private static class GetTdrIdsQuery extends TopiaSqlQuery<String> { + + protected String setId; + + @Override + public PreparedStatement prepareQuery(Connection connection) throws SQLException { + String sql = "SELECT t.topiaId " + + "FROM OBSERVE_LONGLINE.TDR t " + + "WHERE t.SET = ?"; + PreparedStatement ps = connection.prepareStatement(sql); + ps.setString(1, setId); + return ps; + } + + @Override + public String prepareResult(ResultSet set) throws SQLException { + String result = set.getString(1); + return result; + } + + public void setSetId(String setId) { + this.setId = setId; + } + } + +} diff --git a/observe-entities/src/main/xmi/observe-common.properties b/observe-entities/src/main/xmi/observe-common.properties index 301c0e4..6ccdc4b 100644 --- a/observe-entities/src/main/xmi/observe-common.properties +++ b/observe-entities/src/main/xmi/observe-common.properties @@ -40,9 +40,9 @@ package.fr.ird.observe.entities.tagvalue.dbSchema=OBSERVE_COMMON ### Champ Commentaire en text ################################################# ############################################################################### -fr.ird.observe.entities.CommentableEntity.attribute.comment.tagValue.hibernateAttributeType.String=text -fr.ird.observe.entities.referentiel.Vessel.attribute.comment.tagValue.hibernateAttributeType.String=text -fr.ird.observe.entities.referentiel.Program.attribute.comment.tagValue.hibernateAttributeType.String=text +fr.ird.observe.entities.CommentableEntity.attribute.comment.tagValue.hibernateAttributeType=text +fr.ird.observe.entities.referentiel.Vessel.attribute.comment.tagValue.hibernateAttributeType=text +fr.ird.observe.entities.referentiel.Program.attribute.comment.tagValue.hibernateAttributeType=text ############################################################################### ### Champ Numeric (utilisation du type sql numeric) ########################### diff --git a/observe-services-topia/src/main/java/fr/ird/observe/services/service/DataSourceDumpProducerServiceTopia.java b/observe-services-topia/src/main/java/fr/ird/observe/services/service/DataSourceDumpProducerServiceTopia.java index 62d3df0..0913aac 100644 --- a/observe-services-topia/src/main/java/fr/ird/observe/services/service/DataSourceDumpProducerServiceTopia.java +++ b/observe-services-topia/src/main/java/fr/ird/observe/services/service/DataSourceDumpProducerServiceTopia.java @@ -1,8 +1,10 @@ package fr.ird.observe.services.service; +import com.google.common.collect.Multimap; import fr.ird.observe.ObserveTopiaApplicationContext; import fr.ird.observe.ObserveTopiaPersistenceContext; import fr.ird.observe.entities.Entities; +import fr.ird.observe.entities.longline.TripLongline; import fr.ird.observe.services.ObserveServiceTopia; import fr.ird.observe.services.ObserveTopiaApplicationContextFactory; import fr.ird.observe.services.configuration.ObserveDataSourceConfigurationTopiaH2; @@ -11,6 +13,7 @@ import fr.ird.observe.services.dto.DataSourceCreateWithNoReferentialImportExcept import fr.ird.observe.services.dto.IncompatibleDataSourceCreateConfigurationException; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; +import org.nuiton.topia.persistence.TopiaEntity; import org.nuiton.topia.replication.TopiaReplicationService; import org.nuiton.topia.replication.model.ReplicationModel; import org.nuiton.util.StringUtil; @@ -19,6 +22,10 @@ import java.io.File; import java.io.IOException; import java.nio.file.Files; import java.nio.file.Path; +import java.util.Arrays; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Set; /** * Created on 23/08/15. @@ -43,53 +50,95 @@ public class DataSourceDumpProducerServiceTopia extends ObserveServiceTopia impl log.info(String.format("Referential temporary database created in %s", StringUtil.convertTime(t0, System.nanoTime()))); } + ObserveTopiaApplicationContext sourceTopiaApplicationContext = serviceContext.getTopiaApplicationContext(); ObserveTopiaApplicationContext temporaryTopiaApplicationContext = ObserveTopiaApplicationContextFactory.getOrCreateTopiaApplicationContext(temporaryDataSourceConfiguration); - try { - t0 = now().getTime(); + replicateReferential(sourceTopiaApplicationContext, temporaryTopiaApplicationContext); - TopiaReplicationService service = temporaryTopiaApplicationContext.getReplicationService(); - ReplicationModel model = service.prepareForAll(Entities.REFERENCE_ENTITIES); + Path dumpFile = exportToFile(temporaryDataSourceConfiguration, temporaryTopiaApplicationContext); - ObserveTopiaApplicationContext sourceTopiaApplicationContext = serviceContext.getTopiaApplicationContext(); - service.doReplicate(model, sourceTopiaApplicationContext); + byte[] content = getBytes(dumpFile); + return content; - if (log.isInfoEnabled()) { - log.info(String.format("Referential replication done in %s", StringUtil.convertTime(t0, System.nanoTime()))); - } + } + + protected void replicateReferential(ObserveTopiaApplicationContext sourceTopiaApplicationContext, ObserveTopiaApplicationContext temporaryTopiaApplicationContext) { + long t0 = now().getTime(); + + TopiaReplicationService service = sourceTopiaApplicationContext.getReplicationService(); + ReplicationModel model = service.prepareForAll(Entities.REFERENCE_ENTITIES); + + try { + service.doReplicate(model, temporaryTopiaApplicationContext); } catch (Exception e) { + //TODO Avoir une exception concrete throw new RuntimeException("Could not replicate referantial", e); } - Path dumpFile = exportToFile(temporaryDataSourceConfiguration, temporaryTopiaApplicationContext); - try { - byte[] content = Files.readAllBytes(dumpFile); - return content; - } catch (IOException e) { - //TODO Avoir une exception concrete - throw new RuntimeException("Could not read file " + dumpFile + " content", e); + if (log.isInfoEnabled()) { + log.info(String.format("Referential replication done in %s", StringUtil.convertTime(t0, System.nanoTime()))); } } - protected Path exportToFile(ObserveDataSourceConfigurationTopiaH2 temporaryDataSourceConfiguration, - ObserveTopiaApplicationContext temporaryTopiaApplicationContext) { + @Override + public byte[] getDataDump(String... importDataIds) { - Path dumpFile = temporaryDataSourceConfiguration.getDirectory().toPath().resolve("dump.sql"); + Set<String> ids = new LinkedHashSet<>(); + + if (importDataIds.length == 0) { + + // cas limite où on exporte toutes les données + List<String> tripSeineIds = getTopiaPersistenceContext().getTripSeineDao().findAllIds(); + ids.addAll(tripSeineIds); + List<String> tripLonglineIds = getTopiaPersistenceContext().getTripLonglineDao().findAllIds(); + ids.addAll(tripLonglineIds); + } else { + + ids.addAll(Arrays.asList(importDataIds)); + } + + long t0 = now().getTime(); + + ObserveDataSourceConfigurationTopiaH2 temporaryDataSourceConfiguration = createTemporaryDatabase("referentialDump"); if (log.isInfoEnabled()) { - log.info("Export sql to file: " + dumpFile); + log.info(String.format("Data temporary database created in %s", StringUtil.convertTime(t0, System.nanoTime()))); } - try (ObserveTopiaPersistenceContext temporaryPersistenceContext = temporaryTopiaApplicationContext.newPersistenceContext()) { - String scriptSqlQuery = String.format(EXPORT_SQL_STATEMENT, dumpFile); + ObserveTopiaApplicationContext temporaryTopiaApplicationContext = ObserveTopiaApplicationContextFactory.getOrCreateTopiaApplicationContext(temporaryDataSourceConfiguration); + ObserveTopiaApplicationContext sourceTopiaApplicationContext = serviceContext.getTopiaApplicationContext(); - temporaryPersistenceContext.getSqlSupport().executeSql(scriptSqlQuery); + replicateReferential(sourceTopiaApplicationContext, temporaryTopiaApplicationContext); + try { + + t0 = now().getTime(); + + for (String id : ids) { + long t1 = now().getTime(); + + replicateOneData(sourceTopiaApplicationContext, temporaryTopiaApplicationContext, id); + + if (log.isInfoEnabled()) { + log.info(String.format("Data replication [%s] done in %s", id, StringUtil.convertTime(t1, System.nanoTime()))); + } + } + + if (log.isInfoEnabled()) { + log.info(String.format("Data replication for %s data done in %s", ids.size(), StringUtil.convertTime(t0, System.nanoTime()))); + } + + } catch (Exception e) { + throw new RuntimeException("Could not replicate data", e); } - return dumpFile; + + Path dumpFile = exportToFile(temporaryDataSourceConfiguration, temporaryTopiaApplicationContext); + + byte[] content = getBytes(dumpFile); + return content; } @@ -118,8 +167,88 @@ public class DataSourceDumpProducerServiceTopia extends ObserveServiceTopia impl } - @Override - public byte[] getDataDump(String... importDataIds) { - return new byte[0]; + protected Path exportToFile(ObserveDataSourceConfigurationTopiaH2 temporaryDataSourceConfiguration, + ObserveTopiaApplicationContext temporaryTopiaApplicationContext) { + + Path dumpFile = temporaryDataSourceConfiguration.getDirectory().toPath().resolve("dump.sql"); + + if (log.isInfoEnabled()) { + log.info("Export sql to file: " + dumpFile); + } + try (ObserveTopiaPersistenceContext temporaryPersistenceContext = temporaryTopiaApplicationContext.newPersistenceContext()) { + + String scriptSqlQuery = String.format(EXPORT_SQL_STATEMENT, dumpFile); + + temporaryPersistenceContext.getSqlSupport().executeSql(scriptSqlQuery); + + } + return dumpFile; + + } + + protected byte[] getBytes(Path dumpFile) { + try { + byte[] content = Files.readAllBytes(dumpFile); + return content; + } catch (IOException e) { + //TODO Avoir une exception concrete + throw new RuntimeException("Could not read file " + dumpFile + " content", e); + } } + + /** + * Duplication de l'unique donnée observateur depuis ce service vers le service donné. + * + * La duplication utilise une transaction dédiée afin de ne pas saturer le serveur + * et aussi une meilleure maitrise du rollback en cas d'une erreur. + * + * Voir http://forge.codelutin.com/issues/4837 + * + * @param id l'id de la donnée à répliquer + */ + protected void replicateOneData(ObserveTopiaApplicationContext sourceTopiaApplicationContext, ObserveTopiaApplicationContext temporaryTopiaApplicationContext, String id) { + + TopiaReplicationService service = sourceTopiaApplicationContext.getReplicationService(); + + ReplicationModel model = service.prepare(Entities.DATA_ENTITIES, false, id); + + // To fix missing tdr associations (see https://forge.codelutin.com/issues/6611) + Multimap<String, String> tdrAssociation = null; + + try (ObserveTopiaPersistenceContext sourcePersistenceContext = sourceTopiaApplicationContext.newPersistenceContext()) { + + TopiaEntity e = getTopiaPersistenceContext().findByTopiaId(id); + + if (e instanceof TripLongline) { + + // Grab tdr missing associations (see https://forge.codelutin.com/issues/6611) + if (log.isInfoEnabled()) { + log.info("Should keep SetLongline - Tdr association ids for: " + id); + } + + tdrAssociation = sourcePersistenceContext.getTdrDao().getTdrIdsBySeineIds((TripLongline) e); + + } + + } + + // do the replicate + try { + service.doReplicate(model, temporaryTopiaApplicationContext); + } catch (Exception e) { + throw new RuntimeException("Could not replicate data: " + id, e); + } + + if (tdrAssociation != null) { + + // Apply back tdr missing associations (see https://forge.codelutin.com/issues/6611) + try (ObserveTopiaPersistenceContext observeTopiaPersistenceContext = temporaryTopiaApplicationContext.newPersistenceContext()) { + + observeTopiaPersistenceContext.getTdrDao().applyTdrAssociationFix(tdrAssociation); + + } + + } + } + } diff --git a/observe-services-topia/src/test/java/fr/ird/observe/services/service/DataSourceDumpProducerServiceTopiaTest.java b/observe-services-topia/src/test/java/fr/ird/observe/services/service/DataSourceDumpProducerServiceTopiaTest.java index 2ab540c..4eb5c36 100644 --- a/observe-services-topia/src/test/java/fr/ird/observe/services/service/DataSourceDumpProducerServiceTopiaTest.java +++ b/observe-services-topia/src/test/java/fr/ird/observe/services/service/DataSourceDumpProducerServiceTopiaTest.java @@ -23,7 +23,7 @@ public class DataSourceDumpProducerServiceTopiaTest extends AbstractServiceTopia @Override public String getScriptName() { - return "referentiel"; + return "dataForTestSeine"; } @Before @@ -47,6 +47,5 @@ public class DataSourceDumpProducerServiceTopiaTest extends AbstractServiceTopia byte[] referentialDump = service.getDataDump(); Assert.assertNotNull(referentialDump); - } } \ No newline at end of file -- To stop receiving notification emails like this one, please contact codelutin.com SCM administrator <admin+scm@list.forge.codelutin.com>.
This is an automated email from the git hooks/post-receive script. New commit to branch develop in repository observe. See http://git.codelutin.com/observe.git commit a6cabb5d17de867a29584a85eaa5ac9e2e80f11e Merge: 428ea83 9c7e8d7 Author: Tony CHEMIT <chemit@codelutin.com> Date: Wed Aug 26 15:46:20 2015 +0200 Mettre en place le service qui permet de repliquer des données (termine #7476) Merge branch 'feature/7476' into develop .../ird/observe/entities/longline/TdrTopiaDao.java | 82 +++++++ .../src/main/xmi/observe-common.properties | 6 +- .../service/DataSourceDumpProducerService.java | 2 +- .../ird/observe/services/ObserveServiceTopia.java | 3 + .../DataSourceDumpProducerServiceTopia.java | 237 ++++++++++++++++++++- .../DataSourceDumpProducerServiceTopiaTest.java | 51 +++++ 6 files changed, 375 insertions(+), 6 deletions(-) -- To stop receiving notification emails like this one, please contact codelutin.com SCM administrator <admin+scm@list.forge.codelutin.com>.
participants (1)
-
codelutin.com scm