December 2011 - Nuiton-utils-commits - mailman.cloud.codelutin.com

r2256 - in trunk: nuiton-csv/src nuiton-csv/src/site nuiton-csv/src/site/apt src/site/apt
by tchemit＠users.nuiton.org Dec. 14, 2011

Dec. 14, 2011

Author: tchemit Date: 2011-12-14 08:38:25 +0100 (Wed, 14 Dec 2011) New Revision: 2256 Url: http://nuiton.org/repositories/revision/nuiton-utils/2256 Log: begin of doc for nuiton-csv Added: trunk/nuiton-csv/src/site/ trunk/nuiton-csv/src/site/apt/ trunk/nuiton-csv/src/site/apt/index.apt trunk/nuiton-csv/src/site/site_fr.xml Modified: trunk/src/site/apt/index.apt Added: trunk/nuiton-csv/src/site/apt/index.apt =================================================================== --- trunk/nuiton-csv/src/site/apt/index.apt (rev 0) +++ trunk/nuiton-csv/src/site/apt/index.apt 2011-12-14 07:38:25 UTC (rev 2256) @@ -0,0 +1,147 @@ +~~~ +~~ #%L +~~ Nuiton Utils :: Nuiton Csv +~~ +~~ $Id$ +~~ $HeadURL$ +~~ %% +~~ Copyright (C) 2011 CodeLutin +~~ %% +~~ This program is free software: you can redistribute it and/or modify +~~ it under the terms of the GNU Lesser General Public License as +~~ published by the Free Software Foundation, either version 3 of the +~~ License, or (at your option) any later version. +~~ +~~ This program is distributed in the hope that it will be useful, +~~ but WITHOUT ANY WARRANTY; without even the implied warranty of +~~ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +~~ GNU General Lesser Public License for more details. +~~ +~~ You should have received a copy of the GNU General Lesser Public +~~ License along with this program. If not, see +~~ <http://www.gnu.org/licenses/lgpl-3.0.html>. +~~ #L% +~~~ + ---- + Nuiton csv + ---- + ---- + 2011-12-14 + ---- + +Présentation + + nuiton-csv définit une api simple d'import-export au format csv. Cette api + permet de construire un modèle d'import (et/ou d'export) où les différentes + préocupations sont bien séparées. + + Pour un import, et pour chaque ligne à importer, on a trois choses à faire : + + - lire la donnée depuis la source d'entrée + + - convertir la donnée en objet + + - persister la donnée convertie dans un objet + + La classe <<org.nuiton.util.csv.Import>> permet simplement d'effectuer des + imports, une fois le modèle crée. + + Pour un export et pour chaque objet à persister dans le fichier csv, on a + aussi trois choses : + + - lire la donnée depuis l'objet à persister + + - convertir la donnée au format texte + + - persister la donnée convertie dans le flux de sortie + + La classe <<org.nuiton.util.csv.Export>> permet simplement d'effectuer un + export, une fois le modèle d'export crée et les données à persister récupérées. + +Api + +* org.nuiton.util.csv.ValueFormatter + + A faire. + +* org.nuiton.util.csv.ValueParser + + A faire. + +* org.nuiton.util.csv.ValueParserFormatter + + A faire. + +* org.nuiton.util.csv.ImportModel + + A faire. + +* org.nuiton.util.csv.ExportModel + + A faire. + +* org.nuiton.util.csv.ImportExportModel + + A faire. + +* org.nuiton.util.csv.ModelBuilder + + A faire. + +* org.nuiton.util.csv.Import + + A faire. + +* org.nuiton.util.csv.Export + + A faire. + +Exemple + +* Construire un modèle d'import/export + +-------------------------------------------------------------------------------- +A faire... +-------------------------------------------------------------------------------- + +* Importer des données + +-------------------------------------------------------------------------------- + +// création du modèle d'import csv +ImportModel<E> csvModel = null; + +// creation d'un importer à partir d'un modèle et d'un reader sur fichier csv +Import<E> importer = Import.newImport(csvModel, reader); + +try { + + // parcours des objets crées à partir de chaque ligne du fichier csv + for (E entity : importer) { + + // A vous :) + + } +} finally { + + // fermeture de l'importer (ne ferme pas le flux d'entrée) + importer.close(); +} +-------------------------------------------------------------------------------- + +* Exporter des données + +-------------------------------------------------------------------------------- + +// création du modèle d'export csv +ExportModel<E> csvModel = null; + +// les données à exporter +Iterable<E> datas = null; + +// création d'un exporter +Export<E> exporter = Export.newExport(csvModel, datas); + +// lancement de l'export vers le fichier +exporter.exportToFile(new File("output.csv")); +-------------------------------------------------------------------------------- Property changes on: trunk/nuiton-csv/src/site/apt/index.apt ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision HeadURL Added: svn:eol-style + native Added: trunk/nuiton-csv/src/site/site_fr.xml =================================================================== --- trunk/nuiton-csv/src/site/site_fr.xml (rev 0) +++ trunk/nuiton-csv/src/site/site_fr.xml 2011-12-14 07:38:25 UTC (rev 2256) @@ -0,0 +1,56 @@ +<?xml version="1.0" encoding="UTF-8"?> + + + + +<project name="${project.name}"> + + <bannerLeft> + <name>${project.name}</name> + <href>index.html</href> + </bannerLeft> + + <poweredBy> + <logo href="http://maven.apache.org" name="Maven" + img="images/logos/maven-feather.png"/> + </poweredBy> + + <body> + + <breadcrumbs> + <item name="${project.name}" href="http://maven-site.nuiton.org/nuiton-utils/nuiton-csv/index.html"/> + </breadcrumbs> + + <menu ref="modules"/> + + <menu name="Utilisateur"> + <item name="Accueil" href="index.html"/> + </menu> + + <menu ref="reports"/> + + </body> +</project> Property changes on: trunk/nuiton-csv/src/site/site_fr.xml ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision HeadURL Added: svn:eol-style + native Modified: trunk/src/site/apt/index.apt =================================================================== --- trunk/src/site/apt/index.apt 2011-12-11 18:07:30 UTC (rev 2255) +++ trunk/src/site/apt/index.apt 2011-12-14 07:38:25 UTC (rev 2256) @@ -63,3 +63,9 @@ permettant nottament de préciser des scopes (info, erreur, warning, ...) {{{./nuiton-validator}Plus d'infos}} + +Librairie Nuiton-csv + + Cette librairie propose une Api simple d'import export au format csv. + + {{{./nuiton-csv}Plus d'infos}}

1 0

r2255 - trunk/nuiton-csv/src/main/java/org/nuiton/util/csv
by tchemit＠users.nuiton.org Dec. 11, 2011

Dec. 11, 2011

Author: tchemit Date: 2011-12-11 19:07:30 +0100 (Sun, 11 Dec 2011) New Revision: 2255 Url: http://nuiton.org/repositories/revision/nuiton-utils/2255 Log: Add some usefull methods on Export + reformat Modified: trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Export.java trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Import.java Modified: trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Export.java =================================================================== --- trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Export.java 2011-12-09 10:55:25 UTC (rev 2254) +++ trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Export.java 2011-12-11 18:07:30 UTC (rev 2255) @@ -29,13 +29,36 @@ import org.apache.commons.logging.LogFactory; import org.nuiton.util.StringUtil; +import java.io.BufferedWriter; +import java.io.File; +import java.io.FileWriter; import java.io.InputStream; +import java.io.StringWriter; +import java.io.Writer; import java.util.Collection; import java.util.LinkedList; import java.util.List; /** - * TODO + * Object to realize a export from a {@link ExportModel} and some datas. + * <h2>Usefull static methods</h2> + * There is some usefull methods here to do operation in one line : + * <h3>To export to a file</h3> + * <pre> + * Export.exportToFile(model, data, file); + * </pre> + * <h3>To export to a writer</h3> + * <pre> + * Export.exportToWriter(model, data, writer); + * </pre> + * <h3>To export as a string</h3> + * <pre> + * String exportcontent = Export.exportToString(model, data); + * </pre> + * <h3>To obtain a new instance of an exporter</h3> + * <pre> + * Export<E> exporter = Export.newExport(model, data); + * </pre> * * @author bleny <leny(a)codelutin.com> * @author tchemit <chemit(a)codelutin.com> @@ -55,14 +78,42 @@ return new Export<E>(model, data); } + public static <E> void exportToFile(ExportModel<E> model, + Iterable<E> data, + File file) throws Exception { + Export<E> exporter = newExport(model, data); + exporter.exportToFile(file); + } + + public static <E> String exportToString(ExportModel<E> model, + Iterable<E> data) throws Exception { + Export<E> exporter = newExport(model, data); + return exporter.startExportAsString(); + } + + public static <E> void exportToWriter(ExportModel<E> model, + Iterable<E> data, + Writer writer) throws Exception { + Export<E> exporter = newExport(model, data); + exporter.startExport(writer); + } + protected Export(ExportModel<E> model, Iterable<E> data) { this.model = model; this.data = data; } - public String startExportAsString() throws Exception { + public void exportToFile(File file) throws Exception { + Writer writer = new BufferedWriter(new FileWriter(file)); + try { + startExport(writer); + } finally { + writer.close(); + } + } + + public void startExport(Writer writer) throws Exception { String separator = String.valueOf(model.getSeparator()); - StringBuilder csv = new StringBuilder(); // add headers List<String> headerNames = new LinkedList<String>(); @@ -70,12 +121,12 @@ headerNames.add(column.getHeaderName()); } String headersLine = StringUtil.join(headerNames, separator, true); - csv.append(headersLine).append('\n'); - + writer.write(headersLine); + writer.write('\n'); if (log.isDebugEnabled()) { log.debug("headers for export are '" + headersLine + "'"); if (data instanceof Collection) { - log.debug("will export " + ((Collection<E>)data).size() + " lines"); + log.debug("will export " + ((Collection<E>) data).size() + " lines"); } } @@ -90,17 +141,28 @@ if (formattedValue == null) { throw new NullPointerException( "column for header " + column.getHeaderName() + - " returned a null value." + column.toString()); + " returned a null value." + column.toString()); } formattedValue = StringUtil.escapeCsvValue(formattedValue, separator); - csv.append(formattedValue).append(separator); + writer.write(formattedValue); + writer.write(separator); } - csv.append('\n'); + writer.write('\n'); } - return csv.toString(); } + public String startExportAsString() throws Exception { + StringWriter writer = new StringWriter(); + try { + startExport(writer); + String result = writer.toString(); + return result; + } finally { + writer.close(); + } + } + public InputStream startExport() throws Exception { String content = startExportAsString(); return IOUtils.toInputStream(content); Modified: trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Import.java =================================================================== --- trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Import.java 2011-12-09 10:55:25 UTC (rev 2254) +++ trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Import.java 2011-12-11 18:07:30 UTC (rev 2255) @@ -96,7 +96,6 @@ * * @return the Iterator used for csv iteration * @see #prepareAndValidate() - * @see Iterable#iterator() */ @Override public Iterator<E> iterator() { @@ -135,10 +134,10 @@ // read value from csv cell String value = readValue(field, lineNumber); - + // contravariance ftw Object parsedValue = parseValue(field, lineNumber, value); - + // set value to element setValue(field, lineNumber, element, parsedValue); } @@ -166,7 +165,7 @@ * model. This will check if headers are unique, known by the model and * if mandatory headers exist in the file. During this phase, the model * will retrieve headers value with {@link - * ImportModel#pushCsvHeaderNames(java.util.List)} call. + * ImportModel#pushCsvHeaderNames(List)} call. * * @since 2.4.1 */ @@ -185,7 +184,8 @@ String[] headers = getHeaders(); if (log.isTraceEnabled()) { - log.trace("headers of the CSV file are : " + Arrays.toString(headers)); + log.trace("headers of the CSV file are : " + + Arrays.toString(headers)); } // hook to do some stuff from the model @@ -219,7 +219,7 @@ String value) { try { T parsedValue = field.parseValue(value); - return parsedValue; + return parsedValue; } catch (Exception e) { String message = _("csv.import.error.unableToParseValue", value, field.getHeaderName(), lineNumber) @@ -259,9 +259,10 @@ model.getColumnsForImport()) { validHeaderNames.add(importableColumn.getHeaderName()); } - String validationMessage = _("csv.import.error.unrecognizedHeaders", - StringUtil.join(csvHeaders, ", ", true), - StringUtil.join(validHeaderNames, ", ", true)); + String validationMessage = + _("csv.import.error.unrecognizedHeaders", + StringUtil.join(csvHeaders, ", ", true), + StringUtil.join(validHeaderNames, ", ", true)); throw new ImportRuntimeException(validationMessage); } } @@ -298,45 +299,37 @@ mandatoryHeadersNames.removeAll(csvHeaders); if (!mandatoryHeadersNames.isEmpty()) { - String validationMessage = _("csv.import.error.missingMandatoryHeaders", - StringUtil.join(mandatoryHeadersNames, ", ", true)); + String validationMessage = + _("csv.import.error.missingMandatoryHeaders", + StringUtil.join(mandatoryHeadersNames, ", ", true)); throw new ImportRuntimeException(validationMessage); } - - // XXX-fdesbois-2011-12-08 : this code is redundant with checkUniqueModelColumnNames() method -// Set<String> headerNames = new HashSet<String>(); -// for (ImportableColumn<E, ?> importableColumn : -// model.getColumnsForImport()) { -// String headerName = importableColumn.getHeaderName(); -// boolean alreadyUsed = !headerNames.add(headerName); -// if (alreadyUsed) { -// throw new ImportRuntimeException( -// "model contains multiple columnsForImport named '" + -// headerName + "'"); -// } -// } } protected String[] getHeaders() throws ImportRuntimeException { try { boolean canReadHeaders = reader.readHeaders(); if (!canReadHeaders) { - throw new ImportRuntimeException(_("csv.import.error.unableToReadHeaders")); + throw new ImportRuntimeException( + _("csv.import.error.unableToReadHeaders")); } } catch (IOException e) { - throw new ImportRuntimeException(_("csv.import.error.unableToReadHeaders"), e); + throw new ImportRuntimeException( + _("csv.import.error.unableToReadHeaders"), e); } try { String[] result = reader.getHeaders(); return result; } catch (IOException e) { - throw new ImportRuntimeException(_("csv.import.error.unableToReadHeaders"), e); + throw new ImportRuntimeException( + _("csv.import.error.unableToReadHeaders"), e); } } protected List<ImportableColumn<E, Object>> getNonIgnoredHeaders() { - List<ImportableColumn<E, Object>> nonIgnoredHeaders = new ArrayList<ImportableColumn<E, Object>>(); + List<ImportableColumn<E, Object>> nonIgnoredHeaders = + new ArrayList<ImportableColumn<E, Object>>(); for (ImportableColumn<E, Object> field : model.getColumnsForImport()) { if (!field.isIgnored()) { nonIgnoredHeaders.add(field); @@ -346,7 +339,8 @@ } protected List<ImportableColumn<E, ?>> getAllMandatoryHeaders() { - List<ImportableColumn<E, ?>> allMandatoryHeaders = new ArrayList<ImportableColumn<E, ?>>(); + List<ImportableColumn<E, ?>> allMandatoryHeaders = + new ArrayList<ImportableColumn<E, ?>>(); for (ImportableColumn<E, ?> field : model.getColumnsForImport()) { if (field.isMandatory()) { allMandatoryHeaders.add(field); @@ -355,8 +349,7 @@ return allMandatoryHeaders; } - protected Import(ImportModel<E> model, - InputStream inputStream) { + protected Import(ImportModel<E> model, InputStream inputStream) { if (inputStream == null) { throw new NullPointerException("inputStream is null"); } @@ -365,8 +358,7 @@ reader.setTrimWhitespace(true); } - protected Import(ImportModel<E> model, - Reader reader) { + protected Import(ImportModel<E> model, Reader reader) { if (reader == null) { throw new NullPointerException("reader is null"); } @@ -380,7 +372,7 @@ * was successfully read. * * @return {@code true} if line was successfully read, says in fact there is - * something after this line. + * something after this line. * @throws ImportRuntimeException if could not read line */ protected boolean readRow() throws ImportRuntimeException {

1 0

r2254 - in trunk/nuiton-csv/src: main/java/org/nuiton/util/csv test/java/org/nuiton/util/csv test/resources
by tchemit＠users.nuiton.org Dec. 9, 2011

Dec. 9, 2011

Author: tchemit Date: 2011-12-09 11:55:25 +0100 (Fri, 09 Dec 2011) New Revision: 2254 Url: http://nuiton.org/repositories/revision/nuiton-utils/2254 Log: - clean unique test (avoid use external test resources for a one snapshot) - use Iterable instead of Collection in that is called Model classes :) Removed: trunk/nuiton-csv/src/test/resources/org/ Modified: trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Export.java trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ExportModel.java trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ImportModel.java trunk/nuiton-csv/src/test/java/org/nuiton/util/csv/ImportTest.java Modified: trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Export.java =================================================================== --- trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Export.java 2011-12-08 10:06:54 UTC (rev 2253) +++ trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Export.java 2011-12-09 10:55:25 UTC (rev 2254) @@ -48,14 +48,14 @@ protected ExportModel<E> model; - protected Collection<E> data; + protected Iterable<E> data; public static <E> Export<E> newExport(ExportModel<E> model, - Collection<E> data) { + Iterable<E> data) { return new Export<E>(model, data); } - protected Export(ExportModel<E> model, Collection<E> data) { + protected Export(ExportModel<E> model, Iterable<E> data) { this.model = model; this.data = data; } @@ -74,10 +74,12 @@ if (log.isDebugEnabled()) { log.debug("headers for export are '" + headersLine + "'"); - log.debug("will export " + data.size() + " lines"); + if (data instanceof Collection) { + log.debug("will export " + ((Collection<E>)data).size() + " lines"); + } } - Collection<ExportableColumn<E, Object>> columnsForExport = + Iterable<ExportableColumn<E, Object>> columnsForExport = model.getColumnsForExport(); for (E object : data) { Modified: trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ExportModel.java =================================================================== --- trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ExportModel.java 2011-12-08 10:06:54 UTC (rev 2253) +++ trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ExportModel.java 2011-12-09 10:55:25 UTC (rev 2254) @@ -24,8 +24,6 @@ */ package org.nuiton.util.csv; -import java.util.Collection; - /** * TODO * @@ -37,6 +35,6 @@ char getSeparator(); - Collection<ExportableColumn<E, Object>> getColumnsForExport(); + Iterable<ExportableColumn<E, Object>> getColumnsForExport(); } Modified: trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ImportModel.java =================================================================== --- trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ImportModel.java 2011-12-08 10:06:54 UTC (rev 2253) +++ trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ImportModel.java 2011-12-09 10:55:25 UTC (rev 2254) @@ -24,7 +24,6 @@ */ package org.nuiton.util.csv; -import java.util.Collection; import java.util.List; /** @@ -42,5 +41,5 @@ E newEmptyInstance(); - Collection<ImportableColumn<E, Object>> getColumnsForImport(); + Iterable<ImportableColumn<E, Object>> getColumnsForImport(); } Modified: trunk/nuiton-csv/src/test/java/org/nuiton/util/csv/ImportTest.java =================================================================== --- trunk/nuiton-csv/src/test/java/org/nuiton/util/csv/ImportTest.java 2011-12-08 10:06:54 UTC (rev 2253) +++ trunk/nuiton-csv/src/test/java/org/nuiton/util/csv/ImportTest.java 2011-12-09 10:55:25 UTC (rev 2254) @@ -24,6 +24,7 @@ */ package org.nuiton.util.csv; +import org.apache.commons.lang3.time.DateUtils; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.junit.Assert; @@ -32,10 +33,9 @@ import org.nuiton.util.csv.Common.BeanProperty; import java.io.IOException; -import java.io.InputStream; -import java.text.DateFormat; +import java.io.Reader; +import java.io.StringReader; import java.text.ParseException; -import java.text.SimpleDateFormat; import java.util.ArrayList; import java.util.Collection; import java.util.Date; @@ -50,32 +50,27 @@ */ public class ImportTest { - private DateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd"); - /** - * Test with {@link SimpleImportModel} that directly implements necessary {@link ImportableColumn} - * with a empty file (says the file has only the header definition but no row - * - * @throws Exception for errors - */ @Test - public void testSimpleImportWithEmptyFile() throws Exception { - ImportModel<Row> model = new SimpleImportModel(dateFormat); - List<Row> rows = importFile(model, "emptyImportTest.csv"); + public void testSimpleImportWithNoData() throws Exception { + String content = "DATE;NUMBER;TITLE"; + + List<Row> rows; + rows = importContent(new SimpleImportModel(), content); Assert.assertEquals(0, rows.size()); + + content = "DATE;NUMBER;TITLE\n"; + rows = importContent(new SimpleImportModel(), content); + Assert.assertEquals(0, rows.size()); } - /** - * Test with {@link SimpleImportModel} that directly implements necessary {@link ImportableColumn} - * with a empty file (says the file has only the header definition but no row - * - * @throws Exception for errors - */ @Test - public void testSimpleImportWithEmptyFile2() throws Exception { - ImportModel<Row> model = new SimpleImportModel(dateFormat); - List<Row> rows = importFile(model, "emptyImport2Test.csv"); - Assert.assertEquals(0, rows.size()); + public void testSimpleImportWithOneLine() throws Exception { + String content = "DATE;NUMBER;TITLE\n2011-12-05;18;\"1ère ligne\""; + + List<Row> rows = importContent(new SimpleImportModel(), content); + Assert.assertEquals(1, rows.size()); + assertRowEquals(rows.get(0), DateUtil.createDate(5, 12, 2011), 18, "1ère ligne"); } /** @@ -85,9 +80,14 @@ */ @Test public void testSimpleImport() throws Exception { - ImportModel<Row> model = new SimpleImportModel(dateFormat); - List<Row> rows = importFile(model, "importTest.csv"); - checkImportTestResult(rows); + String content = "DATE;NUMBER;TITLE\n" + + "2011-12-05;18;\"1ère ligne\"\n" + + "2011-12-06;19;\"2ème ligne\"\n" + + "2011-12-07;21;\"3ème ligne\""; + List<Row> rows = importContent(new SimpleImportModel(), content); + assertRowEquals(rows.get(0), DateUtil.createDate(5, 12, 2011), 18, "1ère ligne"); + assertRowEquals(rows.get(1), DateUtil.createDate(6, 12, 2011), 19, "2ème ligne"); + assertRowEquals(rows.get(2), DateUtil.createDate(7, 12, 2011), 21, "3ème ligne"); } /** @@ -97,15 +97,22 @@ */ @Test public void testColumnImport() throws Exception { - ImportModel<Row> model = new ColumnImportModel(dateFormat); - List<Row> rows = importFile(model, "importTest.csv"); - checkImportTestResult(rows); + String content = "DATE;NUMBER;TITLE\n" + + "2011-12-05;18;\"1ère ligne\"\n" + + "2011-12-06;19;\"2ème ligne\"\n" + + "2011-12-07;21;\"3ème ligne\""; + List<Row> rows = importContent(new ColumnImportModel(), content); + assertRowEquals(rows.get(0), DateUtil.createDate(5, 12, 2011), 18, "1ère ligne"); + assertRowEquals(rows.get(1), DateUtil.createDate(6, 12, 2011), 19, "2ème ligne"); + assertRowEquals(rows.get(2), DateUtil.createDate(7, 12, 2011), 21, "3ème ligne"); + } - protected List<Row> importFile(ImportModel<Row> model, String filePath) throws IOException { - InputStream csvStream = getClass().getResourceAsStream(filePath); + protected List<Row> importContent(ImportModel<Row> model, + String content) throws IOException { + Reader reader = new StringReader(content); try { - Import<Row> rowImport = Import.newImport(model, csvStream); + Import<Row> rowImport = Import.newImport(model, reader); try { List<Row> result = new ArrayList<Row>(); for (Row row : rowImport) { @@ -116,25 +123,14 @@ rowImport.close(); } } finally { - csvStream.close(); + reader.close(); } } - private void checkImportTestResult(List<Row> rows) { - - Assert.assertEquals(3, rows.size()); - Row row1 = rows.get(0); - Assert.assertEquals(DateUtil.createDate(5, 12, 2011), row1.getDate()); - Assert.assertEquals(new Integer(18), row1.getNumber()); - Assert.assertEquals("1ère ligne", row1.getTitle()); - Row row2 = rows.get(1); - Assert.assertEquals(DateUtil.createDate(6, 12, 2011), row2.getDate()); - Assert.assertEquals(new Integer(19), row2.getNumber()); - Assert.assertEquals("2ème ligne", row2.getTitle()); - Row row3 = rows.get(2); - Assert.assertEquals(DateUtil.createDate(7, 12, 2011), row3.getDate()); - Assert.assertEquals(new Integer(21), row3.getNumber()); - Assert.assertEquals("3ème ligne", row3.getTitle()); + private void assertRowEquals(Row row, Date date, Integer number, String title) { + Assert.assertEquals(date, row.getDate()); + Assert.assertEquals(number, row.getNumber()); + Assert.assertEquals(title, row.getTitle()); } public static class Row { @@ -145,6 +141,15 @@ private Integer number; + public Row() { + } + + public Row(Date date, String title, Integer number) { + this.date = date; + this.title = title; + this.number = number; + } + public Date getDate() { return date; } @@ -168,17 +173,32 @@ public void setNumber(Integer number) { this.number = number; } - } - private static class ColumnImportModel implements ImportModel<Row> { + @Override + public boolean equals(Object o) { + if (this == o) return true; + if (!(o instanceof Row)) return false; - private DateFormat dateFormat; + Row row = (Row) o; - private ColumnImportModel(DateFormat dateFormat) { - this.dateFormat = dateFormat; + if (!date.equals(row.date)) return false; + if (!number.equals(row.number)) return false; + return title.equals(row.title); + } @Override + public int hashCode() { + int result = date.hashCode(); + result = 31 * result + title.hashCode(); + result = 31 * result + number.hashCode(); + return result; + } + } + + private static class ColumnImportModel implements ImportModel<Row> { + + @Override public char getSeparator() { return ';'; } @@ -242,7 +262,7 @@ @Override public Date parse(String value) throws ParseException { - return dateFormat.parse(value); + return DateUtils.parseDate(value, "yyyy-MM-dd"); } }; } @@ -252,12 +272,6 @@ private Log log = LogFactory.getLog(SimpleImportModel.class); - private DateFormat dateFormat; - - private SimpleImportModel(DateFormat dateFormat) { - this.dateFormat = dateFormat; - } - @Override public char getSeparator() { return ';'; @@ -337,7 +351,7 @@ @Override public Date parseValue(String value) throws ParseException { - return dateFormat.parse(value); + return DateUtils.parseDate(value, "yyyy-MM-dd"); } @Override

1 0

r2253 - trunk/nuiton-csv/src/main/java/org/nuiton/util/csv
by fdesbois＠users.nuiton.org Dec. 8, 2011

Dec. 8, 2011

Author: fdesbois Date: 2011-12-08 11:06:54 +0100 (Thu, 08 Dec 2011) New Revision: 2253 Url: http://nuiton.org/repositories/revision/nuiton-utils/2253 Log: #1846 : Little improvements Modified: trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Import.java Modified: trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Import.java =================================================================== --- trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Import.java 2011-12-08 10:04:13 UTC (rev 2252) +++ trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Import.java 2011-12-08 10:06:54 UTC (rev 2253) @@ -376,10 +376,10 @@ } /** - * Reads a next row from the reader and return {@code true} if line - * was sucessfull read. + * Read the next row from the reader and return {@code true} if line + * was successfully read. * - * @return {@code true} if line was sucessfull read, says in fact there is + * @return {@code true} if line was successfully read, says in fact there is * something after this line. * @throws ImportRuntimeException if could not read line */

1 0

r2252 - in trunk/nuiton-csv/src: main/java/org/nuiton/util/csv main/resources/i18n test/java/org/nuiton/util/csv
by fdesbois＠users.nuiton.org Dec. 8, 2011

Dec. 8, 2011

Author: fdesbois Date: 2011-12-08 11:04:13 +0100 (Thu, 08 Dec 2011) New Revision: 2252 Url: http://nuiton.org/repositories/revision/nuiton-utils/2252 Log: #1846 : Little improvements Modified: trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Import.java trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ImportToMap.java trunk/nuiton-csv/src/main/resources/i18n/nuiton-csv_en_GB.properties trunk/nuiton-csv/src/main/resources/i18n/nuiton-csv_es_ES.properties trunk/nuiton-csv/src/main/resources/i18n/nuiton-csv_fr_FR.properties trunk/nuiton-csv/src/test/java/org/nuiton/util/csv/ImportTest.java Modified: trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Import.java =================================================================== --- trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Import.java 2011-12-08 02:59:10 UTC (rev 2251) +++ trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Import.java 2011-12-08 10:04:13 UTC (rev 2252) @@ -84,10 +84,20 @@ return new Import<E>(model, inputStream); } - public static <E> Import<E> newImport(ImportModel<E> model, Reader reader) { + public static <E> Import<E> newImport(ImportModel<E> model, + Reader reader) { return new Import<E>(model, reader); } + /** + * Define iterator over import. First of all, the input stream will be + * validated based on defined model. Iteration will be done on all csv + * rows except first headers line. + * + * @return the Iterator used for csv iteration + * @see #prepareAndValidate() + * @see Iterable#iterator() + */ @Override public Iterator<E> iterator() { @@ -127,7 +137,7 @@ String value = readValue(field, lineNumber); // contravariance ftw - Object parsedValue = parserValue(field, lineNumber, value); + Object parsedValue = parseValue(field, lineNumber, value); // set value to element setValue(field, lineNumber, element, parsedValue); @@ -151,6 +161,15 @@ reader.close(); } + /** + * Read the first mandatory headers line and validate it with the input + * model. This will check if headers are unique, known by the model and + * if mandatory headers exist in the file. During this phase, the model + * will retrieve headers value with {@link + * ImportModel#pushCsvHeaderNames(java.util.List)} call. + * + * @since 2.4.1 + */ public void prepareAndValidate() { if (validate) { @@ -176,7 +195,6 @@ checkUniqueModelColumnNames(); // check that given headers from csv file are all known - checkHeaderNamesAreAllKnown(headers); // check all mandatories column are on csv header @@ -184,7 +202,7 @@ } protected <T> String readValue(ImportableColumn<E, T> field, - int lineNumber) { + int lineNumber) { try { String value = reader.get(field.getHeaderName()); return value; @@ -196,8 +214,9 @@ } } - protected <T> T parserValue(ImportableColumn<E, T> field, - int lineNumber, String value) { + protected <T> T parseValue(ImportableColumn<E, T> field, + int lineNumber, + String value) { try { T parsedValue = field.parseValue(value); return parsedValue; @@ -284,34 +303,35 @@ throw new ImportRuntimeException(validationMessage); } - Set<String> headerNames = new HashSet<String>(); - for (ImportableColumn<E, ?> importableColumn : - model.getColumnsForImport()) { - String headerName = importableColumn.getHeaderName(); - boolean alreadyUsed = !headerNames.add(headerName); - if (alreadyUsed) { - throw new ImportRuntimeException( - "model contains multiple columnsForImport named '" + - headerName + "'"); - } - } + // XXX-fdesbois-2011-12-08 : this code is redundant with checkUniqueModelColumnNames() method +// Set<String> headerNames = new HashSet<String>(); +// for (ImportableColumn<E, ?> importableColumn : +// model.getColumnsForImport()) { +// String headerName = importableColumn.getHeaderName(); +// boolean alreadyUsed = !headerNames.add(headerName); +// if (alreadyUsed) { +// throw new ImportRuntimeException( +// "model contains multiple columnsForImport named '" + +// headerName + "'"); +// } +// } } protected String[] getHeaders() throws ImportRuntimeException { try { boolean canReadHeaders = reader.readHeaders(); if (!canReadHeaders) { - throw new ImportRuntimeException("can't read headers"); + throw new ImportRuntimeException(_("csv.import.error.unableToReadHeaders")); } } catch (IOException e) { - throw new ImportRuntimeException("can't read headers"); + throw new ImportRuntimeException(_("csv.import.error.unableToReadHeaders"), e); } try { String[] result = reader.getHeaders(); return result; - } catch (IOException eee) { - throw new ImportRuntimeException("can't get headers", eee); + } catch (IOException e) { + throw new ImportRuntimeException(_("csv.import.error.unableToReadHeaders"), e); } } @@ -335,7 +355,8 @@ return allMandatoryHeaders; } - protected Import(ImportModel<E> model, InputStream inputStream) { + protected Import(ImportModel<E> model, + InputStream inputStream) { if (inputStream == null) { throw new NullPointerException("inputStream is null"); } @@ -344,7 +365,8 @@ reader.setTrimWhitespace(true); } - protected Import(ImportModel<E> model, Reader reader) { + protected Import(ImportModel<E> model, + Reader reader) { if (reader == null) { throw new NullPointerException("reader is null"); } @@ -365,9 +387,6 @@ try { boolean hasNext = reader.readRecord(); return hasNext; -// if (emptyFile) { -// throw new ImportRuntimeException("CSV file has no line"); -// } } catch (IOException e) { reader.close(); throw new ImportRuntimeException(_("csv.import.error.unableToReadLine", 1), e); Modified: trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ImportToMap.java =================================================================== --- trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ImportToMap.java 2011-12-08 02:59:10 UTC (rev 2251) +++ trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ImportToMap.java 2011-12-08 10:04:13 UTC (rev 2252) @@ -99,7 +99,7 @@ String value = readValue(field, lineNumber); // contravariance ftw - Object parsedValue = parserValue(field, lineNumber, value); + Object parsedValue = parseValue(field, lineNumber, value); // set value to element setValue(field, lineNumber, element, parsedValue); Modified: trunk/nuiton-csv/src/main/resources/i18n/nuiton-csv_en_GB.properties =================================================================== --- trunk/nuiton-csv/src/main/resources/i18n/nuiton-csv_en_GB.properties 2011-12-08 02:59:10 UTC (rev 2251) +++ trunk/nuiton-csv/src/main/resources/i18n/nuiton-csv_en_GB.properties 2011-12-08 10:04:13 UTC (rev 2252) @@ -2,6 +2,7 @@ csv.import.error.missingMandatoryHeaders=The mandatory fields %s are missing csv.import.error.unableToParseValue=Unable to parse value '%s' (column '%s', line %s) csv.import.error.unableToReadField=Unable to read value of column '%s' at line %s +csv.import.error.unableToReadHeaders=Unable to read headers csv.import.error.unableToReadLine=Unable to read line %s csv.import.error.unableToSetValue=Unable to set value '%s' (object '%s', line %s, column '%s') csv.import.error.unrecognizedHeaders=Fields %s are not recognized. Accepted fields are %s. Modified: trunk/nuiton-csv/src/main/resources/i18n/nuiton-csv_es_ES.properties =================================================================== --- trunk/nuiton-csv/src/main/resources/i18n/nuiton-csv_es_ES.properties 2011-12-08 02:59:10 UTC (rev 2251) +++ trunk/nuiton-csv/src/main/resources/i18n/nuiton-csv_es_ES.properties 2011-12-08 10:04:13 UTC (rev 2252) @@ -2,6 +2,7 @@ csv.import.error.missingMandatoryHeaders=The mandatory fields %s are missing csv.import.error.unableToParseValue=Unable to parse value '%s' (column '%s', line %s) csv.import.error.unableToReadField=Unable to read value of column '%s' at line %s +csv.import.error.unableToReadHeaders=Unable to read headers csv.import.error.unableToReadLine=Unable to read line %s csv.import.error.unableToSetValue=Unable to set value '%s' (object'%s', line %s, column '%s') csv.import.error.unrecognizedHeaders=Fields %s are not recognized. Accepted fields are %s. Modified: trunk/nuiton-csv/src/main/resources/i18n/nuiton-csv_fr_FR.properties =================================================================== --- trunk/nuiton-csv/src/main/resources/i18n/nuiton-csv_fr_FR.properties 2011-12-08 02:59:10 UTC (rev 2251) +++ trunk/nuiton-csv/src/main/resources/i18n/nuiton-csv_fr_FR.properties 2011-12-08 10:04:13 UTC (rev 2252) @@ -2,6 +2,7 @@ csv.import.error.missingMandatoryHeaders=Les champs obligatoires %s sont manquants csv.import.error.unableToParseValue=Erreur lors de l'interprétation de la valeur '%s' (colonne '%s', ligne %s) csv.import.error.unableToReadField=Impossible de lire la colonne '%s' à la ligne %s +csv.import.error.unableToReadHeaders=Impossible de lire les en-têtes de colonnes csv.import.error.unableToReadLine=Impossible de lire la ligne %s csv.import.error.unableToSetValue=Impossible d'enregistrer la valeur '%s' (objet '%s', ligne %s, column '%s') csv.import.error.unrecognizedHeaders=Les champs %s ne sont pas reconnus. Les champs possibles sont %s. Modified: trunk/nuiton-csv/src/test/java/org/nuiton/util/csv/ImportTest.java =================================================================== --- trunk/nuiton-csv/src/test/java/org/nuiton/util/csv/ImportTest.java 2011-12-08 02:59:10 UTC (rev 2251) +++ trunk/nuiton-csv/src/test/java/org/nuiton/util/csv/ImportTest.java 2011-12-08 10:04:13 UTC (rev 2252) @@ -39,7 +39,6 @@ import java.util.ArrayList; import java.util.Collection; import java.util.Date; -import java.util.Iterator; import java.util.List; /** @@ -108,10 +107,9 @@ try { Import<Row> rowImport = Import.newImport(model, csvStream); try { - Iterator<Row> it = rowImport.iterator(); List<Row> result = new ArrayList<Row>(); - while (it.hasNext()) { - result.add(it.next()); + for (Row row : rowImport) { + result.add(row); } return result; } finally {

1 0

r2251 - trunk/nuiton-utils/src/main/java/org/nuiton/util
by tchemit＠users.nuiton.org Dec. 8, 2011

Dec. 8, 2011

Author: tchemit Date: 2011-12-08 03:59:10 +0100 (Thu, 08 Dec 2011) New Revision: 2251 Url: http://nuiton.org/repositories/revision/nuiton-utils/2251 Log: fix some javadoc Modified: trunk/nuiton-utils/src/main/java/org/nuiton/util/ApplicationConfig.java Modified: trunk/nuiton-utils/src/main/java/org/nuiton/util/ApplicationConfig.java =================================================================== --- trunk/nuiton-utils/src/main/java/org/nuiton/util/ApplicationConfig.java 2011-12-07 19:51:32 UTC (rev 2250) +++ trunk/nuiton-utils/src/main/java/org/nuiton/util/ApplicationConfig.java 2011-12-08 02:59:10 UTC (rev 2251) @@ -678,6 +678,7 @@ * example: 0,1,5,6 * * @since 2.4 + * @return ordered action step number */ public List<Integer> getActionStep() { List<Integer> result = new ArrayList<Integer>(actions.keySet()); @@ -688,8 +689,6 @@ /** * Do all action in specified order step (first 0). * - * @param step do action only defined in this step - * * @throws IllegalAccessException if action invocation failed * @throws IllegalArgumentException if action invocation failed * @throws InvocationTargetException if action invocation failed

1 0

r2250 - in trunk/nuiton-csv/src: main/java/org/nuiton/util/csv test/java/org/nuiton/util/csv test/resources/org/nuiton/util/csv
by tchemit＠users.nuiton.org Dec. 7, 2011

Dec. 7, 2011

Author: tchemit Date: 2011-12-07 20:51:32 +0100 (Wed, 07 Dec 2011) New Revision: 2250 Url: http://nuiton.org/repositories/revision/nuiton-utils/2250 Log: Anomalie #1845: Can not import empty file Evolution #1846: Improve Import class code Added: trunk/nuiton-csv/src/test/resources/org/nuiton/util/csv/emptyImport2Test.csv trunk/nuiton-csv/src/test/resources/org/nuiton/util/csv/emptyImportTest.csv Modified: trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Import.java trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ImportToMap.java trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ModelBuilder.java trunk/nuiton-csv/src/test/java/org/nuiton/util/csv/ImportTest.java Modified: trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Import.java =================================================================== --- trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Import.java 2011-12-01 15:22:54 UTC (rev 2249) +++ trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/Import.java 2011-12-07 19:51:32 UTC (rev 2250) @@ -48,9 +48,9 @@ /** * Import engine for a given import model. - * + * <p/> * It acts as an {@link Iterable}, you can use directly inside a foreach. - * + * <p/> * The method {@link #prepareAndValidate()} will be invoked before all and * only once. It mainly obtain header from the csv input, pass it to the model * and then validate the model. @@ -84,8 +84,7 @@ return new Import<E>(model, inputStream); } - public static <E> Import<E> newImport(ImportModel<E> model, - Reader reader) { + public static <E> Import<E> newImport(ImportModel<E> model, Reader reader) { return new Import<E>(model, reader); } @@ -94,16 +93,17 @@ prepareAndValidate(); - readFirstLine(); - return new Iterator<E>() { - boolean hasNext = true; + // read first line since first line is header + boolean hasNext = readRow(); + // get once for all columns to import + List<ImportableColumn<E, Object>> columns = getNonIgnoredHeaders(); + + // to stock the current line number int lineNumber; - E lastElement; - @Override public boolean hasNext() { return hasNext; @@ -121,57 +121,22 @@ E element = model.newEmptyInstance(); - for (ImportableColumn<E, Object> field : getNonIgnoredHeaders()) { + for (ImportableColumn<E, Object> field : columns) { // read value from csv cell - String value; - try { - value = reader.get(field.getHeaderName()); - } catch (Exception e) { - reader.close(); - throw new ImportRuntimeException( - _("csv.import.error.unableToReadField", - field.getHeaderName(), lineNumber), e); - } - + String value = readValue(field, lineNumber); + // contravariance ftw - Object parsedValue; - try { - parsedValue = field.parseValue(value); - } catch (Exception e) { - String message = _("csv.import.error.unableToParseValue", - value, field.getHeaderName(), lineNumber) - + "\n" + e.getMessage(); - throw new ImportRuntimeException(message, e); - } - + Object parsedValue = parserValue(field, lineNumber, value); + // set value to element - try { - field.setValue(element, parsedValue); - } catch (Exception e) { - String message = _("csv.import.error.unableToSetValue", - parsedValue, - element.toString(), - lineNumber, field.getHeaderName()); - if (log.isErrorEnabled()) { - log.error(message); - } - throw new ImportRuntimeException(message, e); - } - - lastElement = element; + setValue(field, lineNumber, element, parsedValue); } - try { - hasNext = reader.readRecord(); - } catch (IOException e) { - reader.close(); - throw new ImportRuntimeException( - _("csv.import.error.unableToReadLine", - lineNumber + 1), e); - } + // check if there is a next row to read + hasNext = readRow(); - return lastElement; + return element; } @Override @@ -201,8 +166,7 @@ String[] headers = getHeaders(); if (log.isTraceEnabled()) { - log.trace("headers of the CSV file are : " + - Arrays.toString(headers)); + log.trace("headers of the CSV file are : " + Arrays.toString(headers)); } // hook to do some stuff from the model @@ -219,6 +183,50 @@ checkAllMandatoryHeadersArePresent(headers); } + protected <T> String readValue(ImportableColumn<E, T> field, + int lineNumber) { + try { + String value = reader.get(field.getHeaderName()); + return value; + } catch (Exception e) { + reader.close(); + throw new ImportRuntimeException( + _("csv.import.error.unableToReadField", + field.getHeaderName(), lineNumber), e); + } + } + + protected <T> T parserValue(ImportableColumn<E, T> field, + int lineNumber, String value) { + try { + T parsedValue = field.parseValue(value); + return parsedValue; + } catch (Exception e) { + String message = _("csv.import.error.unableToParseValue", + value, field.getHeaderName(), lineNumber) + + "\n" + e.getMessage(); + throw new ImportRuntimeException(message, e); + } + } + + protected <T> void setValue(ImportableColumn<E, T> field, + int lineNumber, + E element, + T parsedValue) { + try { + field.setValue(element, parsedValue); + } catch (Exception e) { + String message = _("csv.import.error.unableToSetValue", + parsedValue, + element.toString(), + lineNumber, field.getHeaderName()); + if (log.isErrorEnabled()) { + log.error(message); + } + throw new ImportRuntimeException(message, e); + } + } + protected void checkHeaderNamesAreAllKnown(String[] headers) { List<String> csvHeaders = new ArrayList<String>(); Collections.addAll(csvHeaders, headers); @@ -284,7 +292,7 @@ if (alreadyUsed) { throw new ImportRuntimeException( "model contains multiple columnsForImport named '" + - headerName + "'"); + headerName + "'"); } } } @@ -345,13 +353,21 @@ this.reader.setTrimWhitespace(true); } - protected void readFirstLine() throws ImportRuntimeException { + /** + * Reads a next row from the reader and return {@code true} if line + * was sucessfull read. + * + * @return {@code true} if line was sucessfull read, says in fact there is + * something after this line. + * @throws ImportRuntimeException if could not read line + */ + protected boolean readRow() throws ImportRuntimeException { try { - boolean emptyFile = !reader.readRecord(); - - if (emptyFile) { - throw new ImportRuntimeException("CSV file has no line"); - } + boolean hasNext = reader.readRecord(); + return hasNext; +// if (emptyFile) { +// throw new ImportRuntimeException("CSV file has no line"); +// } } catch (IOException e) { reader.close(); throw new ImportRuntimeException(_("csv.import.error.unableToReadLine", 1), e); Modified: trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ImportToMap.java =================================================================== --- trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ImportToMap.java 2011-12-01 15:22:54 UTC (rev 2249) +++ trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ImportToMap.java 2011-12-07 19:51:32 UTC (rev 2250) @@ -24,19 +24,14 @@ */ package org.nuiton.util.csv; -import org.apache.commons.logging.Log; -import org.apache.commons.logging.LogFactory; - -import java.io.IOException; import java.io.InputStream; import java.io.Reader; import java.util.HashMap; import java.util.Iterator; +import java.util.List; import java.util.Map; import java.util.NoSuchElementException; -import static org.nuiton.i18n.I18n._; - /** * A extended {@link Import} to read csv lines into a single map. * <p/> @@ -48,9 +43,6 @@ */ public class ImportToMap extends Import<Map<String, Object>> { - /** Logger. */ - private static final Log log = LogFactory.getLog(ImportToMap.class); - public static ImportToMap newImport(ImportModel<Map<String, Object>> model, InputStream inputStream) { return new ImportToMap(model, inputStream); @@ -67,16 +59,20 @@ // obtain headers from csv input and validate the model prepareAndValidate(); - // read first line since first line is header - readFirstLine(); - return new Iterator<Map<String, Object>>() { - boolean hasNext = true; + // read first line since first line is header + boolean hasNext = readRow(); + // get once for all columns to import + List<ImportableColumn<Map<String, Object>, Object>> columns = + getNonIgnoredHeaders(); + + // to stock the current line number int lineNumber; - Map<String, Object> element = new HashMap<String, Object>(); + // the map where to object of a row + final Map<String, Object> element = new HashMap<String, Object>(); @Override public boolean hasNext() { @@ -97,51 +93,20 @@ element.clear(); for (ImportableColumn<Map<String, Object>, Object> field : - getNonIgnoredHeaders()) { + columns) { // read value from csv cell - String value; - try { - value = reader.get(field.getHeaderName()); - } catch (Exception e) { - reader.close(); - throw new ImportRuntimeException(_("csv.import.error.unableToReadField", - field.getHeaderName(), lineNumber), e); - } + String value = readValue(field, lineNumber); // contravariance ftw - Object parsedValue; - try { - parsedValue = field.parseValue(value); - } catch (Exception e) { - String message = _("csv.import.error.unableToParseValue", - value, field.getHeaderName(), lineNumber) - + "\n" + e.getMessage(); - throw new ImportRuntimeException(message, e); - } + Object parsedValue = parserValue(field, lineNumber, value); // set value to element - try { - field.setValue(element, parsedValue); - } catch (Exception e) { - String message = _("csv.import.error.unableToSetValue", - parsedValue, - element.toString(), - lineNumber, field.getHeaderName()); - if (log.isErrorEnabled()) { - log.error(message); - } - throw new ImportRuntimeException(message, e); - } + setValue(field, lineNumber, element, parsedValue); } - try { - hasNext = reader.readRecord(); - } catch (IOException e) { - reader.close(); - throw new ImportRuntimeException( - _("csv.import.error.unableToReadLine", lineNumber + 1), e); - } + // check if there is a next row to read + hasNext = readRow(); return element; } Modified: trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ModelBuilder.java =================================================================== --- trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ModelBuilder.java 2011-12-01 15:22:54 UTC (rev 2249) +++ trunk/nuiton-csv/src/main/java/org/nuiton/util/csv/ModelBuilder.java 2011-12-07 19:51:32 UTC (rev 2250) @@ -28,8 +28,8 @@ import java.util.LinkedList; /** - * TODO * + * * @author bleny <leny(a)codelutin.com> * @author tchemit <chemit(a)codelutin.com> * @since 2.4 Modified: trunk/nuiton-csv/src/test/java/org/nuiton/util/csv/ImportTest.java =================================================================== --- trunk/nuiton-csv/src/test/java/org/nuiton/util/csv/ImportTest.java 2011-12-01 15:22:54 UTC (rev 2249) +++ trunk/nuiton-csv/src/test/java/org/nuiton/util/csv/ImportTest.java 2011-12-07 19:51:32 UTC (rev 2250) @@ -1,6 +1,9 @@ /* * #%L * Nuiton Utils :: Nuiton Csv + * + * $Id$ + * $HeadURL$ * %% * Copyright (C) 2011 CodeLutin * %% @@ -24,11 +27,11 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.junit.Assert; -import org.junit.Before; import org.junit.Test; import org.nuiton.util.DateUtil; import org.nuiton.util.csv.Common.BeanProperty; +import java.io.IOException; import java.io.InputStream; import java.text.DateFormat; import java.text.ParseException; @@ -42,25 +45,50 @@ /** * Created on 28/11/11 * - * @author fdesbois <florian.desbois(a)wiztivi.com> + * @author fdesbois <desbois(a)codelutin.com> + * @author tchemit <chemit(a)codelutin.com> + * @since 2.4 */ public class ImportTest { - private DateFormat dateFormat; + private DateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd"); - @Before - public void setUp() { - dateFormat = new SimpleDateFormat("yyyy-MM-dd"); + /** + * Test with {@link SimpleImportModel} that directly implements necessary {@link ImportableColumn} + * with a empty file (says the file has only the header definition but no row + * + * @throws Exception for errors + */ + @Test + public void testSimpleImportWithEmptyFile() throws Exception { + ImportModel<Row> model = new SimpleImportModel(dateFormat); + List<Row> rows = importFile(model, "emptyImportTest.csv"); + Assert.assertEquals(0, rows.size()); } /** + * Test with {@link SimpleImportModel} that directly implements necessary {@link ImportableColumn} + * with a empty file (says the file has only the header definition but no row + * + * @throws Exception for errors + */ + @Test + public void testSimpleImportWithEmptyFile2() throws Exception { + ImportModel<Row> model = new SimpleImportModel(dateFormat); + List<Row> rows = importFile(model, "emptyImport2Test.csv"); + Assert.assertEquals(0, rows.size()); + } + + /** * Test with {@link SimpleImportModel} that directly implements necessary {@link ImportableColumn}. * * @throws Exception for errors */ @Test public void testSimpleImport() throws Exception { - execute(new SimpleImportModel(dateFormat)); + ImportModel<Row> model = new SimpleImportModel(dateFormat); + List<Row> rows = importFile(model, "importTest.csv"); + checkImportTestResult(rows); } /** @@ -70,40 +98,45 @@ */ @Test public void testColumnImport() throws Exception { - execute(new ColumnImportModel(dateFormat)); + ImportModel<Row> model = new ColumnImportModel(dateFormat); + List<Row> rows = importFile(model, "importTest.csv"); + checkImportTestResult(rows); } - private void execute(ImportModel<Row> model) { - - InputStream csvStream = getClass().getResourceAsStream("importTest.csv"); - Import<Row> rowImport = Import.newImport(model, csvStream); + protected List<Row> importFile(ImportModel<Row> model, String filePath) throws IOException { + InputStream csvStream = getClass().getResourceAsStream(filePath); try { - Iterator<Row> it = rowImport.iterator(); - List<Row> rows = newList(it); - Assert.assertEquals(3, rows.size()); - Row row1 = rows.get(0); - Assert.assertEquals(DateUtil.createDate(5, 12, 2011), row1.getDate()); - Assert.assertEquals(new Integer(18), row1.getNumber()); - Assert.assertEquals("1ère ligne", row1.getTitle()); - Row row2 = rows.get(1); - Assert.assertEquals(DateUtil.createDate(6, 12, 2011), row2.getDate()); - Assert.assertEquals(new Integer(19), row2.getNumber()); - Assert.assertEquals("2ème ligne", row2.getTitle()); - Row row3 = rows.get(2); - Assert.assertEquals(DateUtil.createDate(7, 12, 2011), row3.getDate()); - Assert.assertEquals(new Integer(21), row3.getNumber()); - Assert.assertEquals("3ème ligne", row3.getTitle()); + Import<Row> rowImport = Import.newImport(model, csvStream); + try { + Iterator<Row> it = rowImport.iterator(); + List<Row> result = new ArrayList<Row>(); + while (it.hasNext()) { + result.add(it.next()); + } + return result; + } finally { + rowImport.close(); + } } finally { - rowImport.close(); + csvStream.close(); } } - private static <T> List<T> newList(Iterator<T> iterator) { - List<T> result = new ArrayList<T>(); - while (iterator.hasNext()) { - result.add(iterator.next()); - } - return result; + private void checkImportTestResult(List<Row> rows) { + + Assert.assertEquals(3, rows.size()); + Row row1 = rows.get(0); + Assert.assertEquals(DateUtil.createDate(5, 12, 2011), row1.getDate()); + Assert.assertEquals(new Integer(18), row1.getNumber()); + Assert.assertEquals("1ère ligne", row1.getTitle()); + Row row2 = rows.get(1); + Assert.assertEquals(DateUtil.createDate(6, 12, 2011), row2.getDate()); + Assert.assertEquals(new Integer(19), row2.getNumber()); + Assert.assertEquals("2ème ligne", row2.getTitle()); + Row row3 = rows.get(2); + Assert.assertEquals(DateUtil.createDate(7, 12, 2011), row3.getDate()); + Assert.assertEquals(new Integer(21), row3.getNumber()); + Assert.assertEquals("3ème ligne", row3.getTitle()); } public static class Row { @@ -234,7 +267,9 @@ @Override public void pushCsvHeaderNames(List<String> headerNames) { - log.info("Headers are : " + headerNames); + if (log.isDebugEnabled()) { + log.debug("Headers are : " + headerNames); + } } @Override Added: trunk/nuiton-csv/src/test/resources/org/nuiton/util/csv/emptyImport2Test.csv =================================================================== --- trunk/nuiton-csv/src/test/resources/org/nuiton/util/csv/emptyImport2Test.csv (rev 0) +++ trunk/nuiton-csv/src/test/resources/org/nuiton/util/csv/emptyImport2Test.csv 2011-12-07 19:51:32 UTC (rev 2250) @@ -0,0 +1 @@ +DATE;NUMBER;TITLE Property changes on: trunk/nuiton-csv/src/test/resources/org/nuiton/util/csv/emptyImport2Test.csv ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision HeadURL Added: svn:eol-style + native Added: trunk/nuiton-csv/src/test/resources/org/nuiton/util/csv/emptyImportTest.csv =================================================================== --- trunk/nuiton-csv/src/test/resources/org/nuiton/util/csv/emptyImportTest.csv (rev 0) +++ trunk/nuiton-csv/src/test/resources/org/nuiton/util/csv/emptyImportTest.csv 2011-12-07 19:51:32 UTC (rev 2250) @@ -0,0 +1 @@ +DATE;NUMBER;TITLE \ No newline at end of file Property changes on: trunk/nuiton-csv/src/test/resources/org/nuiton/util/csv/emptyImportTest.csv ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision HeadURL Added: svn:eol-style + native

1 0

r2249 - trunk/nuiton-utils/src/test/java/org/nuiton/util
by kmorin＠users.nuiton.org Dec. 1, 2011

Dec. 1, 2011

Author: kmorin Date: 2011-12-01 16:22:54 +0100 (Thu, 01 Dec 2011) New Revision: 2249 Url: http://nuiton.org/repositories/revision/nuiton-utils/2249 Log: debug test (was using Calendar.roll instead of Calendar.add) Modified: trunk/nuiton-utils/src/test/java/org/nuiton/util/DateUtilTest.java Modified: trunk/nuiton-utils/src/test/java/org/nuiton/util/DateUtilTest.java =================================================================== --- trunk/nuiton-utils/src/test/java/org/nuiton/util/DateUtilTest.java 2011-11-30 17:26:45 UTC (rev 2248) +++ trunk/nuiton-utils/src/test/java/org/nuiton/util/DateUtilTest.java 2011-12-01 15:22:54 UTC (rev 2249) @@ -262,8 +262,8 @@ log.info("getAge"); Calendar calendar = Calendar.getInstance(); - calendar.roll(Calendar.YEAR, -20); - calendar.roll(Calendar.DATE, -1); + calendar.add(Calendar.YEAR, -20); + calendar.add(Calendar.DAY_OF_YEAR, -1); int result = DateUtil.getAge(calendar.getTime()); log.info("result1 : " + result);

1 0