branch develop updated (d2cf910 -> 7ed66eb)
This is an automated email from the git hooks/post-receive script. New change to branch develop in repository coselmar. See http://git.codelutin.com/coselmar.git from d2cf910 fixes #6354 on user distinguish deletion and deactivation, and can enable user new 993a5ee prepare document indexation with lucene new d1d48e2 use indexation in document service for creation and search new b9b1dc9 add way to update and delete index information for document. Link delete with service new 81b2cdd add an indexation service for question : base on title, themes and summary new 37c0bed use question indexation in webservice new 7ed66eb Merge branch 'feature/lucene-indexation' into develop The 6 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "adds" were already present in the repository and have only been added to this reference. Detailed log of new commits: commit 7ed66eb28deff9e31ddb7f7ea3cbb038bd2be813 Merge: d2cf910 37c0bed Author: Yannick Martel <martel@©odelutin.com> Date: Wed Dec 31 11:10:51 2014 +0100 Merge branch 'feature/lucene-indexation' into develop commit 37c0bed1b33c6c4d3bc56ff3f88b5f7148548b88 Author: Yannick Martel <martel@©odelutin.com> Date: Wed Dec 31 10:51:30 2014 +0100 use question indexation in webservice commit 81b2cdd4f6adee991d74614788d8c38b28167587 Author: Yannick Martel <martel@©odelutin.com> Date: Tue Dec 30 17:22:46 2014 +0100 add an indexation service for question : base on title, themes and summary commit b9b1dc936cb43ed7a99f0a495ae8a135793a01af Author: Yannick Martel <martel@©odelutin.com> Date: Tue Dec 30 10:15:47 2014 +0100 add way to update and delete index information for document. Link delete with service commit d1d48e237301cfe0455a9fcd0f50233d3ddba571 Author: Yannick Martel <martel@©odelutin.com> Date: Mon Dec 29 17:02:59 2014 +0100 use indexation in document service for creation and search commit 993a5ee7906d1c815d4f52de3025c0c3fb232639 Author: Yannick Martel <martel@©odelutin.com> Date: Mon Dec 29 11:53:28 2014 +0100 prepare document indexation with lucene Summary of changes: coselmar-rest/pom.xml | 18 ++ .../coselmar/services/CoselmarServicesContext.java | 3 + .../services/CoselmarSimpleServiceSupport.java | 5 + .../services/DefaultCoselmarServicesContext.java | 11 + .../services/config/CoselmarServicesConfig.java | 9 + .../config/CoselmarServicesConfigOption.java | 6 + .../indexation/DocumentsIndexationService.java | 165 +++++++++++++ .../coselmar/services/indexation/LuceneUtils.java | 51 ++++ .../indexation/QuestionsIndexationService.java | 163 +++++++++++++ .../coselmar/services/v1/DocumentsWebService.java | 65 ++++- .../coselmar/services/v1/QuestionsWebService.java | 46 ++++ .../services/AbstractCoselmarServiceTest.java} | 15 +- .../indexation/DocumentsIndexationServiceTest.java | 208 ++++++++++++++++ .../indexation/QuestionsIndexationServiceTest.java | 270 +++++++++++++++++++++ pom.xml | 27 +++ 15 files changed, 1058 insertions(+), 4 deletions(-) create mode 100644 coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationService.java create mode 100644 coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/LuceneUtils.java create mode 100644 coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/QuestionsIndexationService.java copy coselmar-rest/src/{main/java/fr/ifremer/coselmar/services/CoselmarService.java => test/java/fr/ifremer/coselmar/services/AbstractCoselmarServiceTest.java} (66%) create mode 100644 coselmar-rest/src/test/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationServiceTest.java create mode 100644 coselmar-rest/src/test/java/fr/ifremer/coselmar/services/indexation/QuestionsIndexationServiceTest.java -- To stop receiving notification emails like this one, please contact codelutin.com SCM administrator <admin+scm@codelutin.com>.
This is an automated email from the git hooks/post-receive script. New commit to branch develop in repository coselmar. See http://git.codelutin.com/coselmar.git commit 993a5ee7906d1c815d4f52de3025c0c3fb232639 Author: Yannick Martel <martel@©odelutin.com> Date: Mon Dec 29 11:53:28 2014 +0100 prepare document indexation with lucene --- coselmar-rest/pom.xml | 18 +++ .../services/config/CoselmarServicesConfig.java | 9 ++ .../config/CoselmarServicesConfigOption.java | 6 + .../indexation/DocumentsIndexationService.java | 150 +++++++++++++++++++++ .../services/AbstractCoselmarServiceTest.java | 42 ++++++ .../services/DocumentsIndexationServiceTest.java | 103 ++++++++++++++ pom.xml | 27 ++++ 7 files changed, 355 insertions(+) diff --git a/coselmar-rest/pom.xml b/coselmar-rest/pom.xml index 9793232..ef22ee3 100644 --- a/coselmar-rest/pom.xml +++ b/coselmar-rest/pom.xml @@ -148,6 +148,24 @@ <scope>runtime</scope> </dependency> + <!-- Indexation - Lucence --> + <dependency> + <groupId>org.apache.lucene</groupId> + <artifactId>lucene-core</artifactId> + </dependency> + <dependency> + <groupId>org.apache.lucene</groupId> + <artifactId>lucene-queryparser</artifactId> + </dependency> + <dependency> + <groupId>org.apache.lucene</groupId> + <artifactId>lucene-queries</artifactId> + </dependency> + <dependency> + <groupId>org.apache.lucene</groupId> + <artifactId>lucene-analyzers-common</artifactId> + </dependency> + <!-- Others --> <dependency> <groupId>com.github.spullara.mustache.java</groupId> diff --git a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/config/CoselmarServicesConfig.java b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/config/CoselmarServicesConfig.java index 2ced2d9..81538ed 100644 --- a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/config/CoselmarServicesConfig.java +++ b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/config/CoselmarServicesConfig.java @@ -100,6 +100,15 @@ public class CoselmarServicesConfig { CoselmarServicesConfigOption.DATA_DIRECTORY.key); } + public File getIndexDirectory() { + File indexFile = applicationConfig.getOptionAsFile( + CoselmarServicesConfigOption.INDEX_DIRECTORY.key); + if (indexFile == null) { + indexFile = applicationConfig.getOptionAsFile(CoselmarServicesConfigOption.DATA_DIRECTORY.key); + } + return indexFile; + } + /** * @return Le nom d'hôte du serveur SMTP. */ diff --git a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/config/CoselmarServicesConfigOption.java b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/config/CoselmarServicesConfigOption.java index 720f302..c729e6b 100644 --- a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/config/CoselmarServicesConfigOption.java +++ b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/config/CoselmarServicesConfigOption.java @@ -40,6 +40,12 @@ public enum CoselmarServicesConfigOption implements ConfigOptionDef { "${java.io.tmpdir}/coselmar", File.class), + INDEX_DIRECTORY( + "coselmar.index.directory", + I18n.n("coselmar.configuration.index.directory"), + null, + File.class), + SMTP_HOST( "coselmar.smtp.host", "Nom d'hôte du serveur SMTP", diff --git a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationService.java b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationService.java new file mode 100644 index 0000000..3d5c573 --- /dev/null +++ b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationService.java @@ -0,0 +1,150 @@ +package fr.ifremer.coselmar.services.indexation; + +import java.io.File; +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Set; + +import fr.ifremer.coselmar.beans.DocumentBean; +import fr.ifremer.coselmar.services.CoselmarSimpleServiceSupport; +import org.apache.lucene.analysis.Analyzer; +import org.apache.lucene.analysis.core.SimpleAnalyzer; +import org.apache.lucene.document.Document; +import org.apache.lucene.document.Field; +import org.apache.lucene.document.StringField; +import org.apache.lucene.document.TextField; +import org.apache.lucene.index.DirectoryReader; +import org.apache.lucene.index.IndexWriter; +import org.apache.lucene.index.IndexWriterConfig; +import org.apache.lucene.index.Term; +import org.apache.lucene.queryparser.classic.ParseException; +import org.apache.lucene.search.BooleanClause; +import org.apache.lucene.search.BooleanQuery; +import org.apache.lucene.search.IndexSearcher; +import org.apache.lucene.search.PhraseQuery; +import org.apache.lucene.search.ScoreDoc; +import org.apache.lucene.search.TermQuery; +import org.apache.lucene.store.Directory; +import org.apache.lucene.store.NIOFSDirectory; +import org.apache.lucene.util.Version; + +/** + * This Services provides operation about {@link fr.ifremer.coselmar.persistence.entity.Document} + * or more exactly {@link fr.ifremer.coselmar.beans.DocumentBean} indexation : + * <ul> + * <li>registration of a document in the indexation db</li> + * <li>modification of a document in the indexation db</li> + * <li>documents search from the indexation db</li> + * </ul> + * + * The purpose is to use power of a indexation db (lucene) to increase search on + * document text field, and make easier fulltext search + * + * @author ymartel <martel@codelutin.com> + */ +public class DocumentsIndexationService extends CoselmarSimpleServiceSupport { + + public static Analyzer analyzer; + public static final IndexWriterConfig indexationConfig = new IndexWriterConfig(Version.LUCENE_4_10_2, getAnalyzer()); + public static IndexWriter indexWriter; + + protected static final String DOCUMENT_ID_INDEX_PROPERTY = "documentId"; + protected static final String DOCUMENT_NAME_INDEX_PROPERTY = "documentName"; + protected static final String DOCUMENT_AUTHORS_INDEX_PROPERTY = "documentAuthors"; + protected static final String DOCUMENT_SUMMARY_INDEX_PROPERTY = "documentSummary"; + protected static final String DOCUMENT_KEYWORD_INDEX_PROPERTY = "documentKeyword"; + protected static final String DOCUMENT_TYPE = "document"; + + public void indexDocument(DocumentBean document) throws IOException { + + Document doc = new Document(); + doc.add(new StringField(DOCUMENT_ID_INDEX_PROPERTY, document.getId(), Field.Store.YES)); + doc.add(new TextField(DOCUMENT_NAME_INDEX_PROPERTY, document.getName(), Field.Store.YES)); + doc.add(new TextField(DOCUMENT_AUTHORS_INDEX_PROPERTY, document.getAuthors(), Field.Store.YES)); + doc.add(new TextField(DOCUMENT_SUMMARY_INDEX_PROPERTY, document.getSummary(), Field.Store.YES)); + doc.add(new Field("type", DOCUMENT_TYPE, TextField.TYPE_STORED)); + + Set<String> keywords = document.getKeywords(); + for (String keyword : keywords) { + doc.add(new Field(DOCUMENT_KEYWORD_INDEX_PROPERTY, keyword, TextField.TYPE_STORED)); + } + + + getIndexWriter().addDocument(doc); + getIndexWriter().commit(); + + } + + public void updateDocument(DocumentBean document) { + //TODO + } + + public List<String> searchDocuments(String text) throws IOException, ParseException { + DirectoryReader ireader = DirectoryReader.open(getIndexWriter(), false); + IndexSearcher isearcher = new IndexSearcher(ireader); + + String[] words = text.split(" "); + + // Parse a simple query that searches for the "text": + BooleanQuery query = new BooleanQuery(); + + PhraseQuery nameQuery = new PhraseQuery(); + PhraseQuery summaryQuery = new PhraseQuery(); + PhraseQuery authorsQuery = new PhraseQuery(); + + for (String word : words) { + nameQuery.add(new Term(DOCUMENT_NAME_INDEX_PROPERTY, word.toLowerCase())); + summaryQuery.add(new Term(DOCUMENT_SUMMARY_INDEX_PROPERTY, word.toLowerCase())); + authorsQuery.add(new Term(DOCUMENT_AUTHORS_INDEX_PROPERTY, word.toLowerCase())); + } + + query.add(nameQuery, BooleanClause.Occur.SHOULD); + query.add(summaryQuery, BooleanClause.Occur.SHOULD); + query.add(authorsQuery, BooleanClause.Occur.SHOULD); + + query.add(new TermQuery(new Term(DOCUMENT_KEYWORD_INDEX_PROPERTY, text)), BooleanClause.Occur.SHOULD); + + + // Combine that with the type + BooleanQuery fullQuery = new BooleanQuery(); + fullQuery.add(query, BooleanClause.Occur.MUST); + fullQuery.add(new TermQuery(new Term("type", DOCUMENT_TYPE)), BooleanClause.Occur.MUST); + + ScoreDoc[] hits = isearcher.search(fullQuery, null, 1000).scoreDocs; + + List<String> documentIds = new ArrayList(hits.length); + + for (ScoreDoc hit : hits) { + Document doc = isearcher.doc(hit.doc); + String documentId = doc.get(DOCUMENT_ID_INDEX_PROPERTY); + documentIds.add(documentId); + } + + ireader.close(); + return documentIds; + } + + + protected static Analyzer getAnalyzer() { + if (analyzer == null) { +// analyzer = new StandardAnalyzer(); + //Use simple analyzer to index all words and be able to search with "close word" classified in StandardAnalyzer + analyzer = new SimpleAnalyzer(); + } + return analyzer; + + } + + protected IndexWriter getIndexWriter() throws IOException { + if (indexWriter == null) { + File indexDirectory = getCoselmarServicesConfig().getIndexDirectory(); + Directory index = NIOFSDirectory.open(indexDirectory); + + indexationConfig.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND); + indexWriter = new IndexWriter(index, indexationConfig); + } + return indexWriter; + } + +} diff --git a/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/AbstractCoselmarServiceTest.java b/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/AbstractCoselmarServiceTest.java new file mode 100644 index 0000000..e187c0a --- /dev/null +++ b/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/AbstractCoselmarServiceTest.java @@ -0,0 +1,42 @@ +package fr.ifremer.coselmar.services; + +/* + * #%L + * Coselmar :: Rest Services + * $Id:$ + * $HeadURL:$ + * %% + * Copyright (C) 2014 Ifremer, Code Lutin + * %% + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public + * License along with this program. If not, see + * <http://www.gnu.org/licenses/gpl-3.0.html>. + * #L% + */ + +import org.apache.commons.logging.Log; +import org.junit.Rule; + +import static org.apache.commons.logging.LogFactory.getLog; + +/** + * @author ymartel <martel@codelutin.com> + */ +public class AbstractCoselmarServiceTest { + + private static final Log log = getLog(AbstractCoselmarServiceTest.class); + + @Rule + public final FakeCoselmarApplicationContext application = new FakeCoselmarApplicationContext("coselmar-test.properties"); + +} diff --git a/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/DocumentsIndexationServiceTest.java b/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/DocumentsIndexationServiceTest.java new file mode 100644 index 0000000..c201e4c --- /dev/null +++ b/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/DocumentsIndexationServiceTest.java @@ -0,0 +1,103 @@ +package fr.ifremer.coselmar.services; + +import java.util.Date; +import java.util.List; +import java.util.Locale; + +import com.google.common.collect.Lists; +import fr.ifremer.coselmar.beans.DocumentBean; +import fr.ifremer.coselmar.persistence.entity.Privacy; +import fr.ifremer.coselmar.services.indexation.DocumentsIndexationService; +import org.junit.Assert; +import org.junit.Test; + +/** + * @author ymartel <martel@codelutin.com> + */ +public class DocumentsIndexationServiceTest extends AbstractCoselmarServiceTest { + + protected FakeCoselmarServicesContext serviceContext; + + protected FakeCoselmarServicesContext getServiceContext() { + + if (serviceContext == null) { + serviceContext = application.newServiceContext(application.newPersistenceContext(), Locale.FRANCE); + } + + return serviceContext; + } + + @Test + public void testAddDocument() throws Exception { + CoselmarServicesContext serviceContext = getServiceContext(); + DocumentsIndexationService documentsIndexationService = + serviceContext.newService(DocumentsIndexationService.class); + + DocumentBean documentOne = new DocumentBean("document1", + "Ceci n'est pas un document", "John Doe", Privacy.PUBLIC.name(), + new Date(), Lists.newArrayList("document", "test"), "testDocument", + "This is not a fake document used for test", "fr", null, "Jack, Jane", + null, null, false, null, "http://somewhere", "no comment"); + + documentsIndexationService.indexDocument(documentOne); + + } + + @Test + public void testSearchDocument() throws Exception { + populate(); + + CoselmarServicesContext serviceContext = getServiceContext(); + DocumentsIndexationService documentsIndexationService = + serviceContext.newService(DocumentsIndexationService.class); + + List<String> documentMatchingDoctorIds = documentsIndexationService.searchDocuments("doctor"); + Assert.assertEquals(1, documentMatchingDoctorIds.size()); + Assert.assertEquals("document3", documentMatchingDoctorIds.get(0)); + + List<String> documentMatchingAmyIds = documentsIndexationService.searchDocuments("amy"); + Assert.assertEquals(2, documentMatchingAmyIds.size()); + Assert.assertTrue(documentMatchingAmyIds.contains("document2")); + Assert.assertTrue(documentMatchingAmyIds.contains("document3")); + + List<String> documentMatchingTheMasterIds = documentsIndexationService.searchDocuments("the Master"); + Assert.assertTrue(documentMatchingTheMasterIds.isEmpty()); + + List<String> documentMatchingPartOfSummaryIds = documentsIndexationService.searchDocuments("This is part of"); + Assert.assertEquals(1, documentMatchingPartOfSummaryIds.size()); + Assert.assertEquals("document3", documentMatchingPartOfSummaryIds.get(0)); + + } + + public void populate() throws Exception { + CoselmarServicesContext serviceContext = getServiceContext(); + DocumentsIndexationService documentsIndexationService = + serviceContext.newService(DocumentsIndexationService.class); + + DocumentBean documentOne = new DocumentBean("document1", + "Ceci n'est pas un document", "John Doe", Privacy.PUBLIC.name(), + new Date(), Lists.newArrayList("document", "test"), "testDocument", + "This is not a fake document used for test", "fr", null, "Jack, Jane", + null, null, false, null, "http://somewhere", "no comment"); + + documentsIndexationService.indexDocument(documentOne); + + DocumentBean documentTwo = new DocumentBean("document2", + "Another document", "Amy Pond", Privacy.PUBLIC.name(), + new Date(), Lists.newArrayList("document", "test", "fish"), "testDocument", + "This is just an other document used for test", "fr", null, "Amy, Rory", + null, null, false, null, "http://somewhere", "no comment"); + + documentsIndexationService.indexDocument(documentTwo); + + DocumentBean documentThree = new DocumentBean("document3", + "Tardis documentation", "The Doctor", Privacy.PUBLIC.name(), + new Date(), Lists.newArrayList("tardis", "documentation", "old", "new", "borrowed", "blue"), "testDocument", + "This is part of documentation about the TARDIS", "fr", null, "The Doctor, Rose, Amy, River, Clara", + null, null, false, null, "http://tardis.wikia.com/wiki/TARDIS", "no comment"); + + documentsIndexationService.indexDocument(documentThree); + + } + +} diff --git a/pom.xml b/pom.xml index 4585df2..5f6a966 100644 --- a/pom.xml +++ b/pom.xml @@ -140,6 +140,8 @@ <postgresqlVersion>9.1-901-1.jdbc4</postgresqlVersion> <h2Version>1.4.178</h2Version> + <luceneVersion>4.10.2</luceneVersion> + <tomcatEmbedVersion>7.0.50</tomcatEmbedVersion> <angularUiSelectVersion>0.9.0</angularUiSelectVersion> @@ -247,6 +249,31 @@ <version>${h2Version}</version> </dependency> + <!-- Indexation - Lucence --> + <dependency> + <groupId>org.apache.lucene</groupId> + <artifactId>lucene-core</artifactId> + <version>${luceneVersion}</version> + </dependency> + + <dependency> + <groupId>org.apache.lucene</groupId> + <artifactId>lucene-queryparser</artifactId> + <version>${luceneVersion}</version> + </dependency> + + <dependency> + <groupId>org.apache.lucene</groupId> + <artifactId>lucene-queries</artifactId> + <version>${luceneVersion}</version> + </dependency> + + <dependency> + <groupId>org.apache.lucene</groupId> + <artifactId>lucene-analyzers-common</artifactId> + <version>${luceneVersion}</version> + </dependency> + <!-- Commons --> <dependency> <groupId>org.apache.commons</groupId> -- To stop receiving notification emails like this one, please contact codelutin.com SCM administrator <admin+scm@codelutin.com>.
This is an automated email from the git hooks/post-receive script. New commit to branch develop in repository coselmar. See http://git.codelutin.com/coselmar.git commit d1d48e237301cfe0455a9fcd0f50233d3ddba571 Author: Yannick Martel <martel@©odelutin.com> Date: Mon Dec 29 17:02:59 2014 +0100 use indexation in document service for creation and search --- .../indexation/DocumentsIndexationService.java | 1 + .../coselmar/services/v1/DocumentsWebService.java | 51 +++++++++++++++++++++- 2 files changed, 51 insertions(+), 1 deletion(-) diff --git a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationService.java b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationService.java index 3d5c573..7c166c7 100644 --- a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationService.java +++ b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationService.java @@ -60,6 +60,7 @@ public class DocumentsIndexationService extends CoselmarSimpleServiceSupport { Document doc = new Document(); doc.add(new StringField(DOCUMENT_ID_INDEX_PROPERTY, document.getId(), Field.Store.YES)); + doc.add(new TextField(DOCUMENT_NAME_INDEX_PROPERTY, document.getName(), Field.Store.YES)); doc.add(new TextField(DOCUMENT_AUTHORS_INDEX_PROPERTY, document.getAuthors(), Field.Store.YES)); doc.add(new TextField(DOCUMENT_SUMMARY_INDEX_PROPERTY, document.getSummary(), Field.Store.YES)); diff --git a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/v1/DocumentsWebService.java b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/v1/DocumentsWebService.java index 7bd42f7..0bad993 100644 --- a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/v1/DocumentsWebService.java +++ b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/v1/DocumentsWebService.java @@ -33,6 +33,8 @@ import java.util.ArrayList; import java.util.Date; import java.util.List; +import com.google.common.base.Function; +import com.google.common.base.Joiner; import com.google.common.base.Preconditions; import com.google.common.collect.Lists; import fr.ifremer.coselmar.beans.DocumentBean; @@ -46,9 +48,11 @@ import fr.ifremer.coselmar.services.CoselmarTechnicalException; import fr.ifremer.coselmar.services.CoselmarWebServiceSupport; import fr.ifremer.coselmar.services.errors.InvalidCredentialException; import fr.ifremer.coselmar.services.errors.UnauthorizedException; +import fr.ifremer.coselmar.services.indexation.DocumentsIndexationService; import org.apache.commons.io.FileUtils; import org.apache.commons.lang3.StringUtils; import org.apache.commons.logging.Log; +import org.apache.lucene.queryparser.classic.ParseException; import org.debux.webmotion.server.call.UploadFile; import org.debux.webmotion.server.render.Render; import org.nuiton.topia.persistence.TopiaNoResultException; @@ -117,7 +121,25 @@ public class DocumentsWebService extends CoselmarWebServiceSupport { // Admin and Supervisor can see all documents (public, private and restricted) if (Lists.newArrayList(CoselmarUserRole.ADMIN.name(), CoselmarUserRole.SUPERVISOR.name()).contains(currentUserRole)) { - documentList = findAllDocuments(searchKeywords); + if (searchKeywords != null && !searchKeywords.isEmpty()) { + DocumentsIndexationService documentsIndexationService = getServicesContext().newService(DocumentsIndexationService.class); + + try { + List<String> documentIds = documentsIndexationService.searchDocuments(Joiner.on(" ").join(searchKeywords)); + List<String> documentFullIds = getDocumentsFullId(documentIds); + + documentList = getDocumentDao().forTopiaIdIn(documentFullIds).findAll(); + + } catch (IOException | ParseException e) { + if (log.isErrorEnabled()) { + log.error("Unable to search by lucene, make search directly in database", e); + } + documentList = getDocumentDao().findAllContainingAllKeywords(searchKeywords); + + } + } else { + documentList = getDocumentDao().findAll(); + } } else { //Other can only see public, his own private and restricted for which he is allowed @@ -220,6 +242,20 @@ public class DocumentsWebService extends CoselmarWebServiceSupport { commit(); String lightId = getPersistenceContext().getTopiaIdFactory().getRandomPart(documentEntity.getTopiaId()); DocumentBean result = BeanEntityConverter.toBean(lightId, documentEntity); + + DocumentsIndexationService documentsIndexationService = getServicesContext().newService(DocumentsIndexationService.class); + try { + documentsIndexationService.indexDocument(result); + if (log.isDebugEnabled()) { + String message = String.format("Document '%s' added to index", documentName); + log.debug(message); + } + } catch (IOException e) { + if (log.isErrorEnabled()) { + log.error("Unable to index new document", e); + } + } + return result; } @@ -409,6 +445,19 @@ public class DocumentsWebService extends CoselmarWebServiceSupport { return Document.class.getCanonicalName() + getPersistenceContext().getTopiaIdFactory().getSeparator() + documentId; } + protected List<String> getDocumentsFullId(List<String> documentShortIds) { + + Function<String, String> getFullIds = new Function<String, String>() { + @Override + public String apply(String shortId) { + return getDocumentFullId(shortId); + } + }; + + List<String> fullIds = Lists.transform(documentShortIds, getFullIds); + return fullIds; + } + protected List<Document> findAllDocuments(List<String> searchKeywords) { List<Document> documentList; if (searchKeywords != null && !searchKeywords.isEmpty()) { -- To stop receiving notification emails like this one, please contact codelutin.com SCM administrator <admin+scm@codelutin.com>.
This is an automated email from the git hooks/post-receive script. New commit to branch develop in repository coselmar. See http://git.codelutin.com/coselmar.git commit b9b1dc936cb43ed7a99f0a495ae8a135793a01af Author: Yannick Martel <martel@©odelutin.com> Date: Tue Dec 30 10:15:47 2014 +0100 add way to update and delete index information for document. Link delete with service --- .../indexation/DocumentsIndexationService.java | 51 ++++- .../coselmar/services/v1/DocumentsWebService.java | 14 ++ .../services/DocumentsIndexationServiceTest.java | 103 ---------- .../indexation/DocumentsIndexationServiceTest.java | 208 +++++++++++++++++++++ 4 files changed, 269 insertions(+), 107 deletions(-) diff --git a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationService.java b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationService.java index 7c166c7..08a7c51 100644 --- a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationService.java +++ b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationService.java @@ -77,10 +77,6 @@ public class DocumentsIndexationService extends CoselmarSimpleServiceSupport { } - public void updateDocument(DocumentBean document) { - //TODO - } - public List<String> searchDocuments(String text) throws IOException, ParseException { DirectoryReader ireader = DirectoryReader.open(getIndexWriter(), false); IndexSearcher isearcher = new IndexSearcher(ireader); @@ -126,6 +122,53 @@ public class DocumentsIndexationService extends CoselmarSimpleServiceSupport { return documentIds; } + public void updateDocument(DocumentBean document) throws IOException { + DirectoryReader ireader = DirectoryReader.open(getIndexWriter(), false); + IndexSearcher isearcher = new IndexSearcher(ireader); + + // Retrieve document + BooleanQuery query = new BooleanQuery(); + query.add(new TermQuery(new Term(DOCUMENT_ID_INDEX_PROPERTY, document.getId())), BooleanClause.Occur.MUST); + query.add(new TermQuery(new Term("type", DOCUMENT_TYPE)), BooleanClause.Occur.MUST); + + ScoreDoc[] hits = isearcher.search(query, null, 1000).scoreDocs; + if (hits.length > 0) { + Document doc = isearcher.doc(hits[0].doc); + + doc.removeField(DOCUMENT_NAME_INDEX_PROPERTY); + doc.add(new TextField(DOCUMENT_NAME_INDEX_PROPERTY, document.getName(), Field.Store.YES)); + doc.removeField(DOCUMENT_AUTHORS_INDEX_PROPERTY); + doc.add(new TextField(DOCUMENT_AUTHORS_INDEX_PROPERTY, document.getAuthors(), Field.Store.YES)); + doc.removeField(DOCUMENT_SUMMARY_INDEX_PROPERTY); + doc.add(new TextField(DOCUMENT_SUMMARY_INDEX_PROPERTY, document.getSummary(), Field.Store.YES)); + + doc.removeFields(DOCUMENT_KEYWORD_INDEX_PROPERTY); + Set<String> keywords = document.getKeywords(); + for (String keyword : keywords) { + doc.add(new Field(DOCUMENT_KEYWORD_INDEX_PROPERTY, keyword, TextField.TYPE_STORED)); + } + + getIndexWriter().updateDocument(new Term(DOCUMENT_ID_INDEX_PROPERTY, document.getId()), doc); + getIndexWriter().commit(); + } + } + + public void deleteDocument(String documentId) throws IOException { + + // Retrieve document + BooleanQuery query = new BooleanQuery(); + query.add(new TermQuery(new Term(DOCUMENT_ID_INDEX_PROPERTY, documentId)), BooleanClause.Occur.MUST); + query.add(new TermQuery(new Term("type", DOCUMENT_TYPE)), BooleanClause.Occur.MUST); + + getIndexWriter().deleteDocuments(query); + getIndexWriter().commit(); + + } + + protected void cleanIndex() throws IOException { + getIndexWriter().deleteAll(); + getIndexWriter().commit(); + } protected static Analyzer getAnalyzer() { if (analyzer == null) { diff --git a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/v1/DocumentsWebService.java b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/v1/DocumentsWebService.java index 0bad993..e16667e 100644 --- a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/v1/DocumentsWebService.java +++ b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/v1/DocumentsWebService.java @@ -323,6 +323,20 @@ public class DocumentsWebService extends CoselmarWebServiceSupport { getDocumentDao().delete(document); commit(); + + //Remove index entry + DocumentsIndexationService documentsIndexationService = getServicesContext().newService(DocumentsIndexationService.class); + try { + documentsIndexationService.deleteDocument(documentId); + if (log.isDebugEnabled()) { + String message = String.format("Document '%s' removed from index", documentId); + log.debug(message); + } + } catch (IOException e) { + if (log.isErrorEnabled()) { + log.error("Unable to remove document entry from index", e); + } + } } public List<String> getKeywords() throws InvalidCredentialException, UnauthorizedException { diff --git a/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/DocumentsIndexationServiceTest.java b/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/DocumentsIndexationServiceTest.java deleted file mode 100644 index c201e4c..0000000 --- a/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/DocumentsIndexationServiceTest.java +++ /dev/null @@ -1,103 +0,0 @@ -package fr.ifremer.coselmar.services; - -import java.util.Date; -import java.util.List; -import java.util.Locale; - -import com.google.common.collect.Lists; -import fr.ifremer.coselmar.beans.DocumentBean; -import fr.ifremer.coselmar.persistence.entity.Privacy; -import fr.ifremer.coselmar.services.indexation.DocumentsIndexationService; -import org.junit.Assert; -import org.junit.Test; - -/** - * @author ymartel <martel@codelutin.com> - */ -public class DocumentsIndexationServiceTest extends AbstractCoselmarServiceTest { - - protected FakeCoselmarServicesContext serviceContext; - - protected FakeCoselmarServicesContext getServiceContext() { - - if (serviceContext == null) { - serviceContext = application.newServiceContext(application.newPersistenceContext(), Locale.FRANCE); - } - - return serviceContext; - } - - @Test - public void testAddDocument() throws Exception { - CoselmarServicesContext serviceContext = getServiceContext(); - DocumentsIndexationService documentsIndexationService = - serviceContext.newService(DocumentsIndexationService.class); - - DocumentBean documentOne = new DocumentBean("document1", - "Ceci n'est pas un document", "John Doe", Privacy.PUBLIC.name(), - new Date(), Lists.newArrayList("document", "test"), "testDocument", - "This is not a fake document used for test", "fr", null, "Jack, Jane", - null, null, false, null, "http://somewhere", "no comment"); - - documentsIndexationService.indexDocument(documentOne); - - } - - @Test - public void testSearchDocument() throws Exception { - populate(); - - CoselmarServicesContext serviceContext = getServiceContext(); - DocumentsIndexationService documentsIndexationService = - serviceContext.newService(DocumentsIndexationService.class); - - List<String> documentMatchingDoctorIds = documentsIndexationService.searchDocuments("doctor"); - Assert.assertEquals(1, documentMatchingDoctorIds.size()); - Assert.assertEquals("document3", documentMatchingDoctorIds.get(0)); - - List<String> documentMatchingAmyIds = documentsIndexationService.searchDocuments("amy"); - Assert.assertEquals(2, documentMatchingAmyIds.size()); - Assert.assertTrue(documentMatchingAmyIds.contains("document2")); - Assert.assertTrue(documentMatchingAmyIds.contains("document3")); - - List<String> documentMatchingTheMasterIds = documentsIndexationService.searchDocuments("the Master"); - Assert.assertTrue(documentMatchingTheMasterIds.isEmpty()); - - List<String> documentMatchingPartOfSummaryIds = documentsIndexationService.searchDocuments("This is part of"); - Assert.assertEquals(1, documentMatchingPartOfSummaryIds.size()); - Assert.assertEquals("document3", documentMatchingPartOfSummaryIds.get(0)); - - } - - public void populate() throws Exception { - CoselmarServicesContext serviceContext = getServiceContext(); - DocumentsIndexationService documentsIndexationService = - serviceContext.newService(DocumentsIndexationService.class); - - DocumentBean documentOne = new DocumentBean("document1", - "Ceci n'est pas un document", "John Doe", Privacy.PUBLIC.name(), - new Date(), Lists.newArrayList("document", "test"), "testDocument", - "This is not a fake document used for test", "fr", null, "Jack, Jane", - null, null, false, null, "http://somewhere", "no comment"); - - documentsIndexationService.indexDocument(documentOne); - - DocumentBean documentTwo = new DocumentBean("document2", - "Another document", "Amy Pond", Privacy.PUBLIC.name(), - new Date(), Lists.newArrayList("document", "test", "fish"), "testDocument", - "This is just an other document used for test", "fr", null, "Amy, Rory", - null, null, false, null, "http://somewhere", "no comment"); - - documentsIndexationService.indexDocument(documentTwo); - - DocumentBean documentThree = new DocumentBean("document3", - "Tardis documentation", "The Doctor", Privacy.PUBLIC.name(), - new Date(), Lists.newArrayList("tardis", "documentation", "old", "new", "borrowed", "blue"), "testDocument", - "This is part of documentation about the TARDIS", "fr", null, "The Doctor, Rose, Amy, River, Clara", - null, null, false, null, "http://tardis.wikia.com/wiki/TARDIS", "no comment"); - - documentsIndexationService.indexDocument(documentThree); - - } - -} diff --git a/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationServiceTest.java b/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationServiceTest.java new file mode 100644 index 0000000..7b667a3 --- /dev/null +++ b/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationServiceTest.java @@ -0,0 +1,208 @@ +package fr.ifremer.coselmar.services.indexation; + +import java.util.Date; +import java.util.List; +import java.util.Locale; + +import com.google.common.collect.Lists; +import fr.ifremer.coselmar.beans.DocumentBean; +import fr.ifremer.coselmar.persistence.entity.Privacy; +import fr.ifremer.coselmar.services.AbstractCoselmarServiceTest; +import fr.ifremer.coselmar.services.CoselmarServicesContext; +import fr.ifremer.coselmar.services.FakeCoselmarServicesContext; +import org.junit.After; +import org.junit.Assert; +import org.junit.Test; + +/** + * @author ymartel <martel@codelutin.com> + */ +public class DocumentsIndexationServiceTest extends AbstractCoselmarServiceTest { + +// protected FakeCoselmarServicesContext serviceContext; + + protected FakeCoselmarServicesContext getServiceContext() { + +// if (serviceContext == null) { + FakeCoselmarServicesContext serviceContext = application.newServiceContext(application.newPersistenceContext(), Locale.FRANCE); +// } + + return serviceContext; + } + + @Test + public void testAddDocument() throws Exception { + CoselmarServicesContext serviceContext = getServiceContext(); + DocumentsIndexationService documentsIndexationService = + serviceContext.newService(DocumentsIndexationService.class); + + DocumentBean documentOne = new DocumentBean("document1", + "Ceci n'est pas un document", "John Doe", Privacy.PUBLIC.name(), + new Date(), Lists.newArrayList("document", "test"), "testDocument", + "This is not a fake document used for test", "fr", null, "Jack, Jane", + null, null, false, null, "http://somewhere", "no comment"); + + documentsIndexationService.indexDocument(documentOne); + + } + + @Test + public void testSearchDocument() throws Exception { + populate(); + + CoselmarServicesContext serviceContext = getServiceContext(); + DocumentsIndexationService documentsIndexationService = + serviceContext.newService(DocumentsIndexationService.class); + + List<String> documentMatchingDoctorIds = documentsIndexationService.searchDocuments("doctor"); + Assert.assertEquals(1, documentMatchingDoctorIds.size()); + Assert.assertEquals("document3", documentMatchingDoctorIds.get(0)); + + List<String> documentMatchingAmyIds = documentsIndexationService.searchDocuments("amy"); + Assert.assertEquals(2, documentMatchingAmyIds.size()); + Assert.assertTrue(documentMatchingAmyIds.contains("document2")); + Assert.assertTrue(documentMatchingAmyIds.contains("document3")); + + List<String> documentMatchingTheMasterIds = documentsIndexationService.searchDocuments("the Master"); + Assert.assertTrue(documentMatchingTheMasterIds.isEmpty()); + + List<String> documentMatchingPartOfSummaryIds = documentsIndexationService.searchDocuments("This is part of"); + Assert.assertEquals(1, documentMatchingPartOfSummaryIds.size()); + Assert.assertEquals("document3", documentMatchingPartOfSummaryIds.get(0)); + + } + + @Test + public void testUpdateDocument() throws Exception { + + CoselmarServicesContext serviceContext = getServiceContext(); + DocumentsIndexationService documentsIndexationService = + serviceContext.newService(DocumentsIndexationService.class); + + DocumentBean documentOne = new DocumentBean("document1", + "Ceci n'est pas un document", "John Doe", Privacy.PUBLIC.name(), + new Date(), Lists.newArrayList("document", "test"), "testDocument", + "This is not a fake document used for test", "fr", null, "Jack, Jane", + null, null, false, null, "http://somewhere", "no comment"); + + documentsIndexationService.indexDocument(documentOne); + + List<String> documentMatchingDocumentIds = documentsIndexationService.searchDocuments("document"); + Assert.assertEquals(1, documentMatchingDocumentIds.size()); + Assert.assertEquals("document1", documentMatchingDocumentIds.get(0)); + + List<String> documentMatchingJackIds = documentsIndexationService.searchDocuments("jack"); + Assert.assertEquals(1, documentMatchingJackIds.size()); + Assert.assertTrue(documentMatchingJackIds.contains("document1")); + + List<String> documentMatchingTheMasterIds = documentsIndexationService.searchDocuments("James"); + Assert.assertTrue(documentMatchingTheMasterIds.isEmpty()); + + List<String> documentMatchingPartOfSummaryIds = documentsIndexationService.searchDocuments("This is not a fake document"); + Assert.assertEquals(1, documentMatchingPartOfSummaryIds.size()); + Assert.assertEquals("document1", documentMatchingPartOfSummaryIds.get(0)); + + documentOne = new DocumentBean("document1", + "Doc Premier", "John Doe", Privacy.PUBLIC.name(), + new Date(), Lists.newArrayList("test", "update"), "testDocument", + "This is a fake doc updated for test", "fr", null, "James, JJ", + null, null, false, null, "http://somewhere", "no comment"); + + documentsIndexationService.updateDocument(documentOne); + + documentMatchingDocumentIds = documentsIndexationService.searchDocuments("document"); + Assert.assertTrue(documentMatchingDocumentIds.isEmpty()); + + documentMatchingJackIds = documentsIndexationService.searchDocuments("jack"); + Assert.assertTrue(documentMatchingJackIds.isEmpty()); + + documentMatchingTheMasterIds = documentsIndexationService.searchDocuments("James"); + Assert.assertEquals(1, documentMatchingTheMasterIds.size()); + Assert.assertEquals("document1", documentMatchingTheMasterIds.get(0)); + + documentMatchingPartOfSummaryIds = documentsIndexationService.searchDocuments("This is not a fake document"); + Assert.assertTrue(documentMatchingPartOfSummaryIds.isEmpty()); + + documentMatchingPartOfSummaryIds = documentsIndexationService.searchDocuments("This is a fake doc"); + Assert.assertEquals(1, documentMatchingPartOfSummaryIds.size()); + Assert.assertEquals("document1", documentMatchingPartOfSummaryIds.get(0)); + + List<String> documentMatchingKeywordIds = documentsIndexationService.searchDocuments("update"); + Assert.assertEquals(1, documentMatchingKeywordIds.size()); + Assert.assertEquals("document1", documentMatchingKeywordIds.get(0)); + + List<String> documentMatchingPremierIds = documentsIndexationService.searchDocuments("premier"); + Assert.assertEquals(1, documentMatchingPremierIds.size()); + Assert.assertEquals("document1", documentMatchingPremierIds.get(0)); + + } + + @Test + public void testDeleteDocument() throws Exception { + populate(); + + CoselmarServicesContext serviceContext = getServiceContext(); + DocumentsIndexationService documentsIndexationService = + serviceContext.newService(DocumentsIndexationService.class); + + //Delete document2 + documentsIndexationService.deleteDocument("document2"); + + List<String> documentMatchingDoctorIds = documentsIndexationService.searchDocuments("doctor"); + Assert.assertEquals(1, documentMatchingDoctorIds.size()); + Assert.assertEquals("document3", documentMatchingDoctorIds.get(0)); + + List<String> documentMatchingAmyIds = documentsIndexationService.searchDocuments("amy"); + Assert.assertEquals(1, documentMatchingAmyIds.size()); + Assert.assertTrue(documentMatchingAmyIds.contains("document3")); + + List<String> documentMatchingTheMasterIds = documentsIndexationService.searchDocuments("Another"); + Assert.assertTrue(documentMatchingTheMasterIds.isEmpty()); + + List<String> documentMatchingPartOfSummaryIds = documentsIndexationService.searchDocuments("This is part of"); + Assert.assertEquals(1, documentMatchingPartOfSummaryIds.size()); + Assert.assertEquals("document3", documentMatchingPartOfSummaryIds.get(0)); + + + } + + public void populate() throws Exception { + CoselmarServicesContext serviceContext = getServiceContext(); + DocumentsIndexationService documentsIndexationService = + serviceContext.newService(DocumentsIndexationService.class); + + DocumentBean documentOne = new DocumentBean("document1", + "Ceci n'est pas un document", "John Doe", Privacy.PUBLIC.name(), + new Date(), Lists.newArrayList("document", "test"), "testDocument", + "This is not a fake document used for test", "fr", null, "Jack, Jane", + null, null, false, null, "http://somewhere", "no comment"); + + documentsIndexationService.indexDocument(documentOne); + + DocumentBean documentTwo = new DocumentBean("document2", + "Another document", "Amy Pond", Privacy.PUBLIC.name(), + new Date(), Lists.newArrayList("document", "test", "fish"), "testDocument", + "This is just an other document used for test", "fr", null, "Amy, Rory", + null, null, false, null, "http://somewhere", "no comment"); + + documentsIndexationService.indexDocument(documentTwo); + + DocumentBean documentThree = new DocumentBean("document3", + "Tardis documentation", "The Doctor", Privacy.PUBLIC.name(), + new Date(), Lists.newArrayList("tardis", "documentation", "old", "new", "borrowed", "blue"), "testDocument", + "This is part of documentation about the TARDIS", "fr", null, "The Doctor, Rose, Amy, River, Clara", + null, null, false, null, "http://tardis.wikia.com/wiki/TARDIS", "no comment"); + + documentsIndexationService.indexDocument(documentThree); + + } + + @After + public void cleanUp() throws Exception { + CoselmarServicesContext serviceContext = getServiceContext(); + DocumentsIndexationService documentsIndexationService = + serviceContext.newService(DocumentsIndexationService.class); + documentsIndexationService.cleanIndex(); + + } +} -- To stop receiving notification emails like this one, please contact codelutin.com SCM administrator <admin+scm@codelutin.com>.
This is an automated email from the git hooks/post-receive script. New commit to branch develop in repository coselmar. See http://git.codelutin.com/coselmar.git commit 81b2cdd4f6adee991d74614788d8c38b28167587 Author: Yannick Martel <martel@©odelutin.com> Date: Tue Dec 30 17:22:46 2014 +0100 add an indexation service for question : base on title, themes and summary --- .../coselmar/services/CoselmarServicesContext.java | 3 + .../services/CoselmarSimpleServiceSupport.java | 5 + .../services/DefaultCoselmarServicesContext.java | 11 + .../indexation/DocumentsIndexationService.java | 57 ++--- .../coselmar/services/indexation/LuceneUtils.java | 51 ++++ .../indexation/QuestionsIndexationService.java | 161 ++++++++++++ .../indexation/DocumentsIndexationServiceTest.java | 8 +- .../indexation/QuestionsIndexationServiceTest.java | 271 +++++++++++++++++++++ 8 files changed, 520 insertions(+), 47 deletions(-) diff --git a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/CoselmarServicesContext.java b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/CoselmarServicesContext.java index 43bc0a9..f242785 100644 --- a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/CoselmarServicesContext.java +++ b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/CoselmarServicesContext.java @@ -30,6 +30,7 @@ import java.util.Locale; import fr.ifremer.coselmar.persistence.CoselmarPersistenceContext; import fr.ifremer.coselmar.persistence.CoselmarTopiaApplicationContext; import fr.ifremer.coselmar.services.config.CoselmarServicesConfig; +import fr.ifremer.coselmar.services.indexation.LuceneUtils; /** * Created by martel on 30/10/14. @@ -44,6 +45,8 @@ public interface CoselmarServicesContext { CoselmarServicesConfig getCoselmarServicesConfig(); + LuceneUtils getLuceneUtils(); + <E extends CoselmarService> E newService(Class<E> serviceClass); Locale getLocale(); diff --git a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/CoselmarSimpleServiceSupport.java b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/CoselmarSimpleServiceSupport.java index caca46a..2637293 100644 --- a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/CoselmarSimpleServiceSupport.java +++ b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/CoselmarSimpleServiceSupport.java @@ -27,6 +27,7 @@ package fr.ifremer.coselmar.services; import fr.ifremer.coselmar.persistence.CoselmarPersistenceContext; import fr.ifremer.coselmar.persistence.entity.CoselmarUserTopiaDao; import fr.ifremer.coselmar.services.config.CoselmarServicesConfig; +import fr.ifremer.coselmar.services.indexation.LuceneUtils; /** * @author ymartel <martel@codelutin.com> @@ -52,6 +53,10 @@ public class CoselmarSimpleServiceSupport implements CoselmarService { return getPersistenceContext().getCoselmarUserDao(); } + protected LuceneUtils getLuceneUtils() { + return servicesContext.getLuceneUtils(); + } + public void commit() { getPersistenceContext().commit(); } diff --git a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/DefaultCoselmarServicesContext.java b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/DefaultCoselmarServicesContext.java index 060e5b1..5f81b74 100644 --- a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/DefaultCoselmarServicesContext.java +++ b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/DefaultCoselmarServicesContext.java @@ -35,6 +35,7 @@ import java.util.Locale; import fr.ifremer.coselmar.persistence.CoselmarPersistenceContext; import fr.ifremer.coselmar.persistence.CoselmarTopiaApplicationContext; import fr.ifremer.coselmar.services.config.CoselmarServicesConfig; +import fr.ifremer.coselmar.services.indexation.LuceneUtils; import org.apache.commons.lang3.RandomStringUtils; import org.apache.commons.lang3.StringUtils; import org.apache.commons.logging.Log; @@ -54,6 +55,8 @@ public class DefaultCoselmarServicesContext implements CoselmarServicesContext { protected Locale locale; + protected LuceneUtils luceneUtils; + private CoselmarTopiaApplicationContext topiaApplicationContext; public void setCoselmarServicesConfig(CoselmarServicesConfig servicesConfig) { @@ -94,6 +97,14 @@ public class DefaultCoselmarServicesContext implements CoselmarServicesContext { } @Override + public LuceneUtils getLuceneUtils() { + if (this.luceneUtils == null) { + this.luceneUtils = new LuceneUtils(this.servicesConfig); + } + return this.luceneUtils; + } + + @Override public <E extends CoselmarService> E newService(Class<E> serviceClass) { E service; diff --git a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationService.java b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationService.java index 08a7c51..0c85c5b 100644 --- a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationService.java +++ b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationService.java @@ -1,6 +1,5 @@ package fr.ifremer.coselmar.services.indexation; -import java.io.File; import java.io.IOException; import java.util.ArrayList; import java.util.List; @@ -8,15 +7,11 @@ import java.util.Set; import fr.ifremer.coselmar.beans.DocumentBean; import fr.ifremer.coselmar.services.CoselmarSimpleServiceSupport; -import org.apache.lucene.analysis.Analyzer; -import org.apache.lucene.analysis.core.SimpleAnalyzer; import org.apache.lucene.document.Document; import org.apache.lucene.document.Field; import org.apache.lucene.document.StringField; import org.apache.lucene.document.TextField; import org.apache.lucene.index.DirectoryReader; -import org.apache.lucene.index.IndexWriter; -import org.apache.lucene.index.IndexWriterConfig; import org.apache.lucene.index.Term; import org.apache.lucene.queryparser.classic.ParseException; import org.apache.lucene.search.BooleanClause; @@ -25,9 +20,6 @@ import org.apache.lucene.search.IndexSearcher; import org.apache.lucene.search.PhraseQuery; import org.apache.lucene.search.ScoreDoc; import org.apache.lucene.search.TermQuery; -import org.apache.lucene.store.Directory; -import org.apache.lucene.store.NIOFSDirectory; -import org.apache.lucene.util.Version; /** * This Services provides operation about {@link fr.ifremer.coselmar.persistence.entity.Document} @@ -45,10 +37,6 @@ import org.apache.lucene.util.Version; */ public class DocumentsIndexationService extends CoselmarSimpleServiceSupport { - public static Analyzer analyzer; - public static final IndexWriterConfig indexationConfig = new IndexWriterConfig(Version.LUCENE_4_10_2, getAnalyzer()); - public static IndexWriter indexWriter; - protected static final String DOCUMENT_ID_INDEX_PROPERTY = "documentId"; protected static final String DOCUMENT_NAME_INDEX_PROPERTY = "documentName"; protected static final String DOCUMENT_AUTHORS_INDEX_PROPERTY = "documentAuthors"; @@ -71,14 +59,13 @@ public class DocumentsIndexationService extends CoselmarSimpleServiceSupport { doc.add(new Field(DOCUMENT_KEYWORD_INDEX_PROPERTY, keyword, TextField.TYPE_STORED)); } - - getIndexWriter().addDocument(doc); - getIndexWriter().commit(); + getLuceneUtils().getIndexWriter().addDocument(doc); + getLuceneUtils().getIndexWriter().commit(); } public List<String> searchDocuments(String text) throws IOException, ParseException { - DirectoryReader ireader = DirectoryReader.open(getIndexWriter(), false); + DirectoryReader ireader = DirectoryReader.open(getLuceneUtils().getIndexWriter(), false); IndexSearcher isearcher = new IndexSearcher(ireader); String[] words = text.split(" "); @@ -123,7 +110,7 @@ public class DocumentsIndexationService extends CoselmarSimpleServiceSupport { } public void updateDocument(DocumentBean document) throws IOException { - DirectoryReader ireader = DirectoryReader.open(getIndexWriter(), false); + DirectoryReader ireader = DirectoryReader.open(getLuceneUtils().getIndexWriter(), false); IndexSearcher isearcher = new IndexSearcher(ireader); // Retrieve document @@ -148,9 +135,11 @@ public class DocumentsIndexationService extends CoselmarSimpleServiceSupport { doc.add(new Field(DOCUMENT_KEYWORD_INDEX_PROPERTY, keyword, TextField.TYPE_STORED)); } - getIndexWriter().updateDocument(new Term(DOCUMENT_ID_INDEX_PROPERTY, document.getId()), doc); - getIndexWriter().commit(); + getLuceneUtils().getIndexWriter().updateDocument(new Term(DOCUMENT_ID_INDEX_PROPERTY, document.getId()), doc); + getLuceneUtils().getIndexWriter().commit(); } + + ireader.close(); } public void deleteDocument(String documentId) throws IOException { @@ -160,35 +149,17 @@ public class DocumentsIndexationService extends CoselmarSimpleServiceSupport { query.add(new TermQuery(new Term(DOCUMENT_ID_INDEX_PROPERTY, documentId)), BooleanClause.Occur.MUST); query.add(new TermQuery(new Term("type", DOCUMENT_TYPE)), BooleanClause.Occur.MUST); - getIndexWriter().deleteDocuments(query); - getIndexWriter().commit(); + getLuceneUtils().getIndexWriter().deleteDocuments(query); + getLuceneUtils().getIndexWriter().commit(); } protected void cleanIndex() throws IOException { - getIndexWriter().deleteAll(); - getIndexWriter().commit(); - } - - protected static Analyzer getAnalyzer() { - if (analyzer == null) { -// analyzer = new StandardAnalyzer(); - //Use simple analyzer to index all words and be able to search with "close word" classified in StandardAnalyzer - analyzer = new SimpleAnalyzer(); - } - return analyzer; - + BooleanQuery query = new BooleanQuery(); + query.add(new TermQuery(new Term("type", DOCUMENT_TYPE)), BooleanClause.Occur.MUST); + getLuceneUtils().getIndexWriter().deleteDocuments(query); + getLuceneUtils().getIndexWriter().commit(); } - protected IndexWriter getIndexWriter() throws IOException { - if (indexWriter == null) { - File indexDirectory = getCoselmarServicesConfig().getIndexDirectory(); - Directory index = NIOFSDirectory.open(indexDirectory); - - indexationConfig.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND); - indexWriter = new IndexWriter(index, indexationConfig); - } - return indexWriter; - } } diff --git a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/LuceneUtils.java b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/LuceneUtils.java new file mode 100644 index 0000000..5eeff68 --- /dev/null +++ b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/LuceneUtils.java @@ -0,0 +1,51 @@ +package fr.ifremer.coselmar.services.indexation; + +import java.io.File; +import java.io.IOException; + +import fr.ifremer.coselmar.services.config.CoselmarServicesConfig; +import org.apache.lucene.analysis.Analyzer; +import org.apache.lucene.analysis.core.SimpleAnalyzer; +import org.apache.lucene.index.IndexWriter; +import org.apache.lucene.index.IndexWriterConfig; +import org.apache.lucene.store.Directory; +import org.apache.lucene.store.NIOFSDirectory; +import org.apache.lucene.util.Version; + +/** + * @author ymartel <martel@codelutin.com> + */ +public class LuceneUtils { + + public static Analyzer analyzer; + public static final IndexWriterConfig indexationConfig = new IndexWriterConfig(Version.LUCENE_4_10_2, getAnalyzer()); + public static IndexWriter indexWriter; + + protected CoselmarServicesConfig servicesConfig; + + public LuceneUtils(CoselmarServicesConfig servicesConfig) { + this.servicesConfig = servicesConfig; + } + + protected static Analyzer getAnalyzer() { + if (analyzer == null) { +// analyzer = new StandardAnalyzer(); + //Use simple analyzer to index all words and be able to search with "close word" classified in StandardAnalyzer + analyzer = new SimpleAnalyzer(); + } + return analyzer; + + } + + public IndexWriter getIndexWriter() throws IOException { + if (indexWriter == null) { + File indexDirectory = servicesConfig.getIndexDirectory(); + Directory index = NIOFSDirectory.open(indexDirectory); + + indexationConfig.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND); + indexWriter = new IndexWriter(index, indexationConfig); + } + return indexWriter; + } + +} diff --git a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/QuestionsIndexationService.java b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/QuestionsIndexationService.java new file mode 100644 index 0000000..178c9b4 --- /dev/null +++ b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/QuestionsIndexationService.java @@ -0,0 +1,161 @@ +package fr.ifremer.coselmar.services.indexation; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Set; + +import fr.ifremer.coselmar.beans.QuestionBean; +import fr.ifremer.coselmar.services.CoselmarSimpleServiceSupport; +import org.apache.lucene.document.Document; +import org.apache.lucene.document.Field; +import org.apache.lucene.document.StringField; +import org.apache.lucene.document.TextField; +import org.apache.lucene.index.DirectoryReader; +import org.apache.lucene.index.Term; +import org.apache.lucene.queryparser.classic.ParseException; +import org.apache.lucene.search.BooleanClause; +import org.apache.lucene.search.BooleanQuery; +import org.apache.lucene.search.IndexSearcher; +import org.apache.lucene.search.PhraseQuery; +import org.apache.lucene.search.ScoreDoc; +import org.apache.lucene.search.TermQuery; + +/** + * This Services provides operation about {@link fr.ifremer.coselmar.persistence.entity.Document} + * or more exactly {@link fr.ifremer.coselmar.beans.DocumentBean} indexation : + * <ul> + * <li>registration of a document in the indexation db</li> + * <li>modification of a document in the indexation db</li> + * <li>documents search from the indexation db</li> + * </ul> + * + * The purpose is to use power of a indexation db (lucene) to increase search on + * document text field, and make easier fulltext search + * + * @author ymartel <martel@codelutin.com> + */ +public class QuestionsIndexationService extends CoselmarSimpleServiceSupport { + + protected static final String QUESTION_ID_INDEX_PROPERTY = "questionId"; + protected static final String QUESTION_TITLE_INDEX_PROPERTY = "questionTitle"; + protected static final String QUESTION_SUMMARY_INDEX_PROPERTY = "questionSummary"; + protected static final String QUESTION_THEME_INDEX_PROPERTY = "questionTheme"; + protected static final String DOCUMENT_TYPE = "question"; + + public void indexQuestion(QuestionBean question) throws IOException { + + Document doc = new Document(); + doc.add(new StringField(QUESTION_ID_INDEX_PROPERTY, question.getId(), Field.Store.YES)); + + doc.add(new TextField(QUESTION_TITLE_INDEX_PROPERTY, question.getTitle(), Field.Store.YES)); + doc.add(new TextField(QUESTION_SUMMARY_INDEX_PROPERTY, question.getSummary(), Field.Store.YES)); + doc.add(new Field("type", DOCUMENT_TYPE, TextField.TYPE_STORED)); + + Set<String> themes = question.getThemes(); + for (String theme : themes) { + doc.add(new Field(QUESTION_THEME_INDEX_PROPERTY, theme, TextField.TYPE_STORED)); + } + + + getLuceneUtils().getIndexWriter().addDocument(doc); + getLuceneUtils().getIndexWriter().commit(); + + } + + public List<String> searchQuestion(String text) throws IOException, ParseException { + DirectoryReader ireader = DirectoryReader.open(getLuceneUtils().getIndexWriter(), false); + IndexSearcher isearcher = new IndexSearcher(ireader); + + String[] words = text.replaceAll("[^a-zA-Z ]", "").toLowerCase().split(" "); + + // Parse a simple query that searches for the "text": + BooleanQuery query = new BooleanQuery(); + + PhraseQuery nameQuery = new PhraseQuery(); + PhraseQuery summaryQuery = new PhraseQuery(); + PhraseQuery authorsQuery = new PhraseQuery(); + + for (String word : words) { + nameQuery.add(new Term(QUESTION_TITLE_INDEX_PROPERTY, word.toLowerCase())); + summaryQuery.add(new Term(QUESTION_SUMMARY_INDEX_PROPERTY, word.toLowerCase())); + } + + query.add(nameQuery, BooleanClause.Occur.SHOULD); + query.add(summaryQuery, BooleanClause.Occur.SHOULD); + query.add(authorsQuery, BooleanClause.Occur.SHOULD); + + query.add(new TermQuery(new Term(QUESTION_THEME_INDEX_PROPERTY, text.toLowerCase())), BooleanClause.Occur.SHOULD); + + + // Combine that with the type + BooleanQuery fullQuery = new BooleanQuery(); + fullQuery.add(query, BooleanClause.Occur.MUST); + fullQuery.add(new TermQuery(new Term("type", DOCUMENT_TYPE)), BooleanClause.Occur.MUST); + + ScoreDoc[] hits = isearcher.search(fullQuery, null, 1000).scoreDocs; + + List<String> documentIds = new ArrayList(hits.length); + + for (ScoreDoc hit : hits) { + Document doc = isearcher.doc(hit.doc); + String documentId = doc.get(QUESTION_ID_INDEX_PROPERTY); + documentIds.add(documentId); + } + + ireader.close(); + return documentIds; + } + + public void updateQuestion(QuestionBean question) throws IOException { + DirectoryReader ireader = DirectoryReader.open(getLuceneUtils().getIndexWriter(), false); + IndexSearcher isearcher = new IndexSearcher(ireader); + + // Retrieve document + BooleanQuery query = new BooleanQuery(); + query.add(new TermQuery(new Term(QUESTION_ID_INDEX_PROPERTY, question.getId())), BooleanClause.Occur.MUST); + query.add(new TermQuery(new Term("type", DOCUMENT_TYPE)), BooleanClause.Occur.MUST); + + ScoreDoc[] hits = isearcher.search(query, null, 1000).scoreDocs; + if (hits.length > 0) { + Document doc = isearcher.doc(hits[0].doc); + + doc.removeField(QUESTION_TITLE_INDEX_PROPERTY); + doc.add(new TextField(QUESTION_TITLE_INDEX_PROPERTY, question.getTitle(), Field.Store.YES)); + doc.removeField(QUESTION_SUMMARY_INDEX_PROPERTY); + doc.add(new TextField(QUESTION_SUMMARY_INDEX_PROPERTY, question.getSummary(), Field.Store.YES)); + + doc.removeFields(QUESTION_THEME_INDEX_PROPERTY); + Set<String> themes = question.getThemes(); + for (String theme : themes) { + doc.add(new Field(QUESTION_THEME_INDEX_PROPERTY, theme, TextField.TYPE_STORED)); + } + + getLuceneUtils().getIndexWriter().updateDocument(new Term(QUESTION_ID_INDEX_PROPERTY, question.getId()), doc); + getLuceneUtils().getIndexWriter().commit(); + } + + ireader.close(); + } + + public void deleteQuestion(String documentId) throws IOException { + + // Retrieve document + BooleanQuery query = new BooleanQuery(); + query.add(new TermQuery(new Term(QUESTION_ID_INDEX_PROPERTY, documentId)), BooleanClause.Occur.MUST); + query.add(new TermQuery(new Term("type", DOCUMENT_TYPE)), BooleanClause.Occur.MUST); + + getLuceneUtils().getIndexWriter().deleteDocuments(query); + getLuceneUtils().getIndexWriter().commit(); + + } + + protected void cleanIndex() throws IOException { + BooleanQuery query = new BooleanQuery(); + query.add(new TermQuery(new Term("type", DOCUMENT_TYPE)), BooleanClause.Occur.MUST); + getLuceneUtils().getIndexWriter().deleteDocuments(query); + getLuceneUtils().getIndexWriter().commit(); + } + + +} diff --git a/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationServiceTest.java b/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationServiceTest.java index 7b667a3..251c084 100644 --- a/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationServiceTest.java +++ b/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/indexation/DocumentsIndexationServiceTest.java @@ -19,13 +19,13 @@ import org.junit.Test; */ public class DocumentsIndexationServiceTest extends AbstractCoselmarServiceTest { -// protected FakeCoselmarServicesContext serviceContext; + protected FakeCoselmarServicesContext serviceContext; protected FakeCoselmarServicesContext getServiceContext() { -// if (serviceContext == null) { - FakeCoselmarServicesContext serviceContext = application.newServiceContext(application.newPersistenceContext(), Locale.FRANCE); -// } + if (serviceContext == null) { + serviceContext = application.newServiceContext(application.newPersistenceContext(), Locale.FRANCE); + } return serviceContext; } diff --git a/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/indexation/QuestionsIndexationServiceTest.java b/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/indexation/QuestionsIndexationServiceTest.java new file mode 100644 index 0000000..fff64be --- /dev/null +++ b/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/indexation/QuestionsIndexationServiceTest.java @@ -0,0 +1,271 @@ +package fr.ifremer.coselmar.services.indexation; + +import java.util.Date; +import java.util.List; +import java.util.Locale; + +import com.google.common.collect.Sets; +import fr.ifremer.coselmar.beans.QuestionBean; +import fr.ifremer.coselmar.persistence.entity.Privacy; +import fr.ifremer.coselmar.persistence.entity.Status; +import fr.ifremer.coselmar.services.AbstractCoselmarServiceTest; +import fr.ifremer.coselmar.services.CoselmarServicesContext; +import fr.ifremer.coselmar.services.FakeCoselmarServicesContext; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.nuiton.util.DateUtil; + +/** + * @author ymartel <martel@codelutin.com> + */ +public class QuestionsIndexationServiceTest extends AbstractCoselmarServiceTest { + + protected FakeCoselmarServicesContext serviceContext; + + protected FakeCoselmarServicesContext getServiceContext() { + + if (this.serviceContext == null) { + this.serviceContext = application.newServiceContext(application.newPersistenceContext(), Locale.FRANCE); + } + + return this.serviceContext; + } + + @Test + public void testAddQuestion() throws Exception { + CoselmarServicesContext serviceContext = getServiceContext(); + QuestionsIndexationService questionsIndexationService = + serviceContext.newService(QuestionsIndexationService.class); + + QuestionBean question = new QuestionBean(); + question.setTitle("Awesome question"); + question.setSummary("Can we, just once, ask about it ?"); + question.setDeadline(DateUtil.createDateAfterToday(1, 1, 1)); + question.setExternalExperts(Sets.newHashSet("Amelia", "Rory", "River")); + question.setSubmissionDate(new Date()); + question.setStatus(Status.OPEN.name()); + question.setPrivacy(Privacy.PUBLIC.name()); + question.setThemes(Sets.newHashSet("TARDIS", "Universe", "Time", "Space")); + question.setId("question_" + System.currentTimeMillis()); + + questionsIndexationService.indexQuestion(question); + + } + + @Test + public void testSearchQuestion() throws Exception { + + QuestionBean questionOne = new QuestionBean(); + String questionOneId = "question_1_test_search" + System.currentTimeMillis(); + questionOne.setId(questionOneId); + questionOne.setTitle("Awesome question"); + questionOne.setSummary("Can we, just once, ask about it ?"); + questionOne.setDeadline(DateUtil.createDateAfterToday(1, 0, 1)); + questionOne.setExternalExperts(Sets.newHashSet("Amelia", "Rory", "River")); + questionOne.setSubmissionDate(new Date()); + questionOne.setStatus(Status.OPEN.name()); + questionOne.setPrivacy(Privacy.PUBLIC.name()); + questionOne.setThemes(Sets.newHashSet("TARDIS", "Universe", "Time", "Space")); + + QuestionBean questionTwo = new QuestionBean(); + String questionTwoId = "question_2_test_search" + System.currentTimeMillis(); + questionTwo.setId(questionTwoId); + questionTwo.setTitle("The ultimate"); + questionTwo.setSummary("We need some question"); + questionTwo.setDeadline(DateUtil.createDateAfterToday(16, 0, 0)); + questionTwo.setSubmissionDate(new Date()); + questionTwo.setStatus(Status.OPEN.name()); + questionTwo.setPrivacy(Privacy.PUBLIC.name()); + questionTwo.setThemes(Sets.newHashSet("test", "question")); + + QuestionBean questionThree = new QuestionBean(); + String questionThreeId = "question_3_test_search" + System.currentTimeMillis(); + questionThree.setId(questionThreeId); + questionThree.setTitle("There's someone missing. The question's Who?"); + questionThree.setSummary("Something old, Something new, Something borrowed, Something blue."); + questionThree.setDeadline(DateUtil.createDateAfterToday(16, 0, 0)); + questionThree.setSubmissionDate(new Date()); + questionThree.setStatus(Status.OPEN.name()); + questionThree.setPrivacy(Privacy.PUBLIC.name()); + questionThree.setThemes(Sets.newHashSet("big bang two", "Pandorica", "River", "Universe")); + + + CoselmarServicesContext serviceContext = getServiceContext(); + QuestionsIndexationService questionsIndexationService = + serviceContext.newService(QuestionsIndexationService.class); + + questionsIndexationService.indexQuestion(questionOne); + questionsIndexationService.indexQuestion(questionTwo); + questionsIndexationService.indexQuestion(questionThree); + + // Ok, let's search now ! + List<String> questionMatchingTitleIds = questionsIndexationService.searchQuestion("ultimate"); + Assert.assertEquals(1, questionMatchingTitleIds.size()); + Assert.assertEquals(questionTwoId, questionMatchingTitleIds.get(0)); + + List<String> questionMatchingAmyIds = questionsIndexationService.searchQuestion("amy"); + Assert.assertTrue(questionMatchingAmyIds.isEmpty()); + + List<String> questionMatchingPandoricaIds = questionsIndexationService.searchQuestion("Pandorica"); + Assert.assertEquals(1, questionMatchingPandoricaIds.size()); + Assert.assertEquals(questionThreeId, questionMatchingPandoricaIds.get(0)); + + List<String> questionMatchingUniverseIds = questionsIndexationService.searchQuestion("universe"); + Assert.assertEquals(2, questionMatchingUniverseIds.size()); + Assert.assertTrue(questionMatchingUniverseIds.contains(questionThreeId)); + Assert.assertTrue(questionMatchingUniverseIds.contains(questionOneId)); + + List<String> documentMatchingUnknownIds = questionsIndexationService.searchQuestion("Unknown"); + Assert.assertTrue(documentMatchingUnknownIds.isEmpty()); + + List<String> documentMatchingPartOfSummaryIds = questionsIndexationService.searchQuestion("Something blue"); + Assert.assertEquals(1, documentMatchingPartOfSummaryIds.size()); + Assert.assertEquals(questionThreeId, documentMatchingPartOfSummaryIds.get(0)); + + documentMatchingPartOfSummaryIds = questionsIndexationService.searchQuestion("Something borrowed,"); + Assert.assertEquals(1, documentMatchingPartOfSummaryIds.size()); + Assert.assertEquals(questionThreeId, documentMatchingPartOfSummaryIds.get(0)); + + documentMatchingPartOfSummaryIds = questionsIndexationService.searchQuestion("can we"); + Assert.assertEquals(1, documentMatchingPartOfSummaryIds.size()); + Assert.assertEquals(questionOneId, documentMatchingPartOfSummaryIds.get(0)); + + } + + @Test + public void testUpdateQuestion() throws Exception { + + QuestionBean questionOne = new QuestionBean(); + String questionOneId = "question_1_test_update_" + System.currentTimeMillis(); + questionOne.setId(questionOneId); + questionOne.setTitle("Awesome question"); + questionOne.setSummary("Can we, just once, ask about it ?"); + questionOne.setDeadline(DateUtil.createDateAfterToday(1, 0, 1)); + questionOne.setExternalExperts(Sets.newHashSet("Amelia", "Rory", "River")); + questionOne.setSubmissionDate(new Date()); + questionOne.setStatus(Status.OPEN.name()); + questionOne.setPrivacy(Privacy.PUBLIC.name()); + questionOne.setThemes(Sets.newHashSet("TARDIS", "Universe", "Time", "Space")); + + QuestionBean questionTwo = new QuestionBean(); + String questionTwoId = "question_2_test_update_" + System.currentTimeMillis(); + questionTwo.setId(questionTwoId); + questionTwo.setTitle("There's someone missing. The question's Who?"); + questionTwo.setSummary("Something old, Something new, Something borrowed, Something blue."); + questionTwo.setDeadline(DateUtil.createDateAfterToday(16, 0, 0)); + questionTwo.setSubmissionDate(new Date()); + questionTwo.setStatus(Status.OPEN.name()); + questionTwo.setPrivacy(Privacy.PUBLIC.name()); + questionTwo.setThemes(Sets.newHashSet("big bang two", "Pandorica", "River", "Universe")); + + + CoselmarServicesContext serviceContext = getServiceContext(); + QuestionsIndexationService questionsIndexationService = + serviceContext.newService(QuestionsIndexationService.class); + + questionsIndexationService.indexQuestion(questionOne); + questionsIndexationService.indexQuestion(questionTwo); + + questionOne.setTitle("How old is the doctor who ?"); + questionsIndexationService.updateQuestion(questionOne); + + // Ok, let's search now ! + List<String> questionMatchingTitleIds = questionsIndexationService.searchQuestion("awesome"); + Assert.assertTrue(questionMatchingTitleIds.isEmpty()); + + questionMatchingTitleIds = questionsIndexationService.searchQuestion("who"); + Assert.assertEquals(2, questionMatchingTitleIds.size()); + Assert.assertTrue(questionMatchingTitleIds.contains(questionOneId)); + Assert.assertTrue(questionMatchingTitleIds.contains(questionTwoId)); + + questionMatchingTitleIds = questionsIndexationService.searchQuestion("doctor"); + Assert.assertEquals(1, questionMatchingTitleIds.size()); + Assert.assertTrue(questionMatchingTitleIds.contains(questionOneId)); + + } + + @Test + public void testDeleteQuestion() throws Exception { + + QuestionBean questionOne = new QuestionBean(); + String questionOneId = "question_1_test_delete_" + System.currentTimeMillis(); + questionOne.setId(questionOneId); + questionOne.setTitle("Awesome question"); + questionOne.setSummary("Can we, just once, ask about it ?"); + questionOne.setDeadline(DateUtil.createDateAfterToday(1, 0, 1)); + questionOne.setExternalExperts(Sets.newHashSet("Amelia", "Rory", "River")); + questionOne.setSubmissionDate(new Date()); + questionOne.setStatus(Status.OPEN.name()); + questionOne.setPrivacy(Privacy.PUBLIC.name()); + questionOne.setThemes(Sets.newHashSet("TARDIS", "Universe", "Time", "Space")); + + QuestionBean questionTwo = new QuestionBean(); + String questionTwoId = "question_2_test_delete_" + System.currentTimeMillis(); + questionTwo.setId(questionTwoId); + questionTwo.setTitle("There's someone missing. The question's Who?"); + questionTwo.setSummary("Something old, Something new, Something borrowed, Something blue."); + questionTwo.setDeadline(DateUtil.createDateAfterToday(16, 0, 0)); + questionTwo.setSubmissionDate(new Date()); + questionTwo.setStatus(Status.OPEN.name()); + questionTwo.setPrivacy(Privacy.PUBLIC.name()); + questionTwo.setThemes(Sets.newHashSet("big bang two", "Pandorica", "River", "Universe")); + + + CoselmarServicesContext serviceContext = getServiceContext(); + QuestionsIndexationService questionsIndexationService = + serviceContext.newService(QuestionsIndexationService.class); + + questionsIndexationService.indexQuestion(questionOne); + questionsIndexationService.indexQuestion(questionTwo); + + // Ok, let's search now ! + List<String> questionMatchingTitleIds = questionsIndexationService.searchQuestion("awesome"); + Assert.assertEquals(1, questionMatchingTitleIds.size()); + Assert.assertEquals(questionOneId, questionMatchingTitleIds.get(0)); + + questionMatchingTitleIds = questionsIndexationService.searchQuestion("missing"); + Assert.assertEquals(1, questionMatchingTitleIds.size()); + Assert.assertEquals(questionTwoId, questionMatchingTitleIds.get(0)); + + List<String> questionMatchingUniverseIds = questionsIndexationService.searchQuestion("universe"); + Assert.assertEquals(2, questionMatchingUniverseIds.size()); + Assert.assertTrue(questionMatchingUniverseIds.contains(questionOneId)); + Assert.assertTrue(questionMatchingUniverseIds.contains(questionTwoId)); + + // Ok, this was already tested as ok, let's delete questionOne + questionsIndexationService.deleteQuestion(questionOneId); + + // and let's search same now ! + questionMatchingTitleIds = questionsIndexationService.searchQuestion("awesome"); + Assert.assertTrue(questionMatchingTitleIds.isEmpty()); + + questionMatchingTitleIds = questionsIndexationService.searchQuestion("missing"); + Assert.assertEquals(1, questionMatchingTitleIds.size()); + Assert.assertTrue(questionMatchingTitleIds.contains(questionTwoId)); + + questionMatchingUniverseIds = questionsIndexationService.searchQuestion("universe"); + Assert.assertEquals(1, questionMatchingUniverseIds.size()); + Assert.assertTrue(questionMatchingUniverseIds.contains(questionTwoId)); + + + List<String> documentMatchingPartOfSummaryIds = questionsIndexationService.searchQuestion("can we"); + Assert.assertTrue(documentMatchingPartOfSummaryIds.isEmpty()); + + documentMatchingPartOfSummaryIds = questionsIndexationService.searchQuestion("Something blue"); + Assert.assertEquals(1, documentMatchingPartOfSummaryIds.size()); + Assert.assertEquals(questionTwoId, documentMatchingPartOfSummaryIds.get(0)); + + + } + + @Before + public void cleanUp() throws Exception { + CoselmarServicesContext serviceContext = getServiceContext(); + QuestionsIndexationService questionsIndexationService = + serviceContext.newService(QuestionsIndexationService.class); + + questionsIndexationService.cleanIndex(); + + } +} -- To stop receiving notification emails like this one, please contact codelutin.com SCM administrator <admin+scm@codelutin.com>.
This is an automated email from the git hooks/post-receive script. New commit to branch develop in repository coselmar. See http://git.codelutin.com/coselmar.git commit 37c0bed1b33c6c4d3bc56ff3f88b5f7148548b88 Author: Yannick Martel <martel@©odelutin.com> Date: Wed Dec 31 10:51:30 2014 +0100 use question indexation in webservice --- .../indexation/QuestionsIndexationService.java | 84 +++++++++++----------- .../coselmar/services/v1/QuestionsWebService.java | 46 ++++++++++++ .../indexation/QuestionsIndexationServiceTest.java | 3 +- 3 files changed, 90 insertions(+), 43 deletions(-) diff --git a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/QuestionsIndexationService.java b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/QuestionsIndexationService.java index 178c9b4..a1f1773 100644 --- a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/QuestionsIndexationService.java +++ b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/indexation/QuestionsIndexationService.java @@ -45,21 +45,54 @@ public class QuestionsIndexationService extends CoselmarSimpleServiceSupport { public void indexQuestion(QuestionBean question) throws IOException { - Document doc = new Document(); - doc.add(new StringField(QUESTION_ID_INDEX_PROPERTY, question.getId(), Field.Store.YES)); + // First : try to find if already exist to update it + DirectoryReader ireader = DirectoryReader.open(getLuceneUtils().getIndexWriter(), false); + IndexSearcher isearcher = new IndexSearcher(ireader); - doc.add(new TextField(QUESTION_TITLE_INDEX_PROPERTY, question.getTitle(), Field.Store.YES)); - doc.add(new TextField(QUESTION_SUMMARY_INDEX_PROPERTY, question.getSummary(), Field.Store.YES)); - doc.add(new Field("type", DOCUMENT_TYPE, TextField.TYPE_STORED)); + // Retrieve document + BooleanQuery query = new BooleanQuery(); + query.add(new TermQuery(new Term(QUESTION_ID_INDEX_PROPERTY, question.getId())), BooleanClause.Occur.MUST); + query.add(new TermQuery(new Term("type", DOCUMENT_TYPE)), BooleanClause.Occur.MUST); - Set<String> themes = question.getThemes(); - for (String theme : themes) { - doc.add(new Field(QUESTION_THEME_INDEX_PROPERTY, theme, TextField.TYPE_STORED)); - } + ScoreDoc[] hits = isearcher.search(query, null, 1000).scoreDocs; + if (hits.length > 0) { + Document doc = isearcher.doc(hits[0].doc); + + doc.removeField(QUESTION_TITLE_INDEX_PROPERTY); + doc.add(new TextField(QUESTION_TITLE_INDEX_PROPERTY, question.getTitle(), Field.Store.YES)); + doc.removeField(QUESTION_SUMMARY_INDEX_PROPERTY); + doc.add(new TextField(QUESTION_SUMMARY_INDEX_PROPERTY, question.getSummary(), Field.Store.YES)); + + doc.removeFields(QUESTION_THEME_INDEX_PROPERTY); + Set<String> themes = question.getThemes(); + for (String theme : themes) { + doc.add(new Field(QUESTION_THEME_INDEX_PROPERTY, theme, TextField.TYPE_STORED)); + } + + getLuceneUtils().getIndexWriter().updateDocument(new Term(QUESTION_ID_INDEX_PROPERTY, question.getId()), doc); + + } else { + // Not exist yet : add it to index + + Document doc = new Document(); + doc.add(new StringField(QUESTION_ID_INDEX_PROPERTY, question.getId(), Field.Store.YES)); + + doc.add(new TextField(QUESTION_TITLE_INDEX_PROPERTY, question.getTitle(), Field.Store.YES)); + doc.add(new TextField(QUESTION_SUMMARY_INDEX_PROPERTY, question.getSummary(), Field.Store.YES)); + doc.add(new Field("type", DOCUMENT_TYPE, TextField.TYPE_STORED)); + + Set<String> themes = question.getThemes(); + for (String theme : themes) { + doc.add(new Field(QUESTION_THEME_INDEX_PROPERTY, theme, TextField.TYPE_STORED)); + } + getLuceneUtils().getIndexWriter().addDocument(doc); - getLuceneUtils().getIndexWriter().addDocument(doc); + } + + // Commit, close reader. getLuceneUtils().getIndexWriter().commit(); + ireader.close(); } @@ -107,37 +140,6 @@ public class QuestionsIndexationService extends CoselmarSimpleServiceSupport { return documentIds; } - public void updateQuestion(QuestionBean question) throws IOException { - DirectoryReader ireader = DirectoryReader.open(getLuceneUtils().getIndexWriter(), false); - IndexSearcher isearcher = new IndexSearcher(ireader); - - // Retrieve document - BooleanQuery query = new BooleanQuery(); - query.add(new TermQuery(new Term(QUESTION_ID_INDEX_PROPERTY, question.getId())), BooleanClause.Occur.MUST); - query.add(new TermQuery(new Term("type", DOCUMENT_TYPE)), BooleanClause.Occur.MUST); - - ScoreDoc[] hits = isearcher.search(query, null, 1000).scoreDocs; - if (hits.length > 0) { - Document doc = isearcher.doc(hits[0].doc); - - doc.removeField(QUESTION_TITLE_INDEX_PROPERTY); - doc.add(new TextField(QUESTION_TITLE_INDEX_PROPERTY, question.getTitle(), Field.Store.YES)); - doc.removeField(QUESTION_SUMMARY_INDEX_PROPERTY); - doc.add(new TextField(QUESTION_SUMMARY_INDEX_PROPERTY, question.getSummary(), Field.Store.YES)); - - doc.removeFields(QUESTION_THEME_INDEX_PROPERTY); - Set<String> themes = question.getThemes(); - for (String theme : themes) { - doc.add(new Field(QUESTION_THEME_INDEX_PROPERTY, theme, TextField.TYPE_STORED)); - } - - getLuceneUtils().getIndexWriter().updateDocument(new Term(QUESTION_ID_INDEX_PROPERTY, question.getId()), doc); - getLuceneUtils().getIndexWriter().commit(); - } - - ireader.close(); - } - public void deleteQuestion(String documentId) throws IOException { // Retrieve document diff --git a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/v1/QuestionsWebService.java b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/v1/QuestionsWebService.java index ed57270..2434279 100644 --- a/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/v1/QuestionsWebService.java +++ b/coselmar-rest/src/main/java/fr/ifremer/coselmar/services/v1/QuestionsWebService.java @@ -24,6 +24,7 @@ package fr.ifremer.coselmar.services.v1; * #L% */ +import java.io.IOException; import java.util.ArrayList; import java.util.Collection; import java.util.Date; @@ -49,6 +50,7 @@ import fr.ifremer.coselmar.persistence.entity.Status; import fr.ifremer.coselmar.services.CoselmarWebServiceSupport; import fr.ifremer.coselmar.services.errors.InvalidCredentialException; import fr.ifremer.coselmar.services.errors.UnauthorizedException; +import fr.ifremer.coselmar.services.indexation.QuestionsIndexationService; import org.apache.commons.lang3.StringUtils; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; @@ -199,6 +201,21 @@ public class QuestionsWebService extends CoselmarWebServiceSupport { } commit(); + + QuestionBean result = BeanEntityConverter.toBean(getPersistenceContext().getTopiaIdFactory(), questionEntity); + + QuestionsIndexationService questionsIndexationService = getServicesContext().newService(QuestionsIndexationService.class); + try { + questionsIndexationService.indexQuestion(result); + if (log.isDebugEnabled()) { + String message = String.format("Question '%s' added to index", result.getTitle()); + log.debug(message); + } + } catch (IOException e) { + if (log.isErrorEnabled()) { + log.error("Unable to index new question", e); + } + } } public List<QuestionBean> getQuestions() throws InvalidCredentialException, UnauthorizedException { @@ -305,6 +322,20 @@ public class QuestionsWebService extends CoselmarWebServiceSupport { getPersistenceContext().getCoselmarUserGroupDao().delete(participantGroup); commit(); + + // Remove it from index too + QuestionsIndexationService questionsIndexationService = getServicesContext().newService(QuestionsIndexationService.class); + try { + questionsIndexationService.deleteQuestion(questionId); + if (log.isDebugEnabled()) { + String message = String.format("Question '%s' deleted from index", questionId); + log.debug(message); + } + } catch (IOException e) { + if (log.isErrorEnabled()) { + log.error("Unable to remove question from index", e); + } + } } public QuestionBean getQuestion(String questionId) throws InvalidCredentialException, UnauthorizedException { @@ -661,6 +692,21 @@ public class QuestionsWebService extends CoselmarWebServiceSupport { } commit(); + + QuestionBean result = BeanEntityConverter.toBean(getPersistenceContext().getTopiaIdFactory(), questionEntity); + + QuestionsIndexationService questionsIndexationService = getServicesContext().newService(QuestionsIndexationService.class); + try { + questionsIndexationService.indexQuestion(result); + if (log.isDebugEnabled()) { + String message = String.format("Question '%s' added to index", result.getTitle()); + log.debug(message); + } + } catch (IOException e) { + if (log.isErrorEnabled()) { + log.error("Unable to index new question", e); + } + } } public List<String> getThemes() throws InvalidCredentialException, UnauthorizedException { diff --git a/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/indexation/QuestionsIndexationServiceTest.java b/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/indexation/QuestionsIndexationServiceTest.java index fff64be..d6175e8 100644 --- a/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/indexation/QuestionsIndexationServiceTest.java +++ b/coselmar-rest/src/test/java/fr/ifremer/coselmar/services/indexation/QuestionsIndexationServiceTest.java @@ -168,7 +168,7 @@ public class QuestionsIndexationServiceTest extends AbstractCoselmarServiceTest questionsIndexationService.indexQuestion(questionTwo); questionOne.setTitle("How old is the doctor who ?"); - questionsIndexationService.updateQuestion(questionOne); + questionsIndexationService.indexQuestion(questionOne); // Ok, let's search now ! List<String> questionMatchingTitleIds = questionsIndexationService.searchQuestion("awesome"); @@ -256,7 +256,6 @@ public class QuestionsIndexationServiceTest extends AbstractCoselmarServiceTest Assert.assertEquals(1, documentMatchingPartOfSummaryIds.size()); Assert.assertEquals(questionTwoId, documentMatchingPartOfSummaryIds.get(0)); - } @Before -- To stop receiving notification emails like this one, please contact codelutin.com SCM administrator <admin+scm@codelutin.com>.
This is an automated email from the git hooks/post-receive script. New commit to branch develop in repository coselmar. See http://git.codelutin.com/coselmar.git commit 7ed66eb28deff9e31ddb7f7ea3cbb038bd2be813 Merge: d2cf910 37c0bed Author: Yannick Martel <martel@©odelutin.com> Date: Wed Dec 31 11:10:51 2014 +0100 Merge branch 'feature/lucene-indexation' into develop coselmar-rest/pom.xml | 18 ++ .../coselmar/services/CoselmarServicesContext.java | 3 + .../services/CoselmarSimpleServiceSupport.java | 5 + .../services/DefaultCoselmarServicesContext.java | 11 + .../services/config/CoselmarServicesConfig.java | 9 + .../config/CoselmarServicesConfigOption.java | 6 + .../indexation/DocumentsIndexationService.java | 165 +++++++++++++ .../coselmar/services/indexation/LuceneUtils.java | 51 ++++ .../indexation/QuestionsIndexationService.java | 163 +++++++++++++ .../coselmar/services/v1/DocumentsWebService.java | 65 ++++- .../coselmar/services/v1/QuestionsWebService.java | 46 ++++ .../services/AbstractCoselmarServiceTest.java} | 33 +-- .../indexation/DocumentsIndexationServiceTest.java | 208 ++++++++++++++++ .../indexation/QuestionsIndexationServiceTest.java | 270 +++++++++++++++++++++ pom.xml | 27 +++ 15 files changed, 1054 insertions(+), 26 deletions(-) -- To stop receiving notification emails like this one, please contact codelutin.com SCM administrator <admin+scm@codelutin.com>.
participants (1)
-
codelutin.com scm