How do I use an index in memory?
Author: Deron Eriksson
Description: This Java tutorial shows how to use a Lucene index in memory rather than in the file system.
Tutorial created using: Windows XP || JDK 1.5.0_09 || Eclipse Web Tools Platform 2.0 (Eclipse 3.3.0)


In another tutorial, we created a Lucene index in the file system and searched that index. Depending on the application, you may require an index to be stored in-memory rather than in the file system. With Lucene, this can be accomplished easily through the use of the RAMDirectory class, which the javadocs describe as "A memory-resident Directory implementation".

This example will utilize the following project structure:

project structure

Two text files in the "filesToIndex" directory will be indexed. The first one, deron-foods.txt, lists some foods that I like.

deron-foods.txt

Here are some foods that Deron likes:
hamburger
french fries
steak
mushrooms
artichokes

The second text file, nicole-foods.txt, lists some foods that Nicole likes.

nicole-foods.txt

Here are some foods that Nicole likes:
apples
bananas
salad
mushrooms
cheese

Now, let's examine the LuceneMemoryIndexDemo class, which creates an in-memory index and then searches it. The LuceneMemoryIndexDemo class is a slight modification of the LuceneDemo class from the other tutorial. The other tutorial describes the indexing and searching processes, so I won't describe them here.

LuceneMemoryIndexDemo.java

package avajava;

import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.Reader;
import java.util.Iterator;

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.index.CorruptIndexException;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.queryParser.ParseException;
import org.apache.lucene.queryParser.QueryParser;
import org.apache.lucene.search.Hit;
import org.apache.lucene.search.Hits;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.LockObtainFailedException;
import org.apache.lucene.store.RAMDirectory;

public class LuceneMemoryIndexDemo {

	public static final String FILES_TO_INDEX_DIRECTORY = "filesToIndex";

	public static final String FIELD_PATH = "path";
	public static final String FIELD_CONTENTS = "contents";

	private static Directory ramDirectory = new RAMDirectory();

	public static void main(String[] args) throws Exception {

		createIndex();
		searchIndex("mushrooms");
		searchIndex("steak");
		searchIndex("steak AND cheese");
		searchIndex("steak and cheese");
		searchIndex("bacon OR cheese");

	}

	public static void createIndex() throws CorruptIndexException, LockObtainFailedException, IOException {
		Analyzer analyzer = new StandardAnalyzer();
		IndexWriter indexWriter = new IndexWriter(ramDirectory, analyzer);
		File dir = new File(FILES_TO_INDEX_DIRECTORY);
		File[] files = dir.listFiles();
		for (File file : files) {
			Document document = new Document();

			String path = file.getCanonicalPath();
			document.add(new Field(FIELD_PATH, path, Field.Store.YES, Field.Index.UN_TOKENIZED));

			Reader reader = new FileReader(file);
			document.add(new Field(FIELD_CONTENTS, reader));

			indexWriter.addDocument(document);
		}
		indexWriter.optimize();
		indexWriter.close();
	}

	public static void searchIndex(String searchString) throws IOException, ParseException {
		System.out.println("Searching for '" + searchString + "'");
		IndexReader indexReader = IndexReader.open(ramDirectory);
		IndexSearcher indexSearcher = new IndexSearcher(indexReader);

		Analyzer analyzer = new StandardAnalyzer();
		QueryParser queryParser = new QueryParser(FIELD_CONTENTS, analyzer);
		Query query = queryParser.parse(searchString);
		Hits hits = indexSearcher.search(query);
		System.out.println("Number of hits: " + hits.length());

		Iterator<Hit> it = hits.iterator();
		while (it.hasNext()) {
			Hit hit = it.next();
			Document document = hit.getDocument();
			String path = document.get(FIELD_PATH);
			System.out.println("Hit: " + path);
		}

	}

}

The LuceneMemoryIndexDemo class utilizes a Directory that is a RAMDirectory, which is memory-resident:

private static Directory ramDirectory = new RAMDirectory();

The ramDirectory variable is utilized by the IndexWriter object (which creates the index):

IndexWriter indexWriter = new IndexWriter(ramDirectory, analyzer);

The ramDirectory variable is also specified for the IndexReader, which is used to search the index:

IndexReader indexReader = IndexReader.open(ramDirectory);

Making these slight modifications results in a memory-based index that is created via IndexWriter and searched via IndexReader/IndexSearcher. That's all there is to it!

The console output of executing LuceneMemoryIndexDemo is shown here:

Console Output

Searching for 'mushrooms'
Number of hits: 2
Hit: C:\projects\workspace\demo\filesToIndex\nicole-foods.txt
Hit: C:\projects\workspace\demo\filesToIndex\deron-foods.txt
Searching for 'steak'
Number of hits: 1
Hit: C:\projects\workspace\demo\filesToIndex\deron-foods.txt
Searching for 'steak AND cheese'
Number of hits: 0
Searching for 'steak and cheese'
Number of hits: 2
Hit: C:\projects\workspace\demo\filesToIndex\nicole-foods.txt
Hit: C:\projects\workspace\demo\filesToIndex\deron-foods.txt
Searching for 'bacon OR cheese'
Number of hits: 1
Hit: C:\projects\workspace\demo\filesToIndex\nicole-foods.txt