Open-source Hebrew information retrieval (HebMorph, part 3)

Phase 1 - Hebrew Morphological Analyzer for Lucene (done)

1. Exact - the word is being searched in the radix tree for an exact match.
2. Prefixes omitted - the Hebrew prefix letters M$H WKLB are being omitted one-by-one (while exist), and whenever a word is found its linguistic payload is checked against the prefix omitted. The word is only added to the list of candidates if the prefix is legal for the word.
3. Toleration - an optional tolerative lookup is being performed, where letters in the key are being skipped over or added based on a set of rules, so many spellings incompatible with the Academy's spelling rules can still be identified and accepted. For each valid Hebrew prefixes, the prefix letters are removed and the process is repeated. Words identified using tolerators are being given a lower rank, so exact matches will be marked as more probable. Unlike the original hspell's code, multiple toleration methods are supported in a single lookup operation.

Phase 2 - Improving and stabilizing the morphological analyzer

1. Test hspell's coverage against Wikipedia and possibly other Hebrew corporas, and reporting missing words to the original hspell project.
2. Find the best way to handle OOV cases. Since hspell seem to recognize most verbs, perhaps we can assume all OOV cases to be nouns and strip affixes associated with nouns? Another option would be to analyze the tokens stream and look for similar occurrences, taking into account different spellings and missing prefixes.
3. Try to resolve ambiguities using a similar approach as above (by looking at the stream of tokens), or by using synonyms.
4. Decide on a maximum number of lemmas to index per term and a minimum probability ranking so the index stays relatively small and precision stays high.
5. Generate a more complete list of stop words, by looking at the highest frequency terms he-wiki that have little value.
6. Improve tolerance mechanism. While it is meant to prevent unnecessary OOV cases and to provide other likely meanings of a word, it can also be a source of many unwanted ambiguities, and result in adding many incorrect tokens to the index. Therefore:
   • Only tolerate consonants used as vowels (primarily Waw and Yud), and non-doubled consonant Waw letters.
   • Tweak tolerator functions to be less greedy, and avoid tolerations where they are less likely to be needed.
   • Make sure this process only makes valid expansions, and never create spelling errors.
   • While tolerating, never try and guess what the user was looking for, but rather what other probable meaning could this word have.
   • Research at what is the best ranking formula to be used in tolerators, so decision making of the disambiguation process can be more effective.
7. Make the morphological analyzer more flexible, so it gives control over word lookup sensitivity (exact / with toleration / morphological / expanded).

Phase 3 - OpenRelevance

Phase 4 - Creating and comparing various Analyzers

1. Simple n-grams analyzer. 4/5-grams have shown to work pretty well for Arabic, and this may be the case for Hebrew as well. See http://web.jhu.edu/bin/q/b/p75-mcnamee.pdf, mainly the section "Reasons for N-gram Effectiveness".
2. Other variations of n-grams, such as word-internal n-grams.
3. n-grams may not work as well for Hebrew as they do for Arabic, due to writing inconsistencies caused by Niqqud-less spelling (mainly in addition or omission of vowels), and more ambiguous prefixes. Therefore, several n-gram variations such as skipgrams, or some sort of vowels normalization, may work better than simple n-grams.
4. Full scale algorithmic stemmers, such as Segal’s publicly available morphological analyzer, or this one (seem to be used also by the Mila center). Ideally one without any dictionary files. This is probably going to yield similar results to a dictionary-based approach, but worths a try since the different error rates may result in different relevancy. Perhaps making a Hebrew Snowball while we are at it.
5. "Light stemmers", algorithms producing indexing units out of words by just removing prefix and doing other simple normalization operations. This is the best method today for many languages, also proven to work with Arabic (even though it doesn't handle nonlinear morphology, such as broken plurals).
6. Since for Hebrew we will be building one from scratch, there's going to be a lot of trials and errors. A good place to start is probably by learning from the experience of this research: Building a Shallow Arabic Morphological Analyzer in One Day. Some research work is probably going to be required in order to decide which prefix letters can be removed and which should be kept. Good thing is that Hebrew doesn't have the amount of broken plural cases as Arabic does, so this might actually work very well. Extremely light stemmers, such as the Harman "s-stemmer", can be a good starting point.
7. Simple character truncation has slim chances with Hebrew, but yet it worths a shot.
8. Since nouns and adjectives typically matter the most, the morphological lemmatisation process can ignore, low-rank or normalize difficult verbal forms. This may help both precision and index size.
9. Adjust the scoring given in the lemmatisation process (by tolerators or filters) based on relevancy tests.
10. Storing the original word (with a unique suffix, $ for example) alongside the lemma or other indexing unit and querying for it as well.
11. Since queries usually contain no real context (which is important for the disambiguation process done by the morph analyzer), it may pay using different analyzers for search and indexing, where the latter doesn't try to do much disambiguation (assuming the queried word is exact). The user can always refine his search.
12. Multi-field searches. one field with n-grams and the other with lemmas for example, using ConjunctionScorer.
13. ... whatever else we could come up with ...

Phase 5 - Wide availability

Dictionary enhancements

1. Proof-testing of hspell's data against various corporas.
2. Adding a method allowing to save the dictionary contained in the radix in a versioned format. This is necessary to allow for its easy distribution with an index and / or IR code.
3. Loading of external dictionaries (user data for example).
4. Several hspell related code updates so linguistic data is attached to the original dictionary files and isn't compiled to the application.

Future Work

1. More improvements to the morphological lemmatisation process powering the first analyzer created in this project, including: using Niqqud (where provided, even partially) for disambiguation, POS tagging during tokenization or later by processing the token vector, etc.
2. Saving terms with Niqqud (full or partial) by performing Hebrew vowel restoration. May require some serious Hebrew NLP work.
3. Looking for more researches done on Arabic IR, and trying implementing approaches they suggest with Hebrew.
4. Using Lucene's AutomatonQuery with automates that "understand" Hebrew. For an example for one, see Finite-State Registered Automata for Non-Concatenative Morphology.
5. Supporting Hebrew synonyms.