TF*IDF Ranker allows the user getting scores for each term of input text according to the classic formula and its modified version. This is the classic formula that is widely used in term weighting techniques:
w(ij)=tf(ij)*log2 N/n
where
wij = weight of Term Tj in Document Di
tfij = frequency of Term Tj in Document Di
N = number of Documents in corpus
n = number of Documents where term Tj occurs at least once
Once you get a list of terms with their weights arranged in descending order you can copy it to some external editor to use it in various ways, for example to filter stop-words, i.e. words with zero or low scores or, on the contrary, use most salient words that have highest weights to represent the content of input text.
A drawback of this formula is that terms that occur in input text but that cannot be found in corpus get zero scores. In many cases such terms may be important for text understanding. For example a scientist may describe his invention introducing some newly coined terms, or a writer may invent neologisms that are not registered in existing corpora. That is why we modified the classic formula: if a word occurs in input text but doesn’t occur in corpus n in the formula is assigned the value “1”, rather than “0”.
The main problem with this TF*IDF technique is number of texts in corpus, i.e. the value of N. Who many texts must the corpus include to be representative enough? No formal criteria have been developed so far and we work at this problem to suggest a solution in the nearest future.
TF*IDF Ranker works on Windows machines and requires .net framework. It supports English texts in .txt format.
How to use
1) With add button add texts to create a corpus. 2) Upload a text to analyze. 3) Select a classic formula or a modified formula. Classic version is a default option. 4). Click analyze.
4) Get a list of terms arranged in decending order of their weights.
5) Copy the list to an external editor for further processing.
Requirements
Windows 98/NT/2000/XP/Vista/7
300 MHz processor
0,5 MB free disc space
