CS598CXZ Assignment #5: Probabilistic Retrieval Models
(due 11:59pm, Tuesday, Oct. 14, 2014 )

This assignment tests your understanding and analysis of the mathematical background underlying probabilistic retrieval models. In particular, you will be walking through a series of hands-on problems designed to examine and reinforce your abillity to recognize and appreciate how the retrieval formulas are constructed, what each component of the formulas is used for, and why the retrieval models could achieve satisfactory retrieval results. Remember that you are expected to complete the assignment on your own. Please submit your solutions via Compass.

  1. [50 points] Classic Probabilistic Retrieval Model.
    1. [20/50 points] In the derivation of the Robertson-Sparck-Jones (RSJ) model, a multi-variate Bernoulli model was used to model term presence/absence in a relevant document and a non-relevant document. Suppose, we change the model to a multinomial model (see the slide that covers both models for computing query likelihood). Using a similar independence assumption as we used in deriving RSJ, show that ranking based on probability that a document is relevant to a query Q, i.e., p(R=1|D,Q), is equivalent to ranking based on the following formula:
                  \---            p(w|Q,R=1)
      score(Q,D) = >   c(w,D) log ----------
                  /---            p(w|Q,R=0)
                 w in V
      
      where the sum is taken over all the words in our vocabulary (denoted by V), and c(w,D) is the count of word w in document D (i.e., how many times w occurs in D). How many parameters are there in such a retrieval model that we have to estimate?
    2. [5/50 points] The retrieval function above won't work unless we can estimate all the parameters. Suppose we use the entire collection C={D1,...,Dn} as an approximation of the examples of non-relevant documents. Give the formula for the Maximum Likelihood estimate of p(w|Q,R=0).
    3. [5/50 points] Suppose we use the query as the only example of a relevant document. Give the formula for the Maximum Likelihood estimate of p(w|Q,R=1) based on this single example of relevant document.
    4. [5/50 points] One problem with the maximum likelihood estimate of p(w|Q,R=1) is that many words would have zero probability, which limits its accuracy of modeling words in relevant documents. Give the formula for smoothing this maximum likelihood estimate using fixed coefficient linear interpolation (i.e., Jelinek-Mercer) with a collection language model.
    5. [15/50 points] With the two estimates you proposed, i.e., the estimate of p(w|Q,R=0) based on the collection and the estimate of p(w|Q,R=1) based on the query with smoothing, your should now have a retrieval function that can be used to compute a score for any document D and any query Q. Write down your retrieval function by plugging in the two estimates. Can your retrieval function capture the three major retrieval heuristics (i.e., TF, IDF, and document length normalization)? How?
  2. [50 points] Language Models
    1. [20/50 points] Show that if we use the query-likelihood scoring method (i.e., p(Q|D)) and the Jelinek-Mercer smoothing method (i.e., fixed co-efficient interpolation with smoothing parameter a) for retrieval, we can rank documents based on the following scoring function:
                  \---------                  (1-a)*c(w,D)
      score(Q,D) = >         c(w,Q) log (1 + --------------)
                  /---------                 a*p(w|REF)*|D|
                 w in Q and D
      
      where the sum is taken over all the matched query terms in D, |D| is the document length, c(w,D) is the count of word w in document D (i.e., how many times w occurs in D), c(w,Q) is the count of word w in Q, "a" is the smoothing parameter (i.e., the same as lambda on the lecture slide), and p(w|REF) is the probability of word w given by the reference language model estimated using the whole collection.
    2. [10/50 points] This scoring function above can also be interpreted as a vector space model. If we make this interpretation, what would be the query vector? What would be the document vector? What would be the similarity function? Does the term weight in the document vector capture TF-IDF weighting and document length normalization heuristics? Why?
    3. [20/50 points] One way to check whether a retrieval function would over-penalize a long document is to do the following: Imagine that we duplicate a document D k times to generate a new document D' so that each term would have k times more occurrences (naturally, the new document D' is also k times longer than the original D). Intuitively, the score for D' shouldn't be less than the score of D for any query. Check if this is true for the query likelihood retrieval function with both Jelinek-Mercer smoothing and Dirichlet prior smoothing, respectively.