Probabilistic latent semantic analysis

<h2 id="model">Model</h2>

<p>Considering observations in the form of co-occurrences 
  
    
      
        (
        w
        ,
        d
        )
      
    
    {\displaystyle (w,d)}
  
 of words and documents, PLSA models the probability of each co-occurrence as a mixture of conditionally independent <a href="/facts/Multinomial_distribution/y58v1p9J">multinomial distributions</a>:
</p>

P
        (
        w
        ,
        d
        )
        =
        
          ∑
          
            c
          
        
        P
        (
        c
        )
        P
        (
        d
        
          |
        
        c
        )
        P
        (
        w
        
          |
        
        c
        )
        =
        P
        (
        d
        )
        
          ∑
          
            c
          
        
        P
        (
        c
        
          |
        
        d
        )
        P
        (
        w
        
          |
        
        c
        )
      
    
    {\displaystyle P(w,d)=\sum _{c}P(c)P(d|c)P(w|c)=P(d)\sum _{c}P(c|d)P(w|c)}

<p>with 
  
    
      
        c
      
    
    {\displaystyle c}
  
 being the words' topic. Note that the number of topics is a hyperparameter that must be chosen in advance and is not estimated from the data. The first formulation is the <i>symmetric</i> formulation, where 
  
    
      
        w
      
    
    {\displaystyle w}
  
 and 
  
    
      
        d
      
    
    {\displaystyle d}
  
 are both generated from the latent class 
  
    
      
        c
      
    
    {\displaystyle c}
  
 in similar ways (using the conditional probabilities 
  
    
      
        P
        (
        d
        
          |
        
        c
        )
      
    
    {\displaystyle P(d|c)}
  
 and 
  
    
      
        P
        (
        w
        
          |
        
        c
        )
      
    
    {\displaystyle P(w|c)}
  
), whereas the second formulation is the <i>asymmetric</i> formulation, where, for each document 
  
    
      
        d
      
    
    {\displaystyle d}
  
, a latent class is chosen conditionally to the document according to 
  
    
      
        P
        (
        c
        
          |
        
        d
        )
      
    
    {\displaystyle P(c|d)}
  
, and a word is then generated from that class according to 
  
    
      
        P
        (
        w
        
          |
        
        c
        )
      
    
    {\displaystyle P(w|c)}
  
.  Although we have used words and documents in this example, the co-occurrence of any couple of discrete variables may be modelled in exactly the same way.
</p><p>So, the number of parameters is equal to 
  
    
      
        c
        d
        +
        w
        c
      
    
    {\displaystyle cd+wc}
  
. The number of parameters grows linearly with the number of documents.  In addition, although PLSA is a generative model of the documents in the collection it is estimated on, it is not a generative model of new documents.
</p><p>Their parameters are learned using the <a href="/facts/EM_algorithm/32P2bdBc">EM algorithm</a>.
</p>
<h2 id="application">Application</h2>
<p>PLSA may be used in a discriminative setting, via <a href="/facts/Fisher_kernel/wIpRJErI">Fisher kernels</a>.<a class="footnote-ref" id="fnref:1" href="#fn:1"><sup>1</sup></a>
</p><p>PLSA has applications in <a href="/facts/Information_retrieval/MFSmpeNw">information retrieval</a> and <a href="/facts/Information_filtering/BjzDBTkt">filtering</a>, <a href="/facts/Natural_language_processing/1hjMKsSN">natural language processing</a>, <a href="/facts/Machine_learning/e0w0XJTu">machine learning</a> from text, <a href="/facts/Bioinformatics/D5x2L8ee">bioinformatics</a>,<a class="footnote-ref" id="fnref:2" href="#fn:2"><sup>2</sup></a> and related areas.
</p><p>It is reported that the aspect model used in the probabilistic latent semantic analysis has severe <a href="/facts/Overfitting/5xnFLcMg">overfitting</a> problems.<a class="footnote-ref" id="fnref:3" href="#fn:3"><sup>3</sup></a>
</p>
<h2 id="extensions">Extensions</h2>
<ul><li>Hierarchical extensions:
<ul><li>Asymmetric: MASHA ("Multinomial ASymmetric Hierarchical Analysis")<a class="footnote-ref" id="fnref:4" href="#fn:4"><sup>4</sup></a></li>
<li>Symmetric: HPLSA ("Hierarchical Probabilistic Latent Semantic Analysis")<a class="footnote-ref" id="fnref:5" href="#fn:5"><sup>5</sup></a></li></ul></li></ul>
<ul><li>Generative models: The following models have been developed to address an often-criticized shortcoming of PLSA, namely that it is not a proper generative model for new documents.
<ul><li><a href="/facts/Latent_Dirichlet_allocation/BUWlwKWt">Latent Dirichlet allocation</a> – adds a <a href="/facts/Dirichlet_distribution/Qq13f5g2">Dirichlet</a> prior on the per-document topic distribution</li></ul></li>
<li>Higher-order data: Although this is rarely discussed in the scientific literature, PLSA extends naturally to higher order data (three modes and higher), i.e. it can model co-occurrences over three or more variables.  In the symmetric formulation above, this is done simply by adding conditional probability distributions for these additional variables. This is the probabilistic analogue to non-negative tensor factorisation.</li></ul>
<h2 id="history">History</h2>
<p>This is an example of a <a href="/facts/Latent_class_model/LaQ7oJDY">latent class model</a> (see references therein), and it is related<a class="footnote-ref" id="fnref:6" href="#fn:6"><sup>6</sup></a><a class="footnote-ref" id="fnref:7" href="#fn:7"><sup>7</sup></a>  to <a href="/facts/Non-negative_matrix_factorization/DIBLH8B0">non-negative matrix factorization</a>. The present terminology was coined in 1999 by Thomas Hofmann.<a class="footnote-ref" id="fnref:8" href="#fn:8"><sup>8</sup></a>
</p>
<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Compound_term_processing/AssJecnj">Compound term processing</a></li>
<li><a href="/facts/Pachinko_allocation/wzQUMSEl">Pachinko allocation</a></li>
<li><a href="/facts/Vector_space_model/WC1LqLA5">Vector space model</a></li></ul>
<h2 id="references-and-notes">References and notes</h2>

<h2 id="external-links">External links</h2>
<ul><li><a href="https://web.archive.org/web/20050120213347/http://www.cs.brown.edu/people/th/papers/Hofmann-UAI99.pdf">Probabilistic Latent Semantic Analysis</a></li>
<li><a href="https://web.archive.org/web/20170717235351/http://www.semanticquery.com/archive/semanticsearchart/researchpLSA.html">Complete PLSA DEMO in C#</a></li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1"><p>Thomas Hofmann, Learning the Similarity of Documents : an information-geometric approach to document retrieval and categorization, Advances in Neural Information Processing Systems 12, pp-914-920, MIT Press, 2000 <a href="https://papers.nips.cc/paper/1654-learning-the-similarity-of-documents-an-information-geometric-approach-to-document-retrieval-and-categorization.pdf" target="_blank">https://papers.nips.cc/paper/1654-learning-the-similarity-of-documents-an-information-geometric-approach-to-document-retrieval-and-categorization.pdf</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li>
<li id="fn:2"><p>Pinoli, Pietro; et, al. (2013). "Enhanced probabilistic latent semantic analysis with weighting schemes to predict genomic annotations". Proceedings of IEEE BIBE 2013. The 13th IEEE International Conference on BioInformatics and BioEngineering. IEEE. pp. 1–4. doi:10.1109/BIBE.2013.6701702. ISBN 978-147993163-7. <a href="978-147993163-7" target="_blank">978-147993163-7</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li>
<li id="fn:3"><p>Blei, David M.; Andrew Y. Ng; Michael I. Jordan (2003). "Latent Dirichlet Allocation" (PDF). Journal of Machine Learning Research. 3: 993–1022. doi:10.1162/jmlr.2003.3.4-5.993. <a href="http://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf" target="_blank">http://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li>
<li id="fn:4"><p>Alexei Vinokourov and Mark Girolami, A Probabilistic Framework for the Hierarchic Organisation and Classification of Document Collections, in Information Processing and Management, 2002 <a href="http://citeseer.ist.psu.edu/rd/30973750,455249,1,0.25,Download/http://citeseer.ist.psu.edu/cache/papers/cs/22961/http:zSzzSzcis.paisley.ac.ukzSzvino-ci0zSzvinokourov_masha.pdf/vinokourov02probabilistic.pdf" target="_blank">http://citeseer.ist.psu.edu/rd/30973750,455249,1,0.25,Download/http://citeseer.ist.psu.edu/cache/papers/cs/22961/http:zSzzSzcis.paisley.ac.ukzSzvino-ci0zSzvinokourov_masha.pdf/vinokourov02probabilistic.pdf</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li>
<li id="fn:5"><p>Eric Gaussier, Cyril Goutte, Kris Popat and Francine Chen,
A Hierarchical Model for Clustering and Categorising Documents Archived 2016-03-04 at the Wayback Machine, in "Advances in Information Retrieval -- Proceedings of the 24th BCS-IRSG European Colloquium on IR Research (ECIR-02)", 2002 <a href="http://www.xrce.xerox.com/Research-Development/Publications/2002-004" target="_blank">http://www.xrce.xerox.com/Research-Development/Publications/2002-004</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li>
<li id="fn:6"><p>Chris Ding, Tao Li, Wei Peng (2006). "Nonnegative Matrix Factorization and Probabilistic Latent Semantic Indexing: Equivalence Chi-Square Statistic, and a Hybrid Method. AAAI 2006"
 <a href="http://www.aaai.org/Papers/AAAI/2006/AAAI06-055.pdf" target="_blank">http://www.aaai.org/Papers/AAAI/2006/AAAI06-055.pdf</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li>
<li id="fn:7"><p>Chris Ding, Tao Li, Wei Peng (2008). "On the equivalence between Non-negative Matrix Factorization and Probabilistic Latent Semantic Indexing" <a href="http://www.sciencedirect.com/science/article/pii/S0167947308000145" target="_blank">http://www.sciencedirect.com/science/article/pii/S0167947308000145</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li>
<li id="fn:8"><p>Thomas Hofmann, Probabilistic Latent Semantic Indexing, Proceedings of the Twenty-Second Annual International SIGIR Conference on Research and Development in Information Retrieval (SIGIR-99), 1999 <a href="https://arxiv.org/abs/1301.6705" target="_blank">https://arxiv.org/abs/1301.6705</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></p></li>
</ol>

Probabilistic latent semantic analysis open-in-new

Probabilistic latent semantic analysis