Cluster-weighted modeling

<h2 id="basic-form-of-model">Basic form of model</h2>
The procedure for cluster-weighted modeling of an input-output problem can be outlined as follows.<a class="footnote-ref" id="fnref:3" href="#fn:3">3</a> In order to construct predicted values for an output variable y from an input variable x, the modeling and calibration procedure arrives at a <a href="/facts/Joint_probability_distribution/klX2ksGY">joint probability density function</a>, p(y,x). Here the "variables" might be uni-variate, multivariate or time-series. For convenience, any model parameters are not indicated in the notation here and several different treatments of these are possible, including setting them to fixed values as a step in the calibration or treating them using a <a href="/facts/Bayesian_analysis/tEpK3zLx">Bayesian analysis</a>. The required predicted values are obtained by constructing the <a href="/facts/Conditional_probability_distribution/0eGm3P9W">conditional probability density</a> p(y|x) from which the prediction using the <a href="/facts/Conditional_expected_value/3GxG1Q3x">conditional expected value</a> can be obtained, with the <a href="/facts/Conditional_variance/Wh7R7Ype">conditional variance</a> providing an indication of uncertainty. 
The important step of the modeling is that p(y|x) is assumed to take the following form, as a <a href="/facts/Mixture_model/WWe6TJlg">mixture model</a>:

p
        (
        y
        ,
        x
        )
        =
        
          ∑
          
            j
            =
            1
          
          
            n
          
        
        
          w
          
            j
          
        
        
          p
          
            j
          
        
        (
        y
        ,
        x
        )
        ,
      
    
    {\displaystyle p(y,x)=\sum _{j=1}^{n}w_{j}p_{j}(y,x),}

where n is the number of clusters and {wj} are weights that sum to one. The functions pj(y,x) are joint probability density functions that relate to each of the n clusters. These functions are modeled using a decomposition into a conditional and a <a href="/facts/Marginal_distribution/U9XBWAd1">marginal density</a>:

p
          
            j
          
        
        (
        y
        ,
        x
        )
        =
        
          p
          
            j
          
        
        (
        y
        
          |
        
        x
        )
        
          p
          
            j
          
        
        (
        x
        )
        ,
      
    
    {\displaystyle p_{j}(y,x)=p_{j}(y|x)p_{j}(x),}

where:

<ul><li>pj(y|x) is a model for predicting y given x, and given that the input-output pair should be associated with cluster j on the basis of the value of x. This model might be a <a href="/facts/Regression_analysis/n6z5Tf7K">regression model</a> in the simplest cases.</li></ul>
<ul><li>pj(x) is formally a density for values of x, given that the input-output pair should be associated with cluster j. The relative sizes of these functions between the clusters determines whether a particular value of x is associated with any given cluster-center. This density might be a <a href="/facts/Gaussian_function/5bG8Kpp9">Gaussian function</a> centered at a parameter representing the cluster-center.</li></ul>
In the same way as for <a href="/facts/Regression_analysis/n6z5Tf7K">regression analysis</a>, it will be important to consider preliminary <a href="/facts/Data_transformation/gjQCKFIa">data transformations</a> as part of the overall modeling strategy if the core components of the model are to be simple regression models for the cluster-wise condition densities, and <a href="/facts/Normal_distribution/UapjjPyQ">normal distributions</a> for the cluster-weighting densities pj(x).

<h2 id="general-versions">General versions</h2>
The basic CWM algorithm gives a single output cluster for each input cluster. However, CWM can be extended to multiple clusters which are still associated with the same input cluster.<a class="footnote-ref" id="fnref:4" href="#fn:4">4</a> Each cluster in CWM is localized to a Gaussian input region, and this contains its own trainable local model.<a class="footnote-ref" id="fnref:5" href="#fn:5">5</a> It is recognized as a versatile inference algorithm which provides simplicity, generality, and flexibility; even when a feedforward layered network might be preferred, it is sometimes used as a "second opinion" on the nature of the training problem.<a class="footnote-ref" id="fnref:6" href="#fn:6">6</a>
The original form proposed by Gershenfeld describes two innovations:

<ul><li>Enabling CWM to work with continuous streams of data</li>
<li>Addressing the problem of local minima encountered by the CWM parameter adjustment process<a class="footnote-ref" id="fnref:7" href="#fn:7">7</a></li></ul>
CWM can be used to classify media in printer applications, using at least two parameters to generate an output that has a joint dependency on the input parameters.<a class="footnote-ref" id="fnref:8" href="#fn:8">8</a>

<h2 id="references">References</h2>

<ol>
<li id="fn:1">Gershenfeld, N (1997). "Nonlinear Inference and Cluster-Weighted Modeling". Annals of the New York Academy of Sciences. 808: 18–24. Bibcode:1997NYASA.808...18G. doi:10.1111/j.1749-6632.1997.tb51651.x. S2CID 85736539. <a href="/wiki/Bibcode_(identifier)" target="_blank">/wiki/Bibcode_(identifier)</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></li>
<li id="fn:2">Gershenfeld, N.; Schoner; Metois, E. (1999). "Cluster-weighted modelling for time-series analysis". Nature. 397 (6717): 329–332. Bibcode:1999Natur.397..329G. doi:10.1038/16873. S2CID 204990873. <a href="/wiki/Bibcode_(identifier)" target="_blank">/wiki/Bibcode_(identifier)</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></li>
<li id="fn:3">Gershenfeld, N.; Schoner; Metois, E. (1999). "Cluster-weighted modelling for time-series analysis". Nature. 397 (6717): 329–332. Bibcode:1999Natur.397..329G. doi:10.1038/16873. S2CID 204990873. <a href="/wiki/Bibcode_(identifier)" target="_blank">/wiki/Bibcode_(identifier)</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></li>
<li id="fn:4">Feldkamp, L.A.; Prokhorov, D.V.; Feldkamp, T.M. (2001). "Cluster-weighted modeling with multiclusters". IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222). Vol. 3. pp. 1710–1714. doi:10.1109/IJCNN.2001.938419. ISBN 0-7803-7044-9. S2CID 60819260. <a href="0-7803-7044-9" target="_blank">0-7803-7044-9</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></li>
<li id="fn:5">Boyden, Edward S. "Tree-based Cluster Weighted Modeling: Towards A Massively Parallel Real-Time Digital Stradivarius" (PDF). Cambridge, MA: MIT Media Lab. <a href="http://edboyden.org/violin.pdf" target="_blank">http://edboyden.org/violin.pdf</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></li>
<li id="fn:6">Prokhorov, A New Approach to Cluster-Weighted Modeling Danil V.; Lee A. Feldkamp; Timothy M. Feldkamp. "A New Approach to Cluster-Weighted Modeling" (PDF). Dearborn, MI: Ford Research Laboratory. <a href="http://home.comcast.net/~dvp/cwm.pdf" target="_blank">http://home.comcast.net/~dvp/cwm.pdf</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></li>
<li id="fn:7">Prokhorov, A New Approach to Cluster-Weighted Modeling Danil V.; Lee A. Feldkamp; Timothy M. Feldkamp. "A New Approach to Cluster-Weighted Modeling" (PDF). Dearborn, MI: Ford Research Laboratory. <a href="http://home.comcast.net/~dvp/cwm.pdf" target="_blank">http://home.comcast.net/~dvp/cwm.pdf</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></li>
<li id="fn:8">Gao, Jun; Ross R. Allen (2003-07-24). "CLUSTER-WEIGHTED MODELING FOR MEDIA CLASSIFICATION". Palo Alto, CA: World Intellectual Property Organization. Archived from the original on 2012-12-12. <a href="https://archive.today/20121212003528/http://www.wipo.int/pctdb/en/wo.jsp?wo=2003059630" target="_blank">https://archive.today/20121212003528/http://www.wipo.int/pctdb/en/wo.jsp?wo=2003059630</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></li>
</ol>

Cluster-weighted modeling open-in-new

Cluster-weighted modeling