Parameterestimationfortextanalysis资源-CSDN文库

5星 · 超过95%的资源需积分: 10 170 浏览量 2014-10-17 09:55:48 上传评论收藏 443KB PDF 举报

### 参数估计在文本分析中的应用 #### 引言本文档旨在回顾离散域中贝叶斯参数估计的基础，这对于理解基于主题的文本分析方法（如概率潜在语义分析(PLSA)、潜在狄利克雷分配(LDA)等）的工作原理至关重要。尽管这些方法在研究界广为接受，但似乎缺乏一本全面介绍这些方法基础概念的书籍或入门论文。大多数已知的文本都以高斯分布为例进行阐述，这与离散分布的表述有所不同。此外，一些关于主题模型的优秀入门资料（例如[StGr07]）为了清晰地呈现概念而省略了算法细节和其他背景信息。因此，本文将通过一系列简单的二进制数据示例系统性地介绍参数估计的基本概念（第2节）。接着，我们将介绍共轭性的概念，并回顾文本领域中最常用的概率分布（第3节）。通过将共轭性与其实际应用中的共轭对结合在一起介绍，我们能够直观地解释所选择分布的合理性。第4节将引入贝叶斯网络作为一种图形语言来描述系统及其概率模型。掌握了这些基本概念后，我们将在第5节中介绍潜在狄利克雷分配(LDA)，这是一种灵活的方法用于估计文本属性。我们还将通过LDA的例子展示吉布斯采样作为近似推理的一种简单手段。 #### 参数估计的概念在讨论具体的应用之前，让我们先了解一下参数估计的基本概念。参数估计是在给定一组观测数据的情况下，确定某个概率模型中未知参数的过程。这种估计可以分为几种类型：最大似然估计(Maximum Likelihood Estimation, MLE)、后验估计(A Posteriori Estimation, APE)以及贝叶斯估计(Bayesian Estimation)。 **最大似然估计(MLE)**是最常用的方法之一，其目标是找到使得观测数据出现概率最大的参数值。这种方法通常基于数据独立同分布的假设，且假设参数固定不变。 **后验估计(APE)**是一种更复杂的估计方法，它考虑了先验信息。后验估计利用先验分布和观测数据的似然函数来更新参数的概率分布，从而得到参数的估计值。 **贝叶斯估计**则是更一般的方法，它不仅考虑了观测数据的似然函数，还考虑了先验信息，从而给出参数的后验分布。贝叶斯估计允许我们在估计过程中加入主观信念或者先前的经验，因此更加灵活。 #### 共轭性和概率分布在文本分析中，我们经常遇到几种关键的概率分布，包括多项式分布、狄利克雷分布和二项式分布。这些分布之间的关系可以通过共轭性的概念来描述。 - **多项式分布**：当样本来自一个具有多个可能结果的实验时，多项式分布描述了每个结果出现的次数。 - **狄利克雷分布**：狄利克雷分布是多项式分布的共轭先验。这意味着如果我们用狄利克雷分布作为多项式分布的参数的先验分布，那么在给定数据后，后验分布依然是狄利克雷分布。 - **二项式分布**：二项式分布描述了在特定数量的独立伯努利试验中成功次数的概率分布。 - **贝塔分布**：贝塔分布是二项式分布的共轭先验。共轭性的一个重要优点是简化了计算，尤其是在使用贝叶斯估计时。通过选择适当的共轭先验分布，我们可以避免复杂的积分运算，直接获得解析解。 #### 贝叶斯网络贝叶斯网络是一种图形模型，用于表示变量间的概率依赖关系。在网络中，节点代表随机变量，边表示变量之间的条件依赖关系。贝叶斯网络不仅可以直观地展示复杂系统的结构，还能有效地进行概率推理。例如，在文本分析中，我们可以构建一个贝叶斯网络来表示文档的主题分布、词的出现概率等信息。 #### 潜在狄利克雷分配(LDA) 潜在狄利克雷分配(LDA)是一种用于文档建模的生成式概率模型，它能够自动发现文档集合中的主题结构。LDA的核心思想是假设每个文档由多个主题构成，而每个主题则由一系列单词的概率分布表示。LDA模型通过引入狄利克雷分布来处理主题和单词分布的不确定性。 LDA模型的主要步骤包括： 1. **主题-文档分布**: 对于每个文档，从狄利克雷分布中抽取一个主题分布。 2. **主题-词分布**: 对于每个主题，从狄利克雷分布中抽取一个词分布。 3. **文档生成**: 对于文档中的每个单词： - 从该文档的主题分布中抽取一个主题； - 从选中的主题的词分布中抽取一个单词。为了估计LDA模型中的参数，通常采用吉布斯采样的方法。吉布斯采样是一种基于马尔可夫链蒙特卡洛(MCMC)的算法，它通过迭代地从条件后验分布中抽样来逼近联合后验分布。在LDA中，吉布斯采样主要用于估计主题-文档分布和主题-词分布。 #### 总结本文档深入探讨了参数估计在文本分析中的应用，重点介绍了最大似然估计、后验估计以及贝叶斯估计等核心概念。通过共轭性和贝叶斯网络的介绍，我们了解了如何在文本分析中有效利用概率模型。通过潜在狄利克雷分配(LDA)的例子展示了如何将这些理论应用于实践，特别是吉布斯采样在近似推理中的应用。这些方法对于理解和开发先进的文本分析技术至关重要。

资源推荐

资源详情

资源评论

Parameter estimation for text analysis

Gregor Heinrich

Technical Note

vsonix GmbH + University of Leipzig, Germany

gregor@vsonix.com

Abstract. Presents parameter estimation methods common with discrete proba-

bility distributions, which is of particular interest in text modeling. Starting with

maximum likelihood, a posteriori and Bayesian estimation, central concepts like

conjugate distributions and Bayesian networks are reviewed. As an application,

the model of latent Dirichlet allocation (LDA) is explained in detail with a full

derivation of an approximate inference algorithm based on Gibbs sampling, in-

cluding a discussion of Dirichlet hyperparameter estimation.

History: version 1: May 2005, version 2.4: August 2008.

1 Introduction

This technical note is intended to review the foundations of Bayesian parameter esti-

mation in the discrete domain, which is necessary to understand the inner workings of

topic-based text analysis approaches like probabilistic latent semantic analysis (PLSA)

[Hofm99], latent Dirichlet allocation (LDA) [BNJ02] and other mixture models of

count data. Despite their general acceptance in the research community, it appears that

there is no common book or introductory paper that ﬁlls this role: Most known texts use

examples from the Gaussian domain, where formulations appear to be rather diﬀerent.

Other very good introductory work on topic models (e.g., [StGr07]) skips details of

algorithms and other background for clarity of presentation.

We therefore will systematically introduce the basic concepts of parameter estima-

tion with a couple of simple examples on binary data in Section 2. We then will in-

troduce the concept of conjugacy along with a review of the most common probability

distributions needed in the text domain in Section 3. The joint presentation of conjugacy

with associated real-world conjugate pairs directly justiﬁes the choice of distributions

introduced. Section 4 will introduce Bayesian networks as a graphical language to de-

scribe systems via their probabilistic models.

With these basic concepts, we present the idea of latent Dirichlet allocation (LDA)

in Section 5, a ﬂexible model to estimate the properties of text. On the example of

LDA, the usage of Gibbs sampling is shown as a straight-forward means of approximate

inference in Bayesian networks. Two other important aspects of LDA are discussed

afterwards: In Section 6, the inﬂuence of LDA hyperparameters is discussed and an

estimation method proposed, and in Section 7, methods are presented to analyse LDA

models for querying and evaluation.

The probability of a new observation ˜x given the data X can now be found using the

approximation

p( ˜x|X) =

ϑ∈Θ

p( ˜x|ϑ) p(ϑ|X) dϑ (6)

≈

ϑ∈Θ

p( ˜x|

) p(ϑ|X) dϑ = p( ˜x|

), (7)

that is, the next sample is anticipated to be distributed with the estimated parameters

As an example, consider a set C of N Bernoulli experiments with unknown param-

eter p, e.g., realised by tossing a deformed coin. The Bernoulli density function for the

r.v. C for one experiment is:

p(C=c|p) = p

(1 − p)

1−c

, Bern(c|p) (8)

where we deﬁne c=1 for heads and c=0 for tails

Building an ML estimator for the parameter p can be done by expressing the (log)

likelihood as a function of the data:

L = log

i=1

p(C=c

|p) =

i=1

log p(C=c

|p) (9)

= n

(1)

log p(C=1|p) + n

(0)

log p(C=0|p)

= n

(1)

log p + n

(0)

log(1 − p) (10)

where n

(c)

is the number of times a Bernoulli experiment yielded event c. Diﬀerentiating

with respect to (w.r.t.) the parameter p yields:

∂L

∂p

(1)

−

(0)

1 − p

= 0 ⇔ ˆp

(1)

+ n

(0)

(1)

, (11)

which is simply the ratio of heads results to the total number of samples. To put some

numbers into the example, we could imagine that our coin is strongly deformed, and

after 20 trials, we have n

(1)

=12 times heads and n

(0)

=8 times tails. This results in an ML

estimation of of ˆp

= 12/20 = 0.6.

2.2 Maximum a posteriori estimation

Maximum a posteriori (MAP) estimation is very similar to ML estimation but allows

to include some a priori belief on the parameters by weighting them with a prior dis-

tribution p(ϑ). The name derives from the objective to maximise the posterior of the

parameters given the data:

MAP

= argmax

p(ϑ|X). (12)

The ML estimate

is considered a constant, and the integral over the parameters given the

data is the total probability that integrates to one.

The notation in Eq. 8 is somewhat peculiar because it makes use of the values of c to “ﬁlter”

the respective parts in the density function and additionally uses these numbers to represent

disjoint events.

By using Bayes’ rule (Eq. 1), this can be rewritten to:

MAP

= argmax

p(X|ϑ)p(ϑ)

p(X)



p(X) , f (ϑ)

= argmax

p(X|ϑ)p(ϑ) = argmax

{L(ϑ|X) + log p(ϑ)}

= argmax

x∈X

log p(x|ϑ) + log p(ϑ)

. (13)

Compared to Eq. 4, a prior distribution is added to the likelihood. In practice, the prior

p(ϑ) can be used to encode extra knowledge as well as to prevent overﬁtting by enforc-

ing preference to simpler models, which is also called Occam’s razor

With the incorporation of p(ϑ), MAP follows the Bayesian approach to data mod-

elling where the parameters ϑ are thought of as r.v.s. With priors that are parametrised

themselves, i.e., p(ϑ) := p(ϑ|α) with hyperparameters α, the belief in the anticipated

values of ϑ can be expressed within the framework of probability

, and a hierarchy of

parameters is created.

MAP parameter estimates can be found by maximising the term L(ϑ|X) + log p(ϑ),

similar to Eq. 5. Analogous to Eq. 7, the probability of a new observation, ˜x, given the

data, X, can be approximated using:

p( ˜x|X) ≈

ϑ∈Θ

p( ˜x|

MAP

) p(ϑ|X) dϑ = p( ˜x|

MAP

). (14)

Returning to the simplistic demonstration on ML, we can give an example for the

MAP estimator. Consider the above experiment, but now there are values for p that

we believe to be more likely, e.g., we believe that a coin usually is fair. This can be

expressed as a prior distribution that has a high probability around 0.5. We choose the

beta distribution:

p(p|α, β) =

B(α, β)

α−1

(1 − p)

β−1

, Beta(p|α, β), (15)

with the beta function B(α, β) =

Γ(α)Γ(β)

Γ(α+β)

. The function Γ(x) is the Gamma function,

which can be understood as a generalisation of the factorial to the domain of real num-

bers via the identity x! = Γ(x + 1). The beta distribution supports the interval [0,1] and

therefore is useful to generate normalised probability values. For a graphical represen-

tation of the beta probability density function (pdf), see Fig. 1. As can be seen, with

diﬀerent parameters the distribution takes on quite diﬀerent pdfs.

In our example, we believe in a fair coin and set α = β = 5, which results in a

distribution with a mode (maximum) at 0.5. The optimisation problem now becomes

Pluralitas non est ponenda sine necessitate = Plurality should not be posited without necessity.

Occam’s razor is also called the principle of parsimony.

Belief is not identical to probability, which is one of the reasons why Bayesian approaches are

disputed by some theorists despite their practical importance.

剩余30页未读，继续阅读

评论收藏

内容反馈

xiaogogogo

2015-12-10

很好，推导过程很清楚
wendycherry

2015-02-03

还不错，值得看看
宿命的随机数

2016-03-29

下载看了以后，还是不太懂，看来我的基本功不行
陈呆蕉

2015-06-23

真是值得看看的！
laraGuo

2015-05-12

文章写得很出色学习LDA必看

前往

页

littlekideee

粉丝: 4
资源: 7

Parameter estimation for text analysis

Parameter estimation for text analysis.pdf

Parameter estimation and inverse problems

Parameter estimation for rigid body after micro Doppler removal based on L-statistical in the radar analysis

Parameter estimation using EA

Parameter-estimation-for-text-analysis， 2009版

state estimation for robotics

Parameter Estimation and Inverse Problems 的CD程序

Classification, Parameter Estimation, and State Estimation An Engineering Approach Using MATLAB

Efficient parameter estimation for the accelerated failure time model with clustered and censored data

Inverse Problem Theory and Methods for Model Parameter Estimation

Parameter Estimation and Inverse Problems, Second Edition对应的第7章代码

Classification,Parameter Estimation and State Estimation with MATLAB (2004)

DINA Model and Parameter Estimation：A Didactic.pdf

Principles of Signal Detection and Parameter Estimation

State of Charge and Parameter Estimation of Electric Vehicle Batteries

Parameter Estimation and Hypothesis Testing

parameter estimation of mimo ofdm flat fading channels

Model Parameter Estimation and Uncertainty

State Estimation for Robotics20181128

STATE ESTIMATION FOR ROBOTICS pdf

L0 Regularized Stationary-time Estimation for Crowd Analysis

PARAMETER ESTIMATION AND INVERSE PROBLEMS(2013)

仿真电路以及操作方法

python大作业 含爬虫、数据可视化、地图、报告、及源码（整和为一个文件）（2014-2020全国各地区原油加工量）.rar

【纯干货啊】华为IPD流程管理(完整版).pptx

最新资源

python大作业含爬虫、数据可视化、地图、报告、及源码（整和为一个文件）（2014-2020全国各地区原油加工量）.rar