Probability Estimation (estimate)

Probability estimators compute probabilities of values of class variable. They come in two flavours:

  1. for unconditional probabilities (p(C=c), where c is a class) and
  2. for conditional probabilities (p(C=c|V=v), where v is a feature value).

A duality much like the one between learners and classifiers exists between probability estimator constructors and probability estimators: when a probability estimator constructor is called with data, it constructs a probability estimator that can then be called with a value of class variable to obtain a probability of that value. This duality is mainly needed to enable probability estimation for continuous variables, where it is not possible to generate a list of probabilities of all possible values in advance.

First, probability estimation constructors for common probability estimation techniques are enumerated. Base classes, knowledge of which is needed to develop new techniques, are described later in this document.

Probability Estimation Constructors

class Orange.statistics.estimate.RelativeFrequency

Bases: EstimatorConstructor

Compute distribution using relative frequencies of classes.

Return type:EstimatorFromDistribution
class Orange.statistics.estimate.Laplace

Bases: EstimatorConstructor

Use Laplace estimation to compute distribution from frequencies of classes:

p(c) = \\frac{Nc+1}{N+n}

where Nc is number of occurrences of an event (e.g. number of instances in class c), N is the total number of events (instances) and n is the number of different events (classes).

Return type:EstimatorFromDistribution
class Orange.statistics.estimate.M

Bases: EstimatorConstructor

__init__(m)
Parameters:m (int) – Parameter for m-estimation.

Use m-estimation to compute distribution from frequencies of classes:

p(c) = \\frac{Nc+m*ap(c)}{N+m}

where Nc is number of occurrences of an event (e.g. number of instances in class c), N is the total number of events (instances) and ap(c) is the prior probability of event (class) c.

Return type:EstimatorFromDistribution
class Orange.statistics.estimate.Kernel

Bases: EstimatorConstructor

__init__(min_impact, smoothing, n_points)
Parameters:
  • min_impact (float) – A requested minimal weight of a point (default: 0.01); points with lower weights won’t be taken into account.
  • smoothing (float) – Smoothing factor (default: 1.144).
  • n_points (int) – Number of points for the interpolating curve. If negative, say -3 (default), 3 points will be inserted between each data points.

Compute probabilities for continuous variable for certain number of points using Gaussian kernels. The resulting point-wise continuous distribution is stored as Continuous.

Probabilities are always computed at all points that are present in the data (i.e. the existing values of the continuous feature). If n_points is positive and greater than the number of existing data points, additional points are inserted between the existing points to achieve the required number of points. Approximately equal number of new points is inserted between each adjacent existing point each data points. If n_points is negative, its absolute value determines the number of points to be added between each two data points.

Return type:EstimatorFromDistribution
class Orange.statistics.estimate.Loess

Bases: EstimatorConstructor

__init__(window_proportion, n_points)
Parameters:
  • window_proportion (float) – A proportion of points in a window.
  • n_points (int) – Number of points for the interpolating curve. If negative, say -3 (default), 3 points will be inserted between each data points.

Prepare a probability estimator that computes probability at point x as weighted local regression of probabilities for points in the window around this point.

The window contains a prescribed proportion of original data points. The window is as symmetric as possible in the sense that the leftmost point in the window is approximately as far from x as the rightmost. The number of points to the left of x might thus differ from the number of points to the right.

Points are weighted by bi-cubic weight function; a weight of point at x' is (1-|t|^3)^3, where t is (x-x'>)/h and h is the distance to the farther of the two window edge points.

Return type:EstimatorFromDistribution
class Orange.statistics.estimate.ConditionalLoess

Bases: ConditionalEstimatorConstructor

__init__(window_proportion, n_points)
Parameters:
  • window_proportion (float) – A proportion of points in a window.
  • n_points (int) – Number of points for the interpolating curve. If negative, say -3 (default), 3 points will be inserted between each data points.

Construct a conditional probability estimator, in other aspects similar to the one constructed by Loess.

Return type:ConditionalEstimatorFromDistribution.

Base classes

All probability estimators are derived from two base classes: one for unconditional and the other for conditional probability estimation. The same is true for probability estimator constructors.

class Orange.statistics.estimate.EstimatorConstructor

Constructor of an unconditional probability estimator.

__call__([distribution[, prior]][, instances[, weight_id]])
Parameters:

If distribution is given, it can be followed by prior class distribution. Similarly, instances can be followed by with the ID of meta attribute with instance weights. (Hint: to pass a prior distribution and instances, but no distribution, just pass None for the latter.) When both, distribution and instances are given, it is up to constructor to decide what to use.

Note

The instances and weight_id argument are at the moment only used by ConditionalByRows. The rest of the builtin constructors require that distribution is given.

class Orange.statistics.estimate.Estimator
supports_discrete

Tells whether the estimator can handle discrete attributes.

supports_continuous

Tells whether the estimator can handle continuous attributes.

__call__([value])

If value is given, return the probability of the value.

Return type:float

If the value is omitted, an attempt is made to return a distribution of probabilities for all values.

Return type:Distribution (usually Discrete for discrete and Continuous for continuous) or NoneType
class Orange.statistics.estimate.ConditionalEstimatorConstructor

Constructor of a conditional probability estimator.

__call__([table[, prior]][, instances[, weight_id]])
Parameters:

If distribution is given, it can be followed by prior class distribution. Similarly, instances can be followed by with the ID of meta attribute with instance weights. (Hint: to pass a prior distribution and instances, but no distribution, just pass None for the latter.) When both, distribution and instances are given, it is up to constructor to decide what to use.

Note

The instances and weight_id argument are at the moment only used by ConditionalByRows. The rest of the builtin constructors require that table is given.

class Orange.statistics.estimate.ConditionalEstimator

As a counterpart of Estimator, this estimator can return conditional probabilities.

__call__([[value], condition_value])

When given two values, it returns a probability of p(value|condition).

Return type:float

When given only one value, it is interpreted as condition; the estimator attempts to return a distribution of conditional probabilities for all values.

Return type:Distribution (usually Discrete for discrete and Continuous for continuous) or NoneType

When called without arguments, it returns a matrix containing probabilities p(value|condition) for each possible value and condition (a contingency table); condition is used as outer variable.

Return type:Orange.statistics.contingency.Table or NoneType

If estimator cannot return precomputed distributions and/or contingencies, it returns None.

Common Components

class Orange.statistics.estimate.EstimatorFromDistribution

Bases: Estimator

Probability estimator constructors that compute probabilities for all values in advance return this estimator with calculated quantities in the probabilities attribute.

probabilities

A precomputed list of probabilities.

__call__([value])

If value is given, return the probability of the value. For discrete variables, every value has an entry in the probabilities attribute. For continuous variables, a linear interpolation between two nearest points is used to compute the probability.

Return type:float

If the value is omitted, a copy of probabilities is returned.

Return type:Distribution (usually Discrete for discrete and Continuous for continuous).
class Orange.statistics.estimate.ConditionalEstimatorFromDistribution

Bases: ConditionalEstimator

Probability estimator constructors that compute the whole contingency table (Orange.statistics.contingency.Table) of conditional probabilities in advance return this estimator with the table in the probabilities attribute.

probabilities

A precomputed contingency table.

__call__([[value], condition_value])

For detailed description of handling of different combinations of parameters, see the inherited ConditionalEstimator.__call__. For behaviour with continuous variable distributions, see the unconditional counterpart EstimatorFromDistribution.__call__.

class Orange.statistics.estimate.ConditionalByRows

Bases: ConditionalEstimatorConstructor

estimator_constructor

An unconditional probability estimator constructor.

Computes a conditional probability estimator using an unconditional probability estimator constructor. The result can be of type ConditionalEstimatorFromDistribution or ConditionalEstimatorByRows, depending on the type of constructor.

__call__([table, [prior, ]][instances, [weight_id, ]]estimator)
Parameters:

Compute contingency matrix if it has not been computed already. Then call estimator_constructor for each value of condition attribute. If all constructed estimators can return distribution of probabilities for all classes (usually either all or none can), the Distribution instances are put in a contingency table and ConditionalEstimatorFromDistribution is constructed and returned. If constructed estimators are not capable of returning distribution of probabilities, a ConditionalEstimatorByRows is constructed and the estimators are stored in its estimator_list.

Return type:ConditionalEstimatorFromDistribution or ConditionalEstimatorByRows
class Orange.statistics.estimate.ConditionalEstimatorByRows

Bases: ConditionalEstimator

A conditional probability estimator constructors that itself uses a series of estimators, one for each possible condition, stored in its estimator_list attribute.

estimator_list

A list of estimators; one for each value of condition.

__call__([[value], condition_value])

Uses estimators from estimator_list, depending on given condition_value. For detailed description of handling of different combinations of parameters, see the inherited ConditionalEstimator.__call__.

Example

>>> import Orange
>>> iris = Orange.data.Table("iris")
>>>
>>> # discrete class distribution
>>> iris_dist = Orange.statistics.distribution.Distribution("iris", iris)
>>> # m estimate constructor
>>> mest_constructor = Orange.statistics.estimate.M(m=10)
>>>
>>> # create the estimator
>>> mest = mest_constructor(iris_dist)
>>> print "%.2f" % mest(iris[0]['iris'])
0.33
>>> # petal length (continuous) distribution
>>> plength_dist = Orange.statistics.distribution.Distribution("petal length", iris)
>>> plength_dist.normalize()
>>>
>>> # loess contructor
>>> loess_est_constructor = Orange.statistics.estimate.Loess()
>>>
>>> # create the loess estimator
>>> loess_est = loess_est_constructor(plength_dist)
>>>
>>> print "%.2f" % loess_est(iris[0]['petal length'])
0.04
>>> # contingency matrix for the conditional estimator
>>> contingency = Orange.statistics.contingency.VarClass('petal length', iris)
>>> conditional_loess_constructor = Orange.statistics.estimate.ConditionalLoess()
>>>
>>> cloess_est = conditional_loess_constructor(contingency)
>>> print cloess_est(iris[0]['petal length'])
<0.980, 0.008, 0.012>