Top Description Constructors Methods
org.apache.commons.math3.stat.inference

public Class GTest

extends Object
Class Inheritance
Imports
org.apache.commons.math3.distribution.ChiSquaredDistribution, org.apache.commons.math3.exception.DimensionMismatchException, .MaxCountExceededException, .NotPositiveException, .NotStrictlyPositiveException, .OutOfRangeException, .ZeroException, org.apache.commons.math3.exception.util.LocalizedFormats, org.apache.commons.math3.util.FastMath, .MathArrays

Implements G Test statistics.

This is known in statistical genetics as the McDonald-Kreitman test. The implementation handles both known and unknown distributions.

Two samples tests can be used when the distribution is unknown a priori but provided by one sample, or when the hypothesis under test is that the two samples come from the same underlying distribution.

Since
3.1

Constructor Summary

AccessConstructor and Description
public

Method Summary

Modifier and TypeMethod and Description
private double

Returns:

Shannon Entropy of the given Matrix
entropy
(final long[][]
2 Dimensional Matrix of long values (for ex. the counts of a trials)
k
)

Calculates the Shannon entropy for 2 Dimensional Matrix.

private double

Returns:

Shannon Entropy of the given Vector
entropy
(final long[]
Vector (for ex. Row Sums of a trials)
k
)

Calculates the Shannon entropy for a vector.

public double

Returns:

G-Test statistic
g
(final double[]
array of expected frequency counts
expected
,
final long[]
array of observed frequency counts
observed
)

Computes the G statistic for Goodness of Fit comparing observed and expected frequency counts.

public double

Returns:

G-Test statistic
gDataSetsComparison
(final long[]
array of observed frequency counts of the first data set
observed1
,
final long[]
array of observed frequency counts of the second data set
observed2
)

Computes a G (Log-Likelihood Ratio) two sample test statistic for independence comparing frequency counts in observed1 and observed2.

public double

Returns:

p-value
gTest
(final double[]
array of expected frequency counts
expected
,
final long[]
array of observed frequency counts
observed
)

Returns the observed significance level, or p-value, associated with a G-Test for goodness of fit comparing the observed frequency counts to those in the expected array.

public boolean

Returns:

true iff null hypothesis can be rejected with confidence 1 - alpha
gTest
(final double[]
array of expected frequency counts
expected
,
final long[]
array of observed frequency counts
observed
,
final double
significance level of the test
alpha
)

Performs a G-Test (Log-Likelihood Ratio Test) for goodness of fit evaluating the null hypothesis that the observed counts conform to the frequency distribution described by the expected counts, with significance level alpha.

public double

Returns:

p-value
gTestDataSetsComparison
(final long[]
array of observed frequency counts of the first data set
observed1
,
final long[]
array of observed frequency counts of the second data set
observed2
)

Returns the observed significance level, or p-value, associated with a G-Value (Log-Likelihood Ratio) for two sample test comparing bin frequency counts in observed1 and observed2.

public boolean

Returns:

true iff null hypothesis can be rejected with confidence 1 - alpha
gTestDataSetsComparison
(final long[]
array of observed frequency counts of the first data set
observed1
,
final long[]
array of observed frequency counts of the second data set
observed2
,
final double
significance level of the test
alpha
)

Performs a G-Test (Log-Likelihood Ratio Test) comparing two binned data sets.

public double

Returns:

p-value
gTestIntrinsic
(final double[]
array of expected frequency counts
expected
,
final long[]
array of observed frequency counts
observed
)

Returns the intrinsic (Hardy-Weinberg proportions) p-Value, as described in p64-69 of McDonald, J.H.

public double

Returns:

root log-likelihood ratio
rootLogLikelihoodRatio
(final long
number of times the two events occurred together (AB)
k11
,
long
number of times the second event occurred WITHOUT the first event (notA,B)
k12
,
final long
number of times the first event occurred WITHOUT the second event (A, notB)
k21
,
final long
number of times something else occurred (i.e. was neither of these events (notA, notB)
k22
)

Calculates the root log-likelihood ratio for 2 state Datasets.

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Constructor Detail

GTestback to summary
public GTest()

Method Detail

entropyback to summary
private double entropy(final long[][] k)

Calculates the Shannon entropy for 2 Dimensional Matrix. The value returned is the entropy of the vector formed by concatenating the rows (or columns) of k to form a vector. See entropy(long[]).

Parameters
k:long[][]

2 Dimensional Matrix of long values (for ex. the counts of a trials)

Returns:double

Shannon Entropy of the given Matrix

entropyback to summary
private double entropy(final long[] k)

Calculates the Shannon entropy for a vector. The values of k are taken to be incidence counts of the values of a random variable. What is returned is
∑pilog(pi
where pi = k[i] / (sum of elements in k)

Parameters
k:long[]

Vector (for ex. Row Sums of a trials)

Returns:double

Shannon Entropy of the given Vector

gback to summary
public double g(final double[] expected, final long[] observed) throws NotPositiveException, NotStrictlyPositiveException, DimensionMismatchException

Computes the G statistic for Goodness of Fit comparing observed and expected frequency counts.

This statistic can be used to perform a G test (Log-Likelihood Ratio Test) evaluating the null hypothesis that the observed counts follow the expected distribution.

Preconditions:

  • Expected counts must all be positive.
  • Observed counts must all be ≥ 0.
  • The observed and expected arrays must have the same length and their common length must be at least 2.

If any of the preconditions are not met, a MathIllegalArgumentException is thrown.

Note

This implementation rescales the expected array if necessary to ensure that the sum of the expected and observed counts are equal.

Parameters
expected:double[]

array of expected frequency counts

observed:long[]

array of observed frequency counts

Returns:double

G-Test statistic

Exceptions
NotPositiveException:
if observed has negative entries
NotStrictlyPositiveException:
if expected has entries that are not strictly positive
DimensionMismatchException:
if the array lengths do not match or are less than 2.
gDataSetsComparisonback to summary
public double gDataSetsComparison(final long[] observed1, final long[] observed2) throws DimensionMismatchException, NotPositiveException, ZeroException

Computes a G (Log-Likelihood Ratio) two sample test statistic for independence comparing frequency counts in observed1 and observed2. The sums of frequency counts in the two samples are not required to be the same. The formula used to compute the test statistic is

2 * totalSum * [H(rowSums) + H(colSums) - H(k)]

where H is the Shannon Entropy of the random variable formed by viewing the elements of the argument array as incidence counts;
k is a matrix with rows [observed1, observed2];
rowSums, colSums are the row/col sums of k;
and totalSum is the overall sum of all entries in k.

This statistic can be used to perform a G test evaluating the null hypothesis that both observed counts are independent

Preconditions:

  • Observed counts must be non-negative.
  • Observed counts for a specific bin must not both be zero.
  • Observed counts for a specific sample must not all be 0.
  • The arrays observed1 and observed2 must have the same length and their common length must be at least 2.

If any of the preconditions are not met, a MathIllegalArgumentException is thrown.

Parameters
observed1:long[]

array of observed frequency counts of the first data set

observed2:long[]

array of observed frequency counts of the second data set

Returns:double

G-Test statistic

Exceptions
DimensionMismatchException:
the the lengths of the arrays do not match or their common length is less than 2
NotPositiveException:
if any entry in observed1 or observed2 is negative
ZeroException:
if either all counts of observed1 or observed2 are zero, or if the count at the same index is zero for both arrays.
gTestback to summary
public double gTest(final double[] expected, final long[] observed) throws NotPositiveException, NotStrictlyPositiveException, DimensionMismatchException, MaxCountExceededException

Returns the observed significance level, or p-value, associated with a G-Test for goodness of fit comparing the observed frequency counts to those in the expected array.

The number returned is the smallest significance level at which one can reject the null hypothesis that the observed counts conform to the frequency distribution described by the expected counts.

The probability returned is the tail probability beyond g(expected, observed) in the ChiSquare distribution with degrees of freedom one less than the common length of expected and observed.

Preconditions:

  • Expected counts must all be positive.
  • Observed counts must all be ≥ 0.
  • The observed and expected arrays must have the same length and their common length must be at least 2.

If any of the preconditions are not met, a MathIllegalArgumentException is thrown.

Note

This implementation rescales the expected array if necessary to ensure that the sum of the expected and observed counts are equal.

Parameters
expected:double[]

array of expected frequency counts

observed:long[]

array of observed frequency counts

Returns:double

p-value

Exceptions
NotPositiveException:
if observed has negative entries
NotStrictlyPositiveException:
if expected has entries that are not strictly positive
DimensionMismatchException:
if the array lengths do not match or are less than 2.
MaxCountExceededException:
if an error occurs computing the p-value.
gTestback to summary
public boolean gTest(final double[] expected, final long[] observed, final double alpha) throws NotPositiveException, NotStrictlyPositiveException, DimensionMismatchException, OutOfRangeException, MaxCountExceededException

Performs a G-Test (Log-Likelihood Ratio Test) for goodness of fit evaluating the null hypothesis that the observed counts conform to the frequency distribution described by the expected counts, with significance level alpha. Returns true iff the null hypothesis can be rejected with 100 * (1 - alpha) percent confidence.

Example:
To test the hypothesis that observed follows expected at the 99% level, use

gTest(expected, observed, 0.01)

Returns true iff gTestGoodnessOfFitPValue(expected, observed) < alpha

Preconditions:

  • Expected counts must all be positive.
  • Observed counts must all be ≥ 0.
  • The observed and expected arrays must have the same length and their common length must be at least 2.
  • 0 < alpha < 0.5

If any of the preconditions are not met, a MathIllegalArgumentException is thrown.

Note

This implementation rescales the expected array if necessary to ensure that the sum of the expected and observed counts are equal.

Parameters
expected:double[]

array of expected frequency counts

observed:long[]

array of observed frequency counts

alpha:double

significance level of the test

Returns:boolean

true iff null hypothesis can be rejected with confidence 1 - alpha

Exceptions
NotPositiveException:
if observed has negative entries
NotStrictlyPositiveException:
if expected has entries that are not strictly positive
DimensionMismatchException:
if the array lengths do not match or are less than 2.
OutOfRangeException:
if alpha is not strictly greater than zero and less than or equal to 0.5
MaxCountExceededException:
if an error occurs computing the p-value.
gTestDataSetsComparisonback to summary
public double gTestDataSetsComparison(final long[] observed1, final long[] observed2) throws DimensionMismatchException, NotPositiveException, ZeroException, MaxCountExceededException

Returns the observed significance level, or p-value, associated with a G-Value (Log-Likelihood Ratio) for two sample test comparing bin frequency counts in observed1 and observed2.

The number returned is the smallest significance level at which one can reject the null hypothesis that the observed counts conform to the same distribution.

See gTest(double[], long[]) for details on how the p-value is computed. The degrees of of freedom used to perform the test is one less than the common length of the input observed count arrays.

Preconditions:

  • Observed counts must be non-negative.
  • Observed counts for a specific bin must not both be zero.
  • Observed counts for a specific sample must not all be 0.
  • The arrays observed1 and observed2 must have the same length and their common length must be at least 2.

If any of the preconditions are not met, a MathIllegalArgumentException is thrown.

Parameters
observed1:long[]

array of observed frequency counts of the first data set

observed2:long[]

array of observed frequency counts of the second data set

Returns:double

p-value

Exceptions
DimensionMismatchException:
the the length of the arrays does not match or their common length is less than 2
NotPositiveException:
if any of the entries in observed1 or observed2 are negative
ZeroException:
if either all counts of observed1 or observed2 are zero, or if the count at some index is zero for both arrays
MaxCountExceededException:
if an error occurs computing the p-value.
gTestDataSetsComparisonback to summary
public boolean gTestDataSetsComparison(final long[] observed1, final long[] observed2, final double alpha) throws DimensionMismatchException, NotPositiveException, ZeroException, OutOfRangeException, MaxCountExceededException

Performs a G-Test (Log-Likelihood Ratio Test) comparing two binned data sets. The test evaluates the null hypothesis that the two lists of observed counts conform to the same frequency distribution, with significance level alpha. Returns true iff the null hypothesis can be rejected with 100 * (1 - alpha) percent confidence.

See gDataSetsComparison(long[], long[]) for details on the formula used to compute the G (LLR) statistic used in the test and gTest(double[], long[]) for information on how the observed significance level is computed. The degrees of of freedom used to perform the test is one less than the common length of the input observed count arrays.

Preconditions:
  • Observed counts must be non-negative.
  • Observed counts for a specific bin must not both be zero.
  • Observed counts for a specific sample must not all be 0.
  • The arrays observed1 and observed2 must have the same length and their common length must be at least 2.
  • 0 < alpha < 0.5

If any of the preconditions are not met, a MathIllegalArgumentException is thrown.

Parameters
observed1:long[]

array of observed frequency counts of the first data set

observed2:long[]

array of observed frequency counts of the second data set

alpha:double

significance level of the test

Returns:boolean

true iff null hypothesis can be rejected with confidence 1 - alpha

Exceptions
DimensionMismatchException:
the the length of the arrays does not match
NotPositiveException:
if any of the entries in observed1 or observed2 are negative
ZeroException:
if either all counts of observed1 or observed2 are zero, or if the count at some index is zero for both arrays
OutOfRangeException:
if alpha is not in the range (0, 0.5]
MaxCountExceededException:
if an error occurs performing the test
gTestIntrinsicback to summary
public double gTestIntrinsic(final double[] expected, final long[] observed) throws NotPositiveException, NotStrictlyPositiveException, DimensionMismatchException, MaxCountExceededException

Returns the intrinsic (Hardy-Weinberg proportions) p-Value, as described in p64-69 of McDonald, J.H. 2009. Handbook of Biological Statistics (2nd ed.). Sparky House Publishing, Baltimore, Maryland.

The probability returned is the tail probability beyond g(expected, observed) in the ChiSquare distribution with degrees of freedom two less than the common length of expected and observed.

Parameters
expected:double[]

array of expected frequency counts

observed:long[]

array of observed frequency counts

Returns:double

p-value

Exceptions
NotPositiveException:
if observed has negative entries
NotStrictlyPositiveException:
expected has entries that are not strictly positive
DimensionMismatchException:
if the array lengths do not match or are less than 2.
MaxCountExceededException:
if an error occurs computing the p-value.
rootLogLikelihoodRatioback to summary
public double rootLogLikelihoodRatio(final long k11, long k12, final long k21, final long k22)

Calculates the root log-likelihood ratio for 2 state Datasets. See gDataSetsComparison(long[], long[]).

Given two events A and B, let k11 be the number of times both events occur, k12 the incidence of B without A, k21 the count of A without B, and k22 the number of times neither A nor B occurs. What is returned by this method is

(sgn) sqrt(gValueDataSetsComparison({k11, k12}, {k21, k22})

where sgn is -1 if k11 / (k11 + k12) < k21 / (k21 + k22));
1 otherwise.

Signed root LLR has two advantages over the basic LLR: a) it is positive where k11 is bigger than expected, negative where it is lower b) if there is no difference it is asymptotically normally distributed. This allows one to talk about "number of standard deviations" which is a more common frame of reference than the chi^2 distribution.

Parameters
k11:long

number of times the two events occurred together (AB)

k12:long

number of times the second event occurred WITHOUT the first event (notA,B)

k21:long

number of times the first event occurred WITHOUT the second event (A, notB)

k22:long

number of times something else occurred (i.e. was neither of these events (notA, notB)

Returns:double

root log-likelihood ratio