Fast Two Stage P-value Computation Software

FastPval is a two stage p-value computation software which compute the empirical p-value by two stage ranking strategy, it can produce very low P-value based on huge dataset. This fast and powerful tool takes advantage of a delicate cutoff which separate the exactly significant area. Compared to the traditional ranking method, Tspvc has a good time efficiency, lower memory consuming and tiny storage spaces with high accuracy.

Traditionally, exact P-value computation based on permutation is needed for sorting the dataset before ranking. For huge dataset(more than 1 billion values), the time complexity of sorting time will be long(nlogn). Furthermore, surprisingly, the space complexity will be very large(1 billion dataset will take ~4G memory).

We adopt this two stage strategy in order to reduce the computation time and overcome memory restriction using the exact P-value computation method especially in the huge dataset. At first stage, we make a random sampling array (which means the first model) with given sampling number (up to the volume of dataset) and sort it. Then we select the satisfied array (which means the second model) from the dataset with the cutoff number which fetched form the first sampling array. So, directly, we can compute the approximate P-value with those two model.

The core procedure is implemented by C. We also offer a Java-based GUI by encapsulating the C package.

• Is flexible enough to produce model files and quick to compute P-value;
• Is efficient enough to store two approximate models for reduplicated P-value computation without loading all raw dataset;
• Allow visible GUI interface for the functions of model generating, two stage P-value computation and exact P-value computation using traditional method;
• Is compact enough for model size without loading the whole dataset into memory;

The dataset and test sample should follow the format:    dataset example    test sample

1.323204
2.323434
34.767567
0.565465
-2.432545
5.675477

There are three ways to launch the software:
   Please install the latest sun JRE in your local machine firstly.
1. Running from Java Web Start by clicking Launch the application 32bit / 64bit .

2. You can download 32bit / 64bit the package and run using following commands:

java -cp [the path of pvalue.jar] pvalue.MainGUI

java -cp <pvalue jar file path> pvalue.Pvalue -m 0 -d <dataset> -s <test sample>
-o <output path>

Required parameters:
   -m <Integer> The computation method:
      0: the two stage method
      1: the exact method(brute ranking)
      
Optional parameters:(default in [ ]): 
   -i <Integer> The instruction of current method [0]
      This parameter is effective only when '-m' is 0
      0: the general two stage method
      1: generate the model
      2: compute from given model

Required parameters:
   -d <String> The dataset file path
   -s <String> The test sample file path
   -o <String> The output directory
   
Optional parameters:(default in [ ]):
   -n <Integer> The sampling number for first model [100000]
      it is recommended (100000--<100M; 1000000--<10B;)
   -c <Integer> The P-value for the cutoff of first distribution [0.001]
   -a <Integer> The assumed distribution of dataset [0]
      0: normal distribution
      1: extreme distribution

Required parameters:
   -d <String> The dataset file path
   -o <String> The output directory
   
Optional parameters:(default in [ ]):
   -n <Integer> The sampling number for first model [100000]
      it is recommended (100000--<100M; 1000000--<10B;)
   -c <Integer> The P-value for the cutoff of first distribution [0.001]

Required parameters:
   -b1 <String> The first model file path
   -b2 <String> The second model file path
   -s <String> The test sample file path
   -o <String> The output directory
   
Optional parameters:(default in [ ]):
   -a <Integer> The assumed distribution of dataset [0]
      0: normal distribution
      1: extreme distribution

Required parameters:
   -d <String> The dataset file path
   -s <String> The test sample file path
   -o <String> The output directory

Table: The comparison between FastPval and Exact P-value Computation(brutal sorting). Dateset Size: The number of dataset(M: million; B: billion) and the file size.  FastPval(Time): The model generating time and model loading time and P-value computation time using current dataset as test sample.  Exact Computation(Time): The time of sorting dataset and background loading time and exact P-value computation time by binary search using current dataset as test sample.  FastPval(RAM): The total memory consumption of FastPval.  Exact Computation(RAM) : The total memory consumption using exact P-value computation.  FastPval Model Size : The two model files size of FastPval.