rxSummary: Object Summaries
Produce univariate summaries of objects in RevoScaleR.
rxSummary(formula, data, byGroupOutFile = NULL, summaryStats = c("Mean", "StdDev", "Min", "Max", "ValidObs", "MissingObs"), byTerm = TRUE, pweights = NULL, fweights = NULL, rowSelection = NULL, transforms = NULL, transformObjects = NULL, transformFunc = NULL, transformVars = NULL, transformPackages = NULL, transformEnvir = NULL, overwrite = FALSE, useSparseCube = rxGetOption("useSparseCube"), removeZeroCounts = useSparseCube, blocksPerRead = rxGetOption("blocksPerRead"), rowsPerBlock = 100000, reportProgress = rxGetOption("reportProgress"), verbose = 0, computeContext = rxGetOption("computeContext"), ...)
formula, as described in rxFormula. The formula typically does not contain a response variable, i.e. it should be of the form
~ terms. If
~. is used as the formula, summary statistics will be computed for all non-character variables. If a numeric variable is interacted with a factor variable, summary statistics will be computed for each category of the factor.
either a data source object, a character string specifying a .xdf file, or a data frame object to summarize.
NULL, a character string or vector of character strings specifying .xdf file names(s), or an RxXdfData object or list of RxXdfData objects. If not NULL, and the formula includes computations by factor, the by-group summary results will be written out to one or more .xdf files. If more than one
.xdf file is created and a single character string is specified, an integer will be appended to the base byGroupOutFile name for additional file names. The resulting RxXdfData objects will be listed in the
categorical component of the output object.
byGroupOutFile is not supported when using distributed compute contexts such as RxHadoopMR.
a character vector containing one or more of the following values: "Mean", "StdDev", "Min", "Max", "ValidObs", "MissingObs", "Sum".
logical variable. If
TRUE, missings will be removed by term (by variable or by interaction expression) before computing summary statistics. If
FALSE, observations with missings in any term will be removed before computations.
character string specifying the variable to use as probability weights for the observations.
character string specifying the variable to use as frequency weights for the observations.
name of a logical variable in the data set (in quotes) or a logical expression using variables in the data set to specify row selection. For example,
rowSelection = "old" will use only observations in which the value of the variable
rowSelection = (age > 20) & (age < 65) & (log(income) > 10) will use only observations in which the value of the
age variable is between 20 and 65 and the value of the
log of the
income variable is greater than 10. The row selection is performed after processing any data transformations (see the arguments
transformFunc). As with all expressions,
rowSelection can be defined outside of the function call using the expression function.
an expression of the form
list(name = expression, ...) representing the first round of variable transformations. As with all expressions,
rowSelection) can be defined outside of the function call using the expression function.
a named list containing objects that can be referenced by
variable transformation function. See rxTransform for details.
character vector of input data set variables needed for the transformation function. See rxTransform for details.
character vector defining additional R packages (outside of those specified in
rxGetOption("transformPackages")) to be made available and preloaded for use in variable transformation functions, e.g., those explicitly defined in RevoScaleR functions via their
transformFunc arguments or those defined implicitly via their
rowSelection arguments. The
transformPackages argument may also be
NULL, indicating that no packages outside
rxGetOption("transformPackages") will be preloaded.
user-defined environment to serve as a parent to all environments developed internally and used for variable data transformation. If
transformEnvir = NULL, a new "hash" environment with parent
baseenv() is used instead.
logical value. If
TRUE, an existing
byGroupOutFile will be overwritten.
overwrite is ignored
logical value. If
TRUE, sparse cube is used.
logical flag. If
TRUE, rows with no observations will be removed from the output for counts of categorical data. By default, it has the same value as
useSparseCube. For large summary computation, this should be set to
TRUE, otherwise R may run out of memory even if the internal C++ computation succeeds.
number of blocks to read for each chunk of data read from the data source.
maximum number of rows to write to each block in the
byGroupOutFile (if it is not
integer value with options:
0: no progress is reported.
1: the number of processed rows is printed and updated.
2: rows processed and timings are reported.
3: rows processed and all timings are reported.
integer value. If
0, no additional output is printed. If
1, additional summary information is printed.
a valid RxComputeContext. The
RxHadoopMR compute contexts distribute the computation among the nodes specified by the compute context; for other compute contexts, the computation is distributed if possible on the local computer.
additional arguments to be passed directly to the Revolution Compute Engine.
F() can be used in formula to force a variable to be
interpreted as factors.
If the formula contains a single dependent or response variable, summary statistics are
computed for the interaction between that variable and the first term
of the independent variables. (Multiple response variables are not permitted.)
For example, using the formula
y ~ xfac
will give the same results as using the formula
is a continuous variable and
xfac is a factor. Summary statistics
y are computed for each factor level of
x. This facilitates using the
same formula in
rxSummary as in, for example,
an rxSummary object containing the following elements:
number of valid observations.
number of missing observations.
data frame containing summaries for continuous variables.
list of summaries for categorical variables.
types of categorical summaries: can be "counts", or "cube" (for crosstab counts) or "none" (if there is no categorical summaries).
formula used to obtain the summary.
Microsoft Technical Support
# Create a local data frame DF <- data.frame(sex = c("Male", "Male", "Female", "Male"), age = c(20, 20, 12, 10), score = 1.1:4.1) # get summary of sex variable rxSummary(~ sex, DF) # obtain within sex-category statistics of the score variable rxSummary(score ~ sex, DF) # use transforms to create a factor variable and compute # summary statistics by each factor level rxSummary(~score:ageGroup, data=DF, transforms = list(ageGroup = cut(age, seq(0, 30, 10)))) # the following will give the same results rxSummary(score~ageGroup, data=DF, transforms = list(ageGroup = cut(age, seq(0, 30, 10)))) # the same formula can be used in rxCube rxCube(score~ageGroup, data=DF, transforms=list(ageGroup = cut(age, seq(0,30,10)))) # Write summary statistics by group to an .xdf file. Here the groups # are defined as year of age by sex by state (3 states in CensusWorkers file), # so summary statistics for 46 x 2 x 3 groups are computed. The first term # will just compute the Counts for each group, while the second two will # compute by-group Means and ValidObs for incwage and wkswork1 censusWorkers <- file.path(rxGetOption("sampleDataDir"), "CensusWorkers.xdf") sumOutFile <- tempfile(pattern = ".rxTempSumOut", fileext = ".xdf") sumOut <- rxSummary(~F(age):sex:state + incwage:F(age):sex:state + wkswork1:F(age):sex:state, data = censusWorkers, blocksPerRead = 3, byGroupOutFile = sumOutFile, rowsPerBlock = 10, summaryStats = c("Mean", "ValidObs")) rxGetVarInfo(sumOutFile) file.remove(sumOutFile)