Ordinal Regression

In statistics, ordinal regression is a type of regression analysis used for predicting an ordinal variable, i.e. a variable whose value exists on an arbitrary scale where only the relative ordering between different values is significant. The two most common types of ordinal regression models are ordered logit, which applies to data that meet the proportional odds assumption, and ordered probit.

- Training Function
- The ordinal regression training function has the following syntax:
ordinal(source_table, model_table, dependent_varname, independent_varname, cat_order, link_func, grouping_col, optim_params, verbose )

**Arguments**

- source_table
VARCHAR. Name of the table containing the training data.

- model_table
VARCHAR. Name of the generated table containing the model.

The model table produced by ordinal() contains the following columns:

<...> Grouping columns, if provided in input. This could be multiple columns depending on the

`grouping_col`

input.coef_threshold FLOAT8[]. Vector of the threshold coefficients in linear predictor. The threshold coefficients are the intercepts specific to each categorical levels

std_err_threshold FLOAT8[]. Vector of the threshold standard errors of the threshold coefficients.

z_stats_threshold FLOAT8[]. Vector of the threshold z-statistics of the thresholdcoefficients.

p_values_threshold FLOAT8[]. Vector of the threshold p-values of the threshold coefficients.

log_likelihood FLOAT8. The log-likelihood \( l(\boldsymbol \beta) \). The value will be the same across categories within the same group.

coef_feature FLOAT8[]. Vector of the feature coefficients in linear predictor. The feature coefficients are the coefficients for the independent variables. They are the same across categories.

std_err_feature FLOAT8[]. Vector of the feature standard errors of the feature coefficients.

z_stats_feature FLOAT8[]. Vector of the feature z-statistics of the feature coefficients.

p_values_feature FLOAT8[]. Vector of the feature p-values of the feature coefficients.

num_rows_processed BIGINT. Number of rows processed.

num_rows_skipped BIGINT. Number of rows skipped due to missing values or failures.

num_iterations INTEGER. Number of iterations actually completed. This would be different from the `nIterations`

argument if a`tolerance`

parameter is provided and the algorithm converges before all iterations are completed.A summary table named <model_table>_summary is also created at the same time, which has the following columns:

method VARCHAR. String describes the model: 'ordinal'.

source_table VARCHAR. Data source table name.

model_table VARCHAR. Model table name.

dependent_varname VARCHAR. Expression for dependent variable.

independent_varname VARCHAR. Expression for independent variables. The independent variables should not include intercept term. Otherwise there will be an error message indicating Hessian matrix is not finite. In that case, the user should drop the intercept and rerun the function agian.

cat_order VARCHAR. String representation of category order. Default is the sorted categories in data using python sort

link_func VARCHAR. String that contains link function parameters: 'logit' and 'probit' links are implemented now

grouping_col VARCHAR. String representation of grouping columns.

optimizer_params VARCHAR. String that contains optimizer parameters, and has the form of 'optimizer=..., max_iter=..., tolerance=...'.

num_all_groups INTEGER. Number of groups in ordinal regression training.

num_failed_groups INTEGER. Number of failed groups in ordinal regression training.

total_rows_processed BIGINT. Total number of rows processed in all groups.

total_rows_skipped BIGINT. Total number of rows skipped in all groups due to missing values or failures.

- dependent_varname
VARCHAR. Name of the dependent variable column.

- independent_varname
VARCHAR. Expression list to evaluate for the independent variables. The intercept should not be included here since the cumulative probability force to have intercepts for each category level.

- cat_order
VARCHAR, String that represents the order of category. The order is specified by charactor '<'.

- link_function (optional)
VARCHAR, default: 'logit'. Parameters for link function. Currently, we support logit and probit.

- grouping_col (optional)
VARCHAR, default: NULL. An expression list used to group the input dataset into discrete groups, running one regression per group. Similar to the SQL "GROUP BY" clause. When this value is NULL, no grouping is used and a single model is generated.

- optim_params (optional)
VARCHAR, default: 'max_iter=100,optimizer=irls,tolerance=1e-6'. Parameters for optimizer. Currently, we support tolerance=[tolerance for relative error between log-likelihoods], max_iter=[maximum iterations to run], optimizer=irls.

- verbose (optional)
- BOOLEAN, default: FALSE. Provides verbose output of the results of training.

- Note
- To calculate the standard error the coefficient, we are using the square root of the diagnal elements of the expected Fisher information matrix, which is a by-product of iteratively reweighted least square. This method is used in the original ordinal regression paper by McCullagh(1980). In some software like Stata, the standard error is calculated by the observed information matrix, which is supported by Efron and Hinkley (1978). In R, polr() uses the approximated observed information matrix while the optimization is achieved by first order optimization method. Therefore, there will be some difference on standard error, z-stats and p-value from other software.

- Prediction Function
- Ordinal regression prediction function has the following format:
ordinal_predict( model_table, predict_table_input, output_table, predict_type, verbose )

**Arguments**- model_table
TEXT. Name of the generated table containing the model, which is the output table from ordinal().

- predict_table_input
TEXT. The name of the table containing the data to predict on. The table must contain id column as the primary key.

- output_table
TEXT. Name of the generated table containing the predicted values.

The model table produced by ordinal_predict contains the following columns:

id SERIAL. Column to identify the predicted value.

category TEXT. Available if the predicted type = 'response'. Column contains the predicted categories

category_value FLOAT8. The predicted probability for the specific category_value. - predict_type
TEXT. Either 'response' or 'probability'. Using 'response' will give the predicted category with the largest probability. Using probability will give the predicted probabilities for all categories

- verbose
- BOOLEAN. Whether verbose is displayed

- Examples

- Create the training data table.
DROP TABLE IF EXISTS test3; CREATE TABLE test3 ( feat1 INTEGER, feat2 INTEGER, cat INTEGER ); INSERT INTO test3(feat1, feat2, cat) VALUES (1,35,1), (2,33,0), (3,39,1), (1,37,1), (2,31,1), (3,36,0), (2,36,1), (2,31,1), (2,41,1), (2,37,1), (1,44,1), (3,33,2), (1,31,1), (2,44,1), (1,35,1), (1,44,0), (1,46,0), (2,46,1), (2,46,2), (3,49,1), (2,39,0), (2,44,1), (1,47,1), (1,44,1), (1,37,2), (3,38,2), (1,49,0), (2,44,0), (3,61,2), (1,65,2), (3,67,1), (3,65,2), (1,65,2), (2,67,2), (1,65,2), (1,62,2), (3,52,2), (3,63,2), (2,59,2), (3,65,2), (2,59,0), (3,67,2), (3,67,2), (3,60,2), (3,67,2), (3,62,2), (2,54,2), (3,65,2), (3,62,2), (2,59,2), (3,60,2), (3,63,2), (3,65,2), (2,63,1), (2,67,2), (2,65,2), (2,62,2);

- Run the multilogistic regression function.
DROP TABLE IF EXISTS test3_output; DROP TABLE IF EXISTS test3_output_summary; SELECT madlib.ordinal('test3', 'test3_output', 'cat', 'ARRAY[feat1, feat2]', '0<1<2', 'logit' );

- View the regression results.
-- Set extended display on for easier reading of output \x on SELECT * FROM test3_output;

Result:

-[ RECORD 1 ]------+------------------------------------------- coef_threshold | {4.12831944358935,6.55999442887089} std_err_threshold | {1.3603408170882,1.54843501580999} z_stats_threshold | {3.03476848722806,4.23653195768075} p_values_threshold | {0.00240720390579325,2.26998625331282e-05} log_likelihood | -42.1390192418541 coef_feature | {0.574822563129293,0.108115645059558} std_err_feature | {0.394064908788145,0.0276025960683975} z_stats_feature | {1.45870020473791,3.91686509456046} p_values_feature | {0.144647639733733,8.9707915817562e-05} num_rows_processed | 57 num_rows_skipped | 0 iteration | 7

- Predicting dependent variable using ordinal model. (This example uses the original data table to perform the prediction. Typically a different test dataset with the same features as the original training dataset would be used for prediction.)

\x off -- Add the id column for prediction function ALTER TABLE test3 ADD COLUMN id SERIAL; -- Predict probabilities for all categories using the original data SELECT ordinal_predict('test3_output','test3', 'test3_prd_prob', 'probability'); -- Display the predicted value SELECT * FROM test3_prd_prob;

- Model Details

The function ordinal() fit the ordinal response model using a cumulative link model. The ordinal reponse variable, denoted by \( Y_i \), can fall in \( j = 1,.. , J\) categories. Then \( Y_i \) follows a multinomial distribution with parameter \(\pi\) where \(\pi_{ij}\) denote the probability that the \(i\)th observation falls in response category \(j\). We define the cumulative probabilities as

\[ \gamma_{ij} = \Pr(Y_i \le j)= \pi_{i1} +...+ \pi_{ij} . \]

Next we will consider the logit link for illustration purpose. The logit function is defined as \( \mbox{logit}(\pi) = \log[\pi/(1-\pi)] \) and cumulative logits are defined as:

\[ \mbox{logit}(\gamma_{ij})=\mbox{logit}(\Pr(Y_i \le j))=\log \frac{\Pr(Y_i \le j)}{1-\Pr(Y_i\le j)}, j=1,...,Jā1 \]

so that the cumulative logits are defined for all but the last category.

A cumulative link model with a logit link, or simply cumulative logit model is a regression model for cumulative logits:

\[ \mbox{logit}(\gamma_{ij}) = \theta_j - x^T_i \beta \]

where \(x_i\) is a vector of explanatory variables for the \(i\)th observation and \(\beta\) is the corresponding set of regression parameters. The \(\{\theta_j\}\) parameters provide each cumulative logit (for each \(j\)) with its own intercept. A key point is that the regression part \(x^T_i\beta\) is independent of \(j\), so \(\beta\) has the same effect for each of the J ā 1 cumulative logits. Note that \(x^T_i\beta\) does not contain an intercept, since the \(\{\theta_j\}\) act as intercepts. For small values of \(x^T_i\beta\) the response is likely to fall in the first category and for large values of \(x^T_i\beta\) the response is likely to fall in the last category. The horizontal displacements of the curves are given by the values of \(\{\theta_j\}\).

- Literature

A collection of nice write-ups, with valuable pointers into further literature:

[1] Peter McCullagh: Regression Models for Ordinal Data, Journal of the Royal Statistical Society. Series B (Methodological), Volume 42, Issue 2 (1980), 109-142

[2] Rune Haubo B Christensen: Analysis of ordinal data with cumulative link models – estimation with the R-package ordinal. cran.r-project.org/web/packages/ordinal/vignettes/clm_intro.pdf

- Related Topics

File ordinal.sql_in documenting the ordinal regression functions