Multinomial Logistic Regression

- Warning
*This is an old implementation of multinomial logistic regression. Replacement of this function is available as the Multinomial regression module Multinomial Regression*

Multinomial logistic regression is a widely used regression analysis tool that models the outcomes of categorical dependent random variables. The model assumes that the conditional mean of the dependent categorical variables is the logistic function of an affine combination of independent variables. Multinomial logistic regression finds the vector of coefficients that maximizes the likelihood of the observations.

- Training Function
- The multinomial logistic regression training function has the following syntax:
mlogregr_train(source_table, output_table, dependent_varname, independent_varname, ref_category, optimizer_params )

**Arguments**- source_table
TEXT. The name of the table containing the input data.

- output_table
TEXT. The name of the generated table containing the output model. The output table produced by the multinomial logistic regression training function contains the following columns:

category INTEGER. The category. Categories are encoded as integers with values from {0, 1, 2,..., *numCategories*– 1}ref_category INTEGER. The reference category. Categories are encoded as integers with values from {0, 1, 2,..., *numCategories*– 1}coef FLOAT8[]. An array of coefficients, \( \boldsymbol c \). log_likelihood FLOAT8. The log-likelihood, \( l(\boldsymbol c) \). std_err FLOAT8[]. An array of the standard errors. z_stats FLOAT8[]. An array of the Wald z-statistics. p_values FLOAT8[]. An array of the Wald p-values. odds_ratios FLOAT8[]. An array of the odds ratios. condition_no FLOAT8. The condition number of the matrix, computed using the coefficients of the iteration immediately preceding convergence. num_iterations INTEGER. The number of iterations executed before the algorithm completed. A summary table named <out_table>_summary is also created at the same time, and it contains the following columns:

source_table The data source table name. out_table The output table name. dependent_varname The dependent variable. independent_varname The independent variables. optimizer_params The optimizer parameters. It is a copy of the optimizer_params in the training function's arguments. ref_category An integer, the value of reference category used. num_rows_processed INTEGER. The number of rows actually processed, which is equal to the total number of rows in the source table minus the number of skipped rows. num_missing_rows_skipped INTEGER. The number of rows skipped during the training. A row will be skipped if the ind_col is NULL or contains NULL values. - dependent_varname
TEXT. The name of the column containing the dependent variable.

- independent_varname
TEXT. Expression list to evaluate for the independent variables. An intercept variable is not assumed. The number of independent variables cannot exceed 65535.

- ref_category (optional)
INTEGER, default: 0. The reference category ranges from [0,

*numCategories*– 1].- optimizer_params (optional)
- VARCHAR, default: NULL, which uses the default values of optimizer parameters. It should be a string that contains pairs of 'key=value' separated by commas. Supported parameters with their default values: max_iter=20, optimizer='irls', precision=1e-4. Currently, only 'irls' and 'newton' are allowed for 'optimizer'.

- Note
- Table names can be optionally schema qualified and table and column names should follow the same case-sensitivity and quoting rules as in the database.

- Prediction Function
- The prediction function is provided to estimate the conditional mean given a new predictor. It has the following syntax:
mlogregr_predict( model_table, new_data_table, id_col_name, output_table, type)

**Arguments**

- model_table
TEXT. Name of the table containing the multilogistic model. This should be the output table returned from

*mlogregr_train*.- new_data_table
TEXT. Name of the table containing prediction data. This table is expected to contain the same features that were used during training. The table should also contain

*id_col_name*used for identifying each row.- id_col_name
TEXT. Name of the column containing id information in the source data. This is a mandatory argument and is used for correlating prediction table rows with the source. The values of this column are expected to be unique for each tuple.

- output_table
TEXT. Name of the table to output prediction results to. If this table already exists then an error is returned. This output table contains the

*id_col_name*column giving the 'id' for each prediction.If

*type*= 'response', then the table has a single additional column with the prediction value of the response. The type of this column depends on the type of the response variable used during training.If

*type*= 'prob', then the table has multiple additional columns, one for each possible category. The columns are labeled as 'estimated_prob_*category_value*', where*category_value*represents the values of categories (0 to K-1).- type
TEXT, optional, default: 'response'.

When

*type*= 'prob', the probabilities of each category (including the reference category) is given.When

*type*= 'response', a single output is provided which represents the prediction category for each tuple. This represents the category with the highest probability.

- Examples

- Create the training data table.
DROP TABLE IF EXISTS test3; CREATE TABLE test3 ( feat1 INTEGER, feat2 INTEGER, cat INTEGER ); INSERT INTO test3(feat1, feat2, cat) VALUES (1,35,1), (2,33,0), (3,39,1), (1,37,1), (2,31,1), (3,36,0), (2,36,1), (2,31,1), (2,41,1), (2,37,1), (1,44,1), (3,33,2), (1,31,1), (2,44,1), (1,35,1), (1,44,0), (1,46,0), (2,46,1), (2,46,2), (3,49,1), (2,39,0), (2,44,1), (1,47,1), (1,44,1), (1,37,2), (3,38,2), (1,49,0), (2,44,0), (3,61,2), (1,65,2), (3,67,1), (3,65,2), (1,65,2), (2,67,2), (1,65,2), (1,62,2), (3,52,2), (3,63,2), (2,59,2), (3,65,2), (2,59,0), (3,67,2), (3,67,2), (3,60,2), (3,67,2), (3,62,2), (2,54,2), (3,65,2), (3,62,2), (2,59,2), (3,60,2), (3,63,2), (3,65,2), (2,63,1), (2,67,2), (2,65,2), (2,62,2);

- Run the multilogistic regression function.
DROP TABLE IF EXISTS test3_output; DROP TABLE IF EXISTS test3_output_summary; SELECT madlib.mlogregr_train('test3', 'test3_output', 'cat', 'ARRAY[1, feat1, feat2]', 0, 'max_iter=20, optimizer=irls, precision=0.0001' );

- View the result:
-- Set extended display on for easier reading of output \x on SELECT * FROM test3_output;

Results:-[ RECORD 1 ]--+------------------------------------------------------------ category | 1 ref_category | 0 coef | {1.45474045211601,0.0849956182104023,-0.0172383499601956} loglikelihood | -39.14759930999 std_err | {2.13085072854143,0.585021661344715,0.0431487356292144} z_stats | {0.682704063982831,0.145286275409074,-0.39950996729842} p_values | {0.494793861210936,0.884484850387893,0.689517480964129} odd_ratios | {4.28337158128448,1.08871229617973,0.982909380301134} condition_no | 280069.034217586 num_iterations | 5 -[ RECORD 2 ]--+------------------------------------------------------------ category | 2 ref_category | 0 coef | {-7.12908167688326,0.87648787696783,0.127886153027713} loglikelihood | -39.14759930999 std_err | {2.52104008297868,0.639575886323862,0.0445757462972303} z_stats | {-2.82783352990566,1.37042045472615,2.86896269049475} p_values | {0.00468641692252239,0.170555690550421,0.00411820373218956} odd_ratios | {0.000801455044349486,2.40244718187161,1.13642361694409} condition_no | 280069.034217586 num_iterations | 5

- View all parameters used during the training
\x on SELECT * FROM test3_output_summary;

Results:-[ RECORD 1 ]------------+-------------------------------------------------- method | mlogregr source_table | test3 out_table | test3_output dependent_varname | cat independent_varname | ARRAY[1, feat1, feat2] optimizer_params | max_iter=20, optimizer=irls, precision=0.0001 ref_category | 0 num_categories | 3 num_rows_processed | 57 num_missing_rows_skipped | 0 variance_covariance | {{4.54052482732554,3.01080140927409,-0.551901021610841,-0.380754019900586,-0.0784151362989211,-0.0510014701718268},{3.01080140927409,6.35564309998514,-0.351902272617974,-0.766730342510818,-0.051877550252329,-0.0954432017695571},{-0.551901021610841,-0.351902272617974,0.34225034424253,0.231740815080827,-0.00117521831508331,-0.00114043921343171},{-0.380754019900586,-0.766730342510818,0.231740815080827,0.409057314366954,-0.000556498286025567,-0.000404735750986327},{-0.0784151362989211,-0.051877550252329,-0.00117521831508331,-0.000556498286025569,0.00186181338639984,0.00121080293928445},{-0.0510014701718268,-0.0954432017695571,-0.00114043921343171,-0.000404735750986325,0.00121080293928446,0.00198699715795504}} coef | {{1.45474045211601,0.0849956182104023,-0.0172383499601956},{-7.12908167688326,0.87648787696783,0.127886153027713}}

- Technical Background
- Multinomial logistic regression models the outcomes of categorical dependent random variables (denoted \( Y \in \{ 0,1,2 \ldots k \} \)). The model assumes that the conditional mean of the dependent categorical variables is the logistic function of an affine combination of independent variables (usually denoted \( \boldsymbol x \)). That is,
\[ E[Y \mid \boldsymbol x] = \sigma(\boldsymbol c^T \boldsymbol x) \]

for some unknown vector of coefficients \( \boldsymbol c \) and where \( \sigma(x) = \frac{1}{1 + \exp(-x)} \) is the logistic function. Multinomial logistic regression finds the vector of coefficients \( \boldsymbol c \) that maximizes the likelihood of the observations.

Let

- \( \boldsymbol y \in \{ 0,1 \}^{n \times k} \) denote the vector of observed dependent variables, with \( n \) rows and \( k \) columns, containing the observed values of the dependent variable,
- \( X \in \mathbf R^{n \times k} \) denote the design matrix with \( k \) columns and \( n \) rows, containing all observed vectors of independent variables \( \boldsymbol x_i \) as rows.

By definition,

\[ P[Y = y_i | \boldsymbol x_i] = \sigma((-1)^{y_i} \cdot \boldsymbol c^T \boldsymbol x_i) \,. \]

Maximizing the likelihood \( \prod_{i=1}^n \Pr(Y = y_i \mid \boldsymbol x_i) \) is equivalent to maximizing the log-likelihood \( \sum_{i=1}^n \log \Pr(Y = y_i \mid \boldsymbol x_i) \), which simplifies to

\[ l(\boldsymbol c) = -\sum_{i=1}^n \log(1 + \exp((-1)^{y_i} \cdot \boldsymbol c^T \boldsymbol x_i)) \,. \]

The Hessian of this objective is \( H = -X^T A X \) where \( A = \text{diag}(a_1, \dots, a_n) \) is the diagonal matrix with \( a_i = \sigma(\boldsymbol c^T \boldsymbol x) \cdot \sigma(-\boldsymbol c^T \boldsymbol x) \,. \) Since \( H \) is non-positive definite, \( l(\boldsymbol c) \) is convex. There are many techniques for solving convex optimization problems. Currently, logistic regression in MADlib can use:

- Iteratively Reweighted Least Squares

We estimate the standard error for coefficient \( i \) as

\[ \mathit{se}(c_i) = \left( (X^T A X)^{-1} \right)_{ii} \,. \]

The Wald z-statistic is

\[ z_i = \frac{c_i}{\mathit{se}(c_i)} \,. \]

The Wald \( p \)-value for coefficient \( i \) gives the probability (under the assumptions inherent in the Wald test) of seeing a value at least as extreme as the one observed, provided that the null hypothesis ( \( c_i = 0 \)) is true. Letting \( F \) denote the cumulative density function of a standard normal distribution, the Wald \( p \)-value for coefficient \( i \) is therefore

\[ p_i = \Pr(|Z| \geq |z_i|) = 2 \cdot (1 - F( |z_i| )) \]

where \( Z \) is a standard normally distributed random variable.

The odds ratio for coefficient \( i \) is estimated as \( \exp(c_i) \).

The condition number is computed as \( \kappa(X^T A X) \) during the iteration immediately *preceding* convergence (i.e., \( A \) is computed using the coefficients of the previous iteration). A large condition number (say, more than 1000) indicates the presence of significant multicollinearity.

The multinomial logistic regression uses a default reference category of zero, and the regression coefficients in the output are in the order described below. For a problem with \( K \) dependent variables \( (1, ..., K) \) and \( J \) categories \( (0, ..., J-1) \), let \( {m_{k,j}} \) denote the coefficient for dependent variable \( k \) and category \( j \). The output is \( {m_{k_1, j_0}, m_{k_1, j_1} \ldots m_{k_1, j_{J-1}}, m_{k_2, j_0}, m_{k_2, j_1}, \ldots m_{k_2, j_{J-1}} \ldots m_{k_K, j_{J-1}}} \). The order is NOT CONSISTENT with the multinomial regression marginal effect calculation with function *marginal_mlogregr*. This is deliberate because the interfaces of all multinomial regressions (robust, clustered, ...) will be moved to match that used in marginal.

- Literature

A collection of nice write-ups, with valuable pointers into further literature:

[1] Annette J. Dobson: An Introduction to Generalized Linear Models, Second Edition. Nov 2001

[2] Cosma Shalizi: Statistics 36-350: Data Mining, Lecture Notes, 18 November 2009, http://www.stat.cmu.edu/~cshalizi/350/lectures/26/lecture-26.pdf

[3] Scott A. Czepiel: Maximum Likelihood Estimation of Logistic Regression Models: Theory and Implementation, Retrieved Jul 12 2012, http://czep.net/stat/mlelr.pdf

- Related Topics

File multilogistic.sql_in documenting the multinomial logistic regression functions