Evaluating the Efficacy of Multiple Imputation with Basic Generalized Linear Models

1 Vignette Overview

This vignette extends the multiple imputation (MI) summary vignette, “Evaluating the Efficacy of Multiple Imputation Methods for Missing Educational Assessment and Growth Data.” Here, we run a series of fixed-effects regression models to better understand the relationships among different indices of MI efficacy and both data and school characteristics (e.g., missingness type, school size, etc.). For a more detailed recap of the MI simulation, please see the summary vignette. In this simulation, no COVID-19 impact was incorporated into the synthetic data.

Our primary research questions can be stated as follows:

• Is MI an appropriate method for creating “adjusted” scale scores and SGPs when data are missing?
• If MI is indeed appropriate, which method is most effective and in what data contexts?

In this vignette, we fit linear or logistic fixed-effect regression models. Specifically, we use the fixest function (Berge, 2018) for fixed effects models. Fixed effects for grade and content area are included when appropriate. We also include fixed effects for the amputation number (AMP_N).

2 Raw Bias

In the following models, the dependent variable is the raw bias for either the scale scores or student growth percentiles (SPGs). Raw bias is calculated as the average difference between the true and imputed values.

2.1 Scale Scores

2.1.1 Grade and Content Area Level

## NOTE: 11,077 observations removed because of NA values (LHS: 11,077).
## NOTE: 11,077 observations removed because of NA values (LHS: 11,077).

Table 2.1: Fixed-effect regression models for scale score raw bias at the grade/content area level, all grade levels

	Additive Model	Interaction Model
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER_LONG	-0.2324 (0.8332)	-0.2557 (0.8270)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG	-6.186*** (1.290)	-6.209*** (1.284)
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER	1.706*** (0.2581)	1.683*** (0.2510)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN	-5.244*** (1.025)	-5.267*** (1.019)
i(var=IMP_METHOD_REF,ref=TRUE)RQ	-5.274*** (1.034)	-5.297*** (1.027)
i(var=IMP_METHOD_REF,ref=TRUE)PMM	-5.281*** (1.046)	-5.304*** (1.040)
N	0.0832 (0.1535)	0.1333 (0.1458)
Percent_Missing	3.779*** (0.6258)	-0.1060 (0.0726)
MISS_TYPESTATUS_w_DEMOG	5.639*** (0.6895)	5.713*** (0.6871)
MISS_TYPESTATUS_w_GROWTH	10.20*** (1.541)	10.24*** (1.533)
N x Percent_Missing		-0.0397 (0.1990)
N x MISS_TYPESTATUS_w_DEMOG		-0.0203 (0.1338)
N x MISS_TYPESTATUS_w_GROWTH		0.0577 (0.2219)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		3.449*** (0.4913)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		6.183*** (1.021)
Fixed-Effects:	—————–	—————–
CONTENT_AREA^GRADE	Yes	Yes
AMP_N	Yes	Yes
________________________________________	_________________	_________________
S.E.: Clustered	by: CON.^GRA.	by: CON.^GRA.
R2	0.40014	0.44094
Within Adj. R2	0.37081	0.41361
AIC	33,947,231.5	33,609,582.4
BIC	33,948,181.7	33,610,599.5

\(~\)

Figure 2.1: Coefficient plot, additive models for scale score raw bias at the grade/content area level

Figure 2.2: Fixed effects estimates, additive model

Figure 2.3: Fixed effects estimates, interaction model

We can also subset the data to only observations from grades 5 through 8.

## NOTE: 4,102 observations removed because of NA values (LHS: 4,102).
## NOTE: 4,102 observations removed because of NA values (LHS: 4,102).

Table 2.2: Fixed-effect regression models for scale score raw bias at the grade/content area level, grades 5 through 8

	Additive Model	Interaction Model
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER_LONG	-2.558*** (0.4173)	-2.565*** (0.4171)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG	-9.891*** (0.7507)	-9.898*** (0.7504)
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER	1.034*** (0.0939)	1.027*** (0.0933)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN	-8.145*** (0.3334)	-8.152*** (0.3327)
i(var=IMP_METHOD_REF,ref=TRUE)RQ	-8.187*** (0.2972)	-8.194*** (0.2965)
i(var=IMP_METHOD_REF,ref=TRUE)PMM	-8.247*** (0.2629)	-8.254*** (0.2623)
N	0.3559* (0.1044)	0.3004* (0.1159)
Percent_Missing	2.019*** (0.1390)	-0.0843 (0.0605)
MISS_TYPESTATUS_w_DEMOG	3.714*** (0.2449)	3.769*** (0.2417)
MISS_TYPESTATUS_w_GROWTH	5.861*** (0.3684)	5.892*** (0.3707)
N x Percent_Missing		0.3549** (0.0997)
N x MISS_TYPESTATUS_w_DEMOG		0.2801*** (0.0326)
N x MISS_TYPESTATUS_w_GROWTH		0.2824** (0.0675)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		1.977*** (0.1608)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		3.225*** (0.2755)
Fixed-Effects:	——————	——————
CONTENT_AREA^GRADE	Yes	Yes
AMP_N	Yes	Yes
________________________________________	__________________	__________________
S.E.: Clustered	by: CON.^GRA.	by: CON.^GRA.
R2	0.32730	0.34360
Within Adj. R2	0.32609	0.34242
AIC	17,800,395.6	17,736,913.5
BIC	17,801,250.9	17,737,832.7

\(~\)

Figure 2.4: Coefficient plot, additive models for scale score raw bias at the grade/content area level, grades 5 through 8

Figure 2.5: Fixed effects estimates, additive model, grades 5 through 8

Figure 2.6: Fixed effects estimates, interaction model, grades 5 through 8

2.1.2 School Level

## NOTE: 25 observations removed because of NA values (LHS: 25).
## NOTE: 25 observations removed because of NA values (LHS: 25).

Table 2.3: Fixed-effect regression models for scale score raw bias at the school level

	Additive Model	Interaction Model
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER_LONG	-0.7323*** (0.0098)	-0.7327*** (0.0098)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG	-6.655*** (0.0115)	-6.656*** (0.0115)
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER	1.255*** (0.0090)	1.254*** (0.0090)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN	-5.684*** (0.0117)	-5.684*** (0.0116)
i(var=IMP_METHOD_REF,ref=TRUE)RQ	-5.708*** (0.0119)	-5.708*** (0.0118)
i(var=IMP_METHOD_REF,ref=TRUE)PMM	-5.699*** (0.0119)	-5.699*** (0.0119)
N	-0.6250*** (0.0034)	0.0746*** (0.0057)
Percent_Missing	3.316*** (0.0103)	-0.0253 (0.0168)
MISS_TYPESTATUS_w_DEMOG	5.665*** (0.0170)	5.708*** (0.0170)
MISS_TYPESTATUS_w_GROWTH	10.24*** (0.0186)	10.23*** (0.0188)
N x Percent_Missing		-0.5374*** (0.0062)
N x MISS_TYPESTATUS_w_DEMOG		-0.8081*** (0.0114)
N x MISS_TYPESTATUS_w_GROWTH		-1.540*** (0.0097)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		2.954*** (0.0170)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		5.335*** (0.0183)
Fixed-Effects:	——————-	——————-
AMP_N	Yes	Yes
________________________________________	___________________	___________________
S.E.: Clustered	by: AMP_N	by: AMP_N
R2	0.55194	0.61186
Within Adj. R2	0.55191	0.61183
AIC	4,597,267.2	4,492,367.8
BIC	4,597,957.3	4,493,115.4

\(~\)

Figure 2.7: Coefficient plot, additive models for scale score raw bias at the school level

Figure 2.8: Fixed effects estimates, additive model

Figure 2.9: Fixed effects estimates, interaction model

2.2 SGPs

2.2.1 Grade and Content Area Level

## NOTE: 4,102 observations removed because of NA values (LHS: 4,102).
## NOTE: 4,102 observations removed because of NA values (LHS: 4,102).

Table 2.4: Fixed-effect regression models for SGP raw bias at the grade/content area level, grades 5 through 8

	Additive Model	Interaction Model
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER_LONG	-0.3580. (0.1814)	-0.3550. (0.1812)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG	-0.3835. (0.1924)	-0.3805. (0.1922)
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER	-0.3053 (0.1763)	-0.3024 (0.1761)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN	-0.0119 (0.1789)	-0.0090 (0.1785)
i(var=IMP_METHOD_REF,ref=TRUE)RQ	-0.2394 (0.1966)	-0.2364 (0.1963)
i(var=IMP_METHOD_REF,ref=TRUE)PMM	-0.3231 (0.1992)	-0.3202 (0.1990)
N	0.7092** (0.1421)	0.6919*** (0.1264)
Percent_Missing	0.2056. (0.0986)	-0.1821** (0.0462)
MISS_TYPESTATUS_w_DEMOG	-0.0665* (0.0276)	-0.0655. (0.0311)
MISS_TYPESTATUS_w_GROWTH	0.0604 (0.0462)	0.0815 (0.0628)
N x Percent_Missing		0.4945** (0.1404)
N x MISS_TYPESTATUS_w_DEMOG		0.2592** (0.0524)
N x MISS_TYPESTATUS_w_GROWTH		0.1614. (0.0794)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		0.4001* (0.1455)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		0.4527** (0.1191)
Fixed-Effects:	—————–	——————
CONTENT_AREA^GRADE	Yes	Yes
AMP_N	Yes	Yes
________________________________________	_________________	__________________
S.E.: Clustered	by: CON.^GRA.	by: CON.^GRA.
R2	0.00679	0.00970
Within Adj. R2	0.00627	0.00918
AIC	17,515,356.5	17,507,769.4
BIC	17,516,211.8	17,508,688.6

\(~\)

Figure 2.10: Coefficient plot, additive models for SGP raw bias at the grade/content area level, grades 5 through 8

Figure 2.11: Fixed effects estimates, additive model

Figure 2.12: Fixed effects estimates, interaction model

2.2.2 School Level

## NOTE: 323 observations removed because of NA values (LHS: 323).
## NOTE: 323 observations removed because of NA values (LHS: 323).

Table 2.5: Fixed-effect regression models for SGP raw bias at the school level

	Additive Model	Interaction Model
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER_LONG	-0.4025*** (0.0095)	-0.4014*** (0.0095)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG	-0.4144*** (0.0095)	-0.4133*** (0.0095)
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER	-0.3849*** (0.0098)	-0.3838*** (0.0098)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN	0.0393*** (0.0083)	0.0404*** (0.0083)
i(var=IMP_METHOD_REF,ref=TRUE)RQ	-0.2934*** (0.0096)	-0.2923*** (0.0096)
i(var=IMP_METHOD_REF,ref=TRUE)PMM	-0.3117*** (0.0096)	-0.3106*** (0.0096)
N	0.2986*** (0.0030)	0.2857*** (0.0055)
Percent_Missing	0.1553*** (0.0068)	-0.1879*** (0.0104)
MISS_TYPESTATUS_w_DEMOG	-0.1455*** (0.0080)	-0.1373*** (0.0081)
MISS_TYPESTATUS_w_GROWTH	-0.0367*** (0.0070)	-0.0254*** (0.0070)
N x Percent_Missing		0.2572*** (0.0057)
N x MISS_TYPESTATUS_w_DEMOG		0.1297*** (0.0083)
N x MISS_TYPESTATUS_w_GROWTH		0.0763*** (0.0078)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		0.3622*** (0.0117)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		0.5417*** (0.0121)
Fixed-Effects:	——————-	——————-
AMP_N	Yes	Yes
________________________________________	___________________	___________________
S.E.: Clustered	by: AMP_N	by: AMP_N
R2	0.00494	0.00804
Within Adj. R2	0.00491	0.00801
AIC	4,552,053.2	4,549,782.1
BIC	4,552,743.3	4,550,529.7

Figure 2.13: Coefficient plot, additive models for SGP raw bias at the school level

Figure 2.14: Fixed effects estimates, additive model

Figure 2.15: Fixed effects estimates, interaction model

2.3 Random Effects Models

We introduce a random intercept for school number to account for the hierarchical nature of the data. We begin with a relatively simple model using only additive fixed effects. The mixed-effects models are run with the lme4 package (Bates et al., 2015).

## Linear mixed model fit by REML ['lmerMod']
## Formula: SS_Raw_Bias ~ IMP_METHOD_REF + N + Percent_Missing + MISS_TYPE +  
##     GRADE + CONTENT_AREA + (1 | SCHOOL_NUMBER)
##    Data: MIsummary.GC.Eval
## 
## REML criterion at convergence: 644920.2
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -6.6453 -0.5513 -0.0056  0.5273  8.3999 
## 
## Random effects:
##  Groups        Name        Variance Std.Dev.
##  SCHOOL_NUMBER (Intercept)  8.254   2.873   
##  Residual                  47.661   6.904   
## Number of obs: 96075, groups:  SCHOOL_NUMBER, 231
## 
## Fixed effects:
##                           Estimate Std. Error t value
## (Intercept)                6.09879    0.20734  29.414
## IMP_METHOD_REFL2LMER_LONG -0.24098    0.08334  -2.892
## IMP_METHOD_REFL2PAN_LONG  -6.19451    0.08334 -74.330
## IMP_METHOD_REFL2LMER       1.69760    0.08334  20.370
## IMP_METHOD_REFL2PAN       -5.25269    0.08334 -63.029
## IMP_METHOD_REFRQ          -5.28221    0.08334 -63.383
## IMP_METHOD_REFPMM         -5.28980    0.08334 -63.474
## N                         -0.20383    0.05372  -3.794
## Percent_Missing            3.03121    0.02545 119.123
## MISS_TYPESTATUS_w_DEMOG    5.65238    0.05456 103.600
## MISS_TYPESTATUS_w_GROWTH  10.23550    0.05457 187.574
## GRADE4                    -0.30037    0.06633  -4.528
## GRADE5                    -4.35097    0.06714 -64.804
## GRADE6                    -5.80837    0.08156 -71.217
## GRADE7                    -7.16628    0.13106 -54.678
## GRADE8                    -7.43450    0.13520 -54.990
## CONTENT_AREAMATHEMATICS    0.19778    0.04455   4.440

## 
## Correlation matrix not shown by default, as p = 17 > 12.
## Use print(x, correlation=TRUE)  or
##     vcov(x)        if you need it

## Linear mixed model fit by REML ['lmerMod']
## Formula: SGP_Raw_Bias ~ IMP_METHOD_REF + N + Percent_Missing + MISS_TYPE +  
##     GRADE + CONTENT_AREA + (1 | SCHOOL_NUMBER)
##    Data: MIsummary.GC.Eval
## 
## REML criterion at convergence: 318310.9
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -8.2156 -0.5150  0.0037  0.5283  6.5146 
## 
## Random effects:
##  Groups        Name        Variance Std.Dev.
##  SCHOOL_NUMBER (Intercept) 17.92    4.233   
##  Residual                  26.56    5.153   
## Number of obs: 51849, groups:  SCHOOL_NUMBER, 223
## 
## Fixed effects:
##                           Estimate Std. Error t value
## (Intercept)                0.77256    0.29490   2.620
## IMP_METHOD_REFL2LMER_LONG -0.37684    0.08468  -4.450
## IMP_METHOD_REFL2PAN_LONG  -0.40233    0.08468  -4.751
## IMP_METHOD_REFL2LMER      -0.32419    0.08468  -3.828
## IMP_METHOD_REFL2PAN       -0.03080    0.08468  -0.364
## IMP_METHOD_REFRQ          -0.25822    0.08468  -3.049
## IMP_METHOD_REFPMM         -0.34199    0.08468  -4.039
## N                          0.48541    0.08736   5.556
## Percent_Missing           -0.01854    0.02599  -0.713
## MISS_TYPESTATUS_w_DEMOG   -0.04881    0.05545  -0.880
## MISS_TYPESTATUS_w_GROWTH   0.07582    0.05546   1.367
## GRADE6                    -1.24123    0.06776 -18.319
## GRADE7                    -3.41144    0.12027 -28.364
## GRADE8                    -3.42940    0.12512 -27.408
## CONTENT_AREAMATHEMATICS    0.14602    0.04527   3.225

## 
## Correlation matrix not shown by default, as p = 15 > 12.
## Use print(x, correlation=TRUE)  or
##     vcov(x)        if you need it

3 Absolute Bias

Here, the dependent variable is absolute bias, computed by taking the absolute value of the raw bias (for either the scale scores or SGPs). In the additive models, a quadratic term for grade/content area or school size (N) is added based on previous figures indicating a curved relationship between bias and N.

3.1 Scale Scores

3.1.1 Grade and Content Area Level

The first set of models includes all grades (i.e., grade 3 through 8).

## NOTE: 11,077 observations removed because of NA values (LHS: 11,077).
## NOTE: 11,077 observations removed because of NA values (LHS: 11,077).

Table 3.1: Fixed-effect regression models for scale score absolute bias at the grade/content area level, all grade levels

	Additive Model	Interaction Model
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER_LONG	-0.0108 (0.7794)	-0.0284 (0.7758)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG	-4.102*** (0.5919)	-4.119*** (0.5885)
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER	1.809*** (0.2480)	1.791*** (0.2440)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN	-5.041*** (0.8219)	-5.059*** (0.8185)
i(var=IMP_METHOD_REF,ref=TRUE)RQ	-4.692*** (0.7185)	-4.710*** (0.7151)
i(var=IMP_METHOD_REF,ref=TRUE)PMM	-4.710*** (0.7320)	-4.728*** (0.7286)
N	-0.9729*** (0.1601)	-0.8585*** (0.0880)
I(I(N^2))	0.1474** (0.0394)
Percent_Missing	4.680*** (0.5179)	2.084*** (0.1115)
MISS_TYPESTATUS_w_DEMOG	2.828*** (0.3126)	2.838*** (0.3143)
MISS_TYPESTATUS_w_GROWTH	7.048*** (1.085)	6.984*** (1.078)
N x Percent_Missing		-0.4275** (0.1314)
N x MISS_TYPESTATUS_w_DEMOG		0.1927 (0.1092)
N x MISS_TYPESTATUS_w_GROWTH		0.5076* (0.2054)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		2.014*** (0.2752)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		4.535*** (0.8021)
Fixed-Effects:	——————-	——————-
CONTENT_AREA^GRADE	Yes	Yes
AMP_N	Yes	Yes
________________________________________	___________________	___________________
S.E.: Clustered	by: CON.^GRA.	by: CON.^GRA.
R2	0.49143	0.52517
Within Adj. R2	0.47187	0.50691
AIC	31,559,088.2	31,230,024.8
BIC	31,560,051.7	31,231,041.9

\(~\)

Figure 3.1: Coefficient plot, additive models for scale score absolute bias at the grade/content area level

Figure 3.2: Fixed effects estimates, additive model

Figure 3.3: Fixed effects estimates, interaction model

We can also subset the data to only observations from grades 5 through 8.

## NOTE: 4,102 observations removed because of NA values (LHS: 4,102).
## NOTE: 4,102 observations removed because of NA values (LHS: 4,102).

Table 3.2: Fixed-effect regression models for scale score absolute bias at the grade/content area level, grades 5 through 8

	Additive Model	Interaction Model
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER_LONG	-2.227*** (0.2797)	-2.234*** (0.2797)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG	-5.623*** (0.4028)	-5.630*** (0.4026)
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER	1.145*** (0.0991)	1.137*** (0.0989)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN	-7.364*** (0.2267)	-7.371*** (0.2267)
i(var=IMP_METHOD_REF,ref=TRUE)RQ	-6.697*** (0.2560)	-6.704*** (0.2559)
i(var=IMP_METHOD_REF,ref=TRUE)PMM	-6.751*** (0.2546)	-6.758*** (0.2544)
N	-0.9539*** (0.1658)	-0.8217*** (0.1001)
I(I(N^2))	0.1365** (0.0366)
Percent_Missing	3.227*** (0.0922)	1.888*** (0.1031)
MISS_TYPESTATUS_w_DEMOG	2.004*** (0.0733)	2.003*** (0.0973)
MISS_TYPESTATUS_w_GROWTH	4.070*** (0.1308)	4.017*** (0.1463)
N x Percent_Missing		-0.2753** (0.0675)
N x MISS_TYPESTATUS_w_DEMOG		0.2558* (0.0772)
N x MISS_TYPESTATUS_w_GROWTH		0.4347** (0.1065)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		1.204*** (0.0388)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		2.238*** (0.0567)
Fixed-Effects:	——————-	——————-
CONTENT_AREA^GRADE	Yes	Yes
AMP_N	Yes	Yes
________________________________________	___________________	___________________
S.E.: Clustered	by: CON.^GRA.	by: CON.^GRA.
R2	0.43869	0.45014
Within Adj. R2	0.43672	0.44821
AIC	16,257,613.7	16,204,277.2
BIC	16,258,481.9	16,205,196.4

\(~\)

Figure 3.4: Coefficient plot, additive models for scale score absolute bias at the grade/content area level, grades 5 through 8

Figure 3.5: Fixed effects estimates, additive model, grades 5 through 8

Figure 3.6: Fixed effects estimates, interaction model, grades 5 through 8

3.1.2 School Level

## NOTE: 25 observations removed because of NA values (LHS: 25).
## NOTE: 25 observations removed because of NA values (LHS: 25).

Table 3.3: Fixed-effect regression models for scale score absolute bias at the school level

	Additive Model	Interaction Model
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER_LONG	0.0967*** (0.0079)	0.0966*** (0.0079)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG	-5.248*** (0.0081)	-5.248*** (0.0081)
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER	2.132*** (0.0066)	2.132*** (0.0066)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN	-5.635*** (0.0085)	-5.635*** (0.0084)
i(var=IMP_METHOD_REF,ref=TRUE)RQ	-5.331*** (0.0083)	-5.331*** (0.0083)
i(var=IMP_METHOD_REF,ref=TRUE)PMM	-5.341*** (0.0083)	-5.341*** (0.0082)
N	-0.9733*** (0.0053)	-0.3975*** (0.0027)
I(I(N^2))	0.0935*** (0.0008)
Percent_Missing	3.675*** (0.0070)	1.138*** (0.0067)
MISS_TYPESTATUS_w_DEMOG	3.598*** (0.0131)	3.628*** (0.0129)
MISS_TYPESTATUS_w_GROWTH	8.043*** (0.0162)	8.025*** (0.0159)
N x Percent_Missing		-0.5125*** (0.0039)
N x MISS_TYPESTATUS_w_DEMOG		-0.2828*** (0.0072)
N x MISS_TYPESTATUS_w_GROWTH		-0.7110*** (0.0064)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		1.978*** (0.0131)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		4.323*** (0.0132)
Fixed-Effects:	——————-	——————-
AMP_N	Yes	Yes
________________________________________	___________________	___________________
S.E.: Clustered	by: AMP_N	by: AMP_N
R2	0.61641	0.66456
Within Adj. R2	0.61638	0.66453
AIC	4,321,552.0	4,223,551.7
BIC	4,322,253.7	4,224,299.3

\(~\)

Figure 3.7: Coefficient plot, additive models for scale score absolute bias at the school level

Figure 3.8: Fixed effects estimates, additive model

Figure 3.9: Fixed effects estimates, interaction model

3.2 SGPs

3.2.1 Grade and Content Area Level

## NOTE: 4,102 observations removed because of NA values (LHS: 4,102).
## NOTE: 4,102 observations removed because of NA values (LHS: 4,102).

Table 3.4: Fixed-effect regression models for SGP absolute bias at the grade/content area level, grades 5 through 8

	Additive Model	Interaction Model
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER_LONG	0.9489* (0.3121)	0.9458* (0.3117)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG	0.6517. (0.3178)	0.6486. (0.3175)
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER	1.038** (0.2893)	1.035** (0.2890)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN	-0.8256*** (0.1314)	-0.8287*** (0.1311)
i(var=IMP_METHOD_REF,ref=TRUE)RQ	0.2699 (0.2936)	0.2668 (0.2931)
i(var=IMP_METHOD_REF,ref=TRUE)PMM	0.1884 (0.3031)	0.1853 (0.3026)
N	-1.863*** (0.2399)	-1.083*** (0.1177)
I(I(N^2))	0.2727** (0.0575)
Percent_Missing	2.087*** (0.1136)	2.027*** (0.1101)
MISS_TYPESTATUS_w_DEMOG	0.3566*** (0.0341)	0.3611*** (0.0380)
MISS_TYPESTATUS_w_GROWTH	0.2052** (0.0392)	0.1913** (0.0435)
N x Percent_Missing		-0.5985** (0.1207)
N x MISS_TYPESTATUS_w_DEMOG		-0.2223. (0.1127)
N x MISS_TYPESTATUS_w_GROWTH		-0.1768 (0.1181)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		0.1016 (0.0893)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		0.1658. (0.0780)
Fixed-Effects:	——————-	——————-
CONTENT_AREA^GRADE	Yes	Yes
AMP_N	Yes	Yes
________________________________________	___________________	___________________
S.E.: Clustered	by: CON.^GRA.	by: CON.^GRA.
R2	0.23539	0.23174
Within Adj. R2	0.22910	0.22542
AIC	15,009,632.6	15,021,969.2
BIC	15,010,500.7	15,022,888.4

\(~\)

Figure 3.10: Coefficient plot, additive models for SGP absolute bias at the grade/content area level, grades 5 through 8

Figure 3.11: Fixed effects estimates, additive model

Figure 3.12: Fixed effects estimates, interaction model

3.2.2 School Level

## NOTE: 323 observations removed because of NA values (LHS: 323).
## NOTE: 323 observations removed because of NA values (LHS: 323).

Table 3.5: Fixed-effect regression models for SGP absolute bias at the school level

	Additive Model	Interaction Model
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER_LONG	1.420*** (0.0106)	1.419*** (0.0106)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG	1.182*** (0.0107)	1.181*** (0.0107)
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER	1.504*** (0.0106)	1.503*** (0.0106)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN	-0.3762*** (0.0106)	-0.3773*** (0.0106)
i(var=IMP_METHOD_REF,ref=TRUE)RQ	0.8057*** (0.0113)	0.8046*** (0.0112)
i(var=IMP_METHOD_REF,ref=TRUE)PMM	0.6609*** (0.0111)	0.6598*** (0.0110)
N	-1.289*** (0.0048)	-0.7619*** (0.0039)
I(I(N^2))	0.1409*** (0.0007)
Percent_Missing	1.447*** (0.0046)	1.403*** (0.0101)
MISS_TYPESTATUS_w_DEMOG	0.3515*** (0.0071)	0.3463*** (0.0071)
MISS_TYPESTATUS_w_GROWTH	0.1459*** (0.0083)	0.1293*** (0.0083)
N x Percent_Missing		-0.4625*** (0.0035)
N x MISS_TYPESTATUS_w_DEMOG		-0.2529*** (0.0058)
N x MISS_TYPESTATUS_w_GROWTH		-0.1403*** (0.0062)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		-0.0217* (0.0099)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		-0.0075 (0.0115)
Fixed-Effects:	——————-	——————-
AMP_N	Yes	Yes
________________________________________	___________________	___________________
S.E.: Clustered	by: AMP_N	by: AMP_N
R2	0.21857	0.21503
Within Adj. R2	0.21853	0.21498
AIC	3,897,692.5	3,901,004.7
BIC	3,898,394.1	3,901,752.2

\(~\)

Figure 3.13: Coefficient plot, additive models for SGP absolute bias at the school level

Figure 3.14: Fixed effects estimates, additive model

Figure 3.15: Fixed effects estimates, interaction model

3.3 Random Effects Models

We introduce a random intercept for school number to account for the hierarchical nature of the data.

## Linear mixed model fit by REML ['lmerMod']
## Formula: abs(SS_Raw_Bias) ~ IMP_METHOD_REF + N + Percent_Missing + MISS_TYPE +  
##     GRADE + CONTENT_AREA + (1 | SCHOOL_NUMBER)
##    Data: MIsummary.GC.Eval
## 
## REML criterion at convergence: 607762.3
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -5.2048 -0.5951 -0.0523  0.4934 10.3190 
## 
## Random effects:
##  Groups        Name        Variance Std.Dev.
##  SCHOOL_NUMBER (Intercept)  3.037   1.743   
##  Residual                  32.419   5.694   
## Number of obs: 96075, groups:  SCHOOL_NUMBER, 231
## 
## Fixed effects:
##                           Estimate Std. Error t value
## (Intercept)                6.76979    0.13439  50.374
## IMP_METHOD_REFL2LMER_LONG  1.11660    0.06873  16.246
## IMP_METHOD_REFL2PAN_LONG  -3.35460    0.06873 -48.807
## IMP_METHOD_REFL2LMER       2.95948    0.06873  43.058
## IMP_METHOD_REFL2PAN       -4.62051    0.06873 -67.225
## IMP_METHOD_REFRQ          -3.92125    0.06873 -57.052
## IMP_METHOD_REFPMM         -3.90565    0.06873 -56.825
## N                         -0.44658    0.04294 -10.399
## Percent_Missing            4.13906    0.02095 197.573
## MISS_TYPESTATUS_w_DEMOG    3.52260    0.04500  78.285
## MISS_TYPESTATUS_w_GROWTH   8.09505    0.04500 179.874
## GRADE4                    -0.30413    0.05469  -5.561
## GRADE5                    -3.03511    0.05536 -54.827
## GRADE6                    -3.58967    0.06702 -53.559
## GRADE7                    -3.75567    0.10651 -35.261
## GRADE8                    -3.77832    0.10982 -34.406
## CONTENT_AREAMATHEMATICS    0.43593    0.03674  11.865

## 
## Correlation matrix not shown by default, as p = 17 > 12.
## Use print(x, correlation=TRUE)  or
##     vcov(x)        if you need it

## Linear mixed model fit by REML ['lmerMod']
## Formula: 
## abs(SGP_Raw_Bias) ~ IMP_METHOD_REF + N + Percent_Missing + MISS_TYPE +  
##     GRADE + CONTENT_AREA + (1 | SCHOOL_NUMBER)
##    Data: MIsummary.GC.Eval
## 
## REML criterion at convergence: 275879.9
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -3.5939 -0.6004 -0.0583  0.4889  9.0994 
## 
## Random effects:
##  Groups        Name        Variance Std.Dev.
##  SCHOOL_NUMBER (Intercept)  3.934   1.983   
##  Residual                  11.747   3.427   
## Number of obs: 51849, groups:  SCHOOL_NUMBER, 223
## 
## Fixed effects:
##                           Estimate Std. Error t value
## (Intercept)                2.04260    0.14336  14.248
## IMP_METHOD_REFL2LMER_LONG  2.58124    0.05632  45.832
## IMP_METHOD_REFL2PAN_LONG   2.31242    0.05632  41.059
## IMP_METHOD_REFL2LMER       2.64339    0.05632  46.935
## IMP_METHOD_REFL2PAN        0.09846    0.05632   1.748
## IMP_METHOD_REFRQ           1.84838    0.05632  32.819
## IMP_METHOD_REFPMM          1.74888    0.05632  31.053
## N                         -0.50955    0.05542  -9.195
## Percent_Missing            1.79096    0.01727 103.705
## MISS_TYPESTATUS_w_DEMOG    0.47115    0.03688  12.776
## MISS_TYPESTATUS_w_GROWTH   0.47238    0.03689  12.805
## GRADE6                     0.52239    0.04491  11.632
## GRADE7                     0.24475    0.07849   3.118
## GRADE8                     0.78995    0.08153   9.689
## CONTENT_AREAMATHEMATICS    0.72984    0.03011  24.239

## 
## Correlation matrix not shown by default, as p = 15 > 12.
## Use print(x, correlation=TRUE)  or
##     vcov(x)        if you need it

4 Percent Bias

In the following models, percent bias is dichotomized such that 1 indicates an observation with percent bias less than five, and 0 otherwise. Then, a series of logistic regression models are fit, examining the factors associated with a percent bias value lower than the 5% threshold (Miri et al., 2020; Qi et al., 2010).

4.1 Scale Scores

4.1.1 Grade and Content Area Level

The first set of models includes all grades (i.e., grade 3 through 8).

## NOTE: 11,077 observations removed because of NA values (LHS: 11,077).
## NOTE: 11,077 observations removed because of NA values (LHS: 11,077).

Table 4.1: Fixed-effect regression models for scale score percent bias classification at the grade/content area level, all grade levels

	Additive Model	Interaction Model
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER_LONG	0.0873 (0.4177)	0.0817 (0.4232)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG	2.156*** (0.3846)	2.162*** (0.3784)
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER	-0.7487*** (0.1383)	-0.7616*** (0.1416)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN	2.225*** (0.4191)	2.231*** (0.4122)
i(var=IMP_METHOD_REF,ref=TRUE)RQ	2.205*** (0.4119)	2.211*** (0.4052)
i(var=IMP_METHOD_REF,ref=TRUE)PMM	2.217*** (0.4167)	2.222*** (0.4099)
N	0.2418*** (0.0515)	1.997*** (0.2081)
Percent_Missing	-2.848*** (0.1400)	-2.950*** (0.1333)
MISS_TYPESTATUS_w_DEMOG	-1.470*** (0.1325)	-2.055*** (0.2073)
MISS_TYPESTATUS_w_GROWTH	-3.495*** (0.1812)	-3.720*** (0.2396)
N x Percent_Missing		-0.5649*** (0.0999)
N x MISS_TYPESTATUS_w_DEMOG		-0.7127*** (0.1008)
N x MISS_TYPESTATUS_w_GROWTH		-1.357*** (0.1662)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		0.2711. (0.1521)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		-0.0210 (0.2507)
Fixed-Effects:	——————-	——————-
CONTENT_AREA^GRADE	Yes	Yes
AMP_N	Yes	Yes
________________________________________	___________________	___________________
S.E.: Clustered	by: CON.^GRA.	by: CON.^GRA.
Convergence	TRUE	TRUE
AIC	985,765.3	975,875.8
BIC	986,715.5	976,892.9

\(~\)

Figure 4.1: Coefficient plot, additive model for scale score percent bias classification at the grade/content area level

Figure 4.2: Fixed effects estimates, additive model

Figure 4.3: Fixed effects estimates, interaction model

We can also subset the data to only observations from grades 5 through 8.

## NOTE: 4,102 observations removed because of NA values (LHS: 4,102).
## NOTE: 4,102 observations removed because of NA values (LHS: 4,102).

Table 4.2: Fixed-effect regression models for scale score percent bias classification at the grade/content area level, grades 5 through 8

	Additive Model	Interaction Model
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER_LONG	1.442*** (0.1752)	1.431*** (0.1715)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG	4.241*** (0.4041)	4.212*** (0.4100)
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER	-0.3772*** (0.0668)	-0.3821*** (0.0670)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN	5.867*** (0.3929)	5.835*** (0.3877)
i(var=IMP_METHOD_REF,ref=TRUE)RQ	5.325*** (0.5551)	5.294*** (0.5432)
i(var=IMP_METHOD_REF,ref=TRUE)PMM	5.587*** (0.6019)	5.555*** (0.5916)
N	0.2688** (0.0984)	2.225*** (0.4533)
Percent_Missing	-2.446*** (0.1130)	-2.829*** (0.2699)
MISS_TYPESTATUS_w_DEMOG	-1.530*** (0.1424)	-2.175*** (0.3008)
MISS_TYPESTATUS_w_GROWTH	-2.934*** (0.2224)	-3.506*** (0.4284)
N x Percent_Missing		-0.5670*** (0.1682)
N x MISS_TYPESTATUS_w_DEMOG		-0.9727*** (0.2682)
N x MISS_TYPESTATUS_w_GROWTH		-1.588*** (0.3627)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		0.4111* (0.1613)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		0.3470. (0.1949)
Fixed-Effects:	——————-	——————-
CONTENT_AREA^GRADE	Yes	Yes
AMP_N	Yes	Yes
________________________________________	___________________	___________________
S.E.: Clustered	by: CON.^GRA.	by: CON.^GRA.
Convergence	TRUE	TRUE
AIC	342,920.5	339,735.3
BIC	343,775.9	340,654.5

\(~\)

Figure 4.4: Coefficient plot, additive model for scale score percent bias classification at the grade/content area level, grades 5 through 8

Figure 4.5: Fixed effects estimates, additive model, grades 5 through 8

Figure 4.6: Fixed effects estimates, interaction model, grades 5 through 8

4.1.2 School Level

## NOTE: 25 observations removed because of NA values (LHS: 25).
## NOTE: 25 observations removed because of NA values (LHS: 25).

Table 4.3: Fixed-effect regression models for scale score percent bias classification at the school level

	Additive Model	Interaction Model
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER_LONG	0.1201*** (0.0170)	0.1215*** (0.0173)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG	3.944*** (0.0488)	4.010*** (0.0492)
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER	-1.246*** (0.0137)	-1.256*** (0.0138)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN	4.027*** (0.0385)	4.093*** (0.0388)
i(var=IMP_METHOD_REF,ref=TRUE)RQ	3.869*** (0.0344)	3.934*** (0.0350)
i(var=IMP_METHOD_REF,ref=TRUE)PMM	4.034*** (0.0394)	4.100*** (0.0396)
N	0.5177*** (0.0086)	14.76*** (0.9039)
Percent_Missing	-3.724*** (0.0153)	-6.432*** (0.8168)
MISS_TYPESTATUS_w_DEMOG	-4.897*** (0.4057)	-20.08*** (1.467)
MISS_TYPESTATUS_w_GROWTH	-7.749*** (0.4072)	-22.05*** (1.462)
N x Percent_Missing		-0.4676*** (0.0285)
N x MISS_TYPESTATUS_w_DEMOG		-13.14*** (0.9040)
N x MISS_TYPESTATUS_w_GROWTH		-13.80*** (0.9005)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		3.088*** (0.8206)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		2.355** (0.8175)
Fixed-Effects:	——————	——————-
AMP_N	Yes	Yes
________________________________________	__________________	___________________
S.E.: Clustered	by: AMP_N	by: AMP_N
Convergence	TRUE	TRUE
AIC	78,887.4	78,208.9
BIC	79,577.5	78,956.5

\(~\)

Figure 4.7: Coefficient plot, additive model for scale score percent bias classification at the school level

Figure 4.8: Fixed effects estimates, additive model

Figure 4.9: Fixed effects estimates, interaction model

4.2 SGPs

4.2.1 Grade and Content Area Level

## NOTE: 4,102 observations removed because of NA values (LHS: 4,102).
## NOTE: 4,102 observations removed because of NA values (LHS: 4,102).

Table 4.4: Fixed-effect regression models for SGP percent bias classification at the grade/content area level, grades 5 through 8

	Additive Model	Interaction Model
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER_LONG	-0.6953*** (0.0862)	-0.6959*** (0.0861)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG	-0.5423*** (0.0978)	-0.5427*** (0.0977)
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER	-0.7452*** (0.0850)	-0.7459*** (0.0850)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN	0.0793** (0.0257)	0.0790** (0.0256)
i(var=IMP_METHOD_REF,ref=TRUE)RQ	-0.3544*** (0.0969)	-0.3546*** (0.0969)
i(var=IMP_METHOD_REF,ref=TRUE)PMM	-0.3429*** (0.1008)	-0.3432*** (0.1008)
N	0.3655*** (0.0732)	0.3634*** (0.0774)
Percent_Missing	-0.6978*** (0.0202)	-0.6627*** (0.0181)
MISS_TYPESTATUS_w_DEMOG	0.0437* (0.0189)	0.0365. (0.0198)
MISS_TYPESTATUS_w_GROWTH	0.0674** (0.0241)	0.0644* (0.0255)
N x Percent_Missing		-0.0754** (0.0255)
N x MISS_TYPESTATUS_w_DEMOG		0.0038 (0.0385)
N x MISS_TYPESTATUS_w_GROWTH		-0.0237 (0.0359)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		-0.0384 (0.0257)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		-0.0318 (0.0238)
Fixed-Effects:	——————-	——————-
CONTENT_AREA^GRADE	Yes	Yes
AMP_N	Yes	Yes
________________________________________	___________________	___________________
S.E.: Clustered	by: CON.^GRA.	by: CON.^GRA.
Convergence	TRUE	TRUE
AIC	3,090,115.7	3,088,938.8
BIC	3,090,971.1	3,089,858.0

\(~\)

Figure 4.10: Coefficient plot, additive model for SGP percent bias classification at the grade/content area level, grades 5 through 8

Figure 4.11: Fixed effects estimates, additive model

Figure 4.12: Fixed effects estimates, interaction model

4.2.2 School Level

## NOTE: 323 observations removed because of NA values (LHS: 323).
## NOTE: 323 observations removed because of NA values (LHS: 323).

Table 4.5: Fixed-effect regression models for SGP percent bias classification at the school level

	Additive Model	Interaction Model
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER_LONG	-0.9346*** (0.0063)	-0.9348*** (0.0063)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG	-0.7821*** (0.0063)	-0.7820*** (0.0063)
i(var=IMP_METHOD_REF,ref=TRUE)L2LMER	-0.9967*** (0.0061)	-0.9971*** (0.0061)
i(var=IMP_METHOD_REF,ref=TRUE)L2PAN	0.0842*** (0.0054)	0.0836*** (0.0054)
i(var=IMP_METHOD_REF,ref=TRUE)RQ	-0.5846*** (0.0061)	-0.5843*** (0.0061)
i(var=IMP_METHOD_REF,ref=TRUE)PMM	-0.5378*** (0.0063)	-0.5375*** (0.0063)
N	0.4745*** (0.0029)	0.4571*** (0.0063)
Percent_Missing	-0.6821*** (0.0025)	-0.6919*** (0.0070)
MISS_TYPESTATUS_w_DEMOG	-0.0233*** (0.0058)	-0.0135* (0.0057)
MISS_TYPESTATUS_w_GROWTH	0.0409*** (0.0072)	0.0446*** (0.0070)
N x Percent_Missing		-0.0909*** (0.0033)
N x MISS_TYPESTATUS_w_DEMOG		0.0648*** (0.0087)
N x MISS_TYPESTATUS_w_GROWTH		0.0215* (0.0089)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		-0.0091 (0.0078)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		0.0007 (0.0088)
Fixed-Effects:	——————-	——————-
AMP_N	Yes	Yes
________________________________________	___________________	___________________
S.E.: Clustered	by: AMP_N	by: AMP_N
Convergence	TRUE	TRUE
AIC	895,408.7	894,711.1
BIC	896,098.7	895,458.7

\(~\)

Figure 4.13: Coefficient plot, additive model for SGP percent bias classification at the school level

Figure 4.14: Fixed effects estimates, additive model

Figure 4.15: Fixed effects estimates, interaction model

5 Cross-Sectional L2PAN

For the final set of models, we examine factors influencing absolute scale score and SGP bias only among data imputed using cross-sectional L2PAN.

5.1 Scale Scores

5.1.1 Grade and Content Area Level

The first set of models includes all grades (i.e., grade 3 through 8).

Table 5.1: L2PAN: Fixed-effect regression models for scale score absolute bias at the grade/content area level, all grade levels

	Additive Model	Interaction Model
N	-1.041*** (0.1368)	-0.7737*** (0.0909)
I(I(N^2))	0.1684*** (0.0350)
Percent_Missing	3.082*** (0.6097)	1.443*** (0.1426)
MISS_TYPESTATUS_w_DEMOG	1.273** (0.3300)	1.281** (0.3254)
MISS_TYPESTATUS_w_GROWTH	4.647** (1.445)	4.588** (1.425)
N x Percent_Missing		-0.4557** (0.1299)
N x MISS_TYPESTATUS_w_DEMOG		0.0743 (0.1252)
N x MISS_TYPESTATUS_w_GROWTH		0.2740 (0.2747)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		0.9912** (0.2485)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		3.202** (0.9712)
Fixed-Effects:	——————	——————-
CONTENT_AREA^GRADE	Yes	Yes
AMP_N	Yes	Yes
________________________________________	__________________	___________________
S.E.: Clustered	by: CON.^GRA.	by: CON.^GRA.
R2	0.44032	0.47492
Within Adj. R2	0.34614	0.38656
AIC	4,190,263.3	4,146,477.5
BIC	4,191,018.3	4,147,278.3

\(~\)

Figure 5.1: Coefficient plot, L2PAN additive models for scale score absolute bias at the grade/content area level

Figure 5.2: Fixed effects estimates, additive model

Figure 5.3: Fixed effects estimates, interaction model

We can also subset the data to only observations from grades 5 through 8.

Table 5.2: L2PAN: Fixed-effect regression models for scale score absolute bias at the grade/content area level, grades 5 through 8

	Additive Model	Interaction Model
N	-1.208*** (0.1316)	-0.7080*** (0.0745)
I(I(N^2))	0.1852** (0.0424)
Percent_Missing	1.373*** (0.0888)	1.143*** (0.0951)
MISS_TYPESTATUS_w_DEMOG	0.3916*** (0.0517)	0.3967*** (0.0519)
MISS_TYPESTATUS_w_GROWTH	0.6270*** (0.0614)	0.6106*** (0.0577)
N x Percent_Missing		-0.4077** (0.0758)
N x MISS_TYPESTATUS_w_DEMOG		-0.1194 (0.0658)
N x MISS_TYPESTATUS_w_GROWTH		-0.0494 (0.0689)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		0.2466** (0.0549)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		0.4124*** (0.0546)
Fixed-Effects:	——————	——————-
CONTENT_AREA^GRADE	Yes	Yes
AMP_N	Yes	Yes
________________________________________	__________________	___________________
S.E.: Clustered	by: CON.^GRA.	by: CON.^GRA.
R2	0.26537	0.26314
Within Adj. R2	0.25580	0.25354
AIC	1,770,244.4	1,771,373.4
BIC	1,770,915.3	1,772,087.7

\(~\)

Figure 5.4: Coefficient plot, L2PAN additive models for scale score absolute bias at the grade/content area level, grades 5 through 8

Figure 5.5: Fixed effects estimates, additive model, grades 5 through 8

Figure 5.6: Fixed effects estimates, interaction model, grades 5 through 8

5.1.2 School Level

Table 5.3: L2PAN: Fixed-effect regression models for scale score absolute bias at the school level

	Additive Model	Interaction Model
N	-1.343*** (0.0078)	-0.4207*** (0.0043)
I(I(N^2))	0.1267*** (0.0010)
Percent_Missing	2.108*** (0.0085)	0.5998*** (0.0097)
MISS_TYPESTATUS_w_DEMOG	1.629*** (0.0149)	1.640*** (0.0149)
MISS_TYPESTATUS_w_GROWTH	5.159*** (0.0162)	5.127*** (0.0160)
N x Percent_Missing		-0.6677*** (0.0049)
N x MISS_TYPESTATUS_w_DEMOG		-0.5042*** (0.0091)
N x MISS_TYPESTATUS_w_GROWTH		-1.299*** (0.0077)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		0.8419*** (0.0167)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		2.856*** (0.0168)
Fixed-Effects:	——————	——————-
AMP_N	Yes	Yes
________________________________________	__________________	___________________
S.E.: Clustered	by: AMP_N	by: AMP_N
R2	0.48319	0.56577
Within Adj. R2	0.48307	0.56566
AIC	545,903.5	527,734.9
BIC	546,429.1	528,298.7

\(~\)

Figure 5.7: Coefficient plot, L2PAN additive models for scale score absolute bias at the school level

Figure 5.8: Fixed effects estimates, additive model

Figure 5.9: Fixed effects estimates, interaction model

5.2 SGPs

5.2.1 Grade and Content Area Level

Table 5.4: L2PAN: Fixed-effect regression models for SGP absolute bias at the grade/content area level, grades 5 through 8

	Additive Model	Interaction Model
N	-1.752*** (0.2144)	-0.9905*** (0.0786)
I(I(N^2))	0.2569** (0.0589)
Percent_Missing	1.780*** (0.1045)	1.575*** (0.0961)
MISS_TYPESTATUS_w_DEMOG	0.6175*** (0.0690)	0.6318*** (0.0700)
MISS_TYPESTATUS_w_GROWTH	0.7087*** (0.0710)	0.7036*** (0.0637)
N x Percent_Missing		-0.5713** (0.1160)
N x MISS_TYPESTATUS_w_DEMOG		-0.2499* (0.0999)
N x MISS_TYPESTATUS_w_GROWTH		-0.2024. (0.0986)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		0.2695* (0.1028)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		0.3508** (0.0895)
Fixed-Effects:	——————	——————-
CONTENT_AREA^GRADE	Yes	Yes
AMP_N	Yes	Yes
________________________________________	__________________	___________________
S.E.: Clustered	by: CON.^GRA.	by: CON.^GRA.
R2	0.25911	0.25528
Within Adj. R2	0.25361	0.24974
AIC	1,981,971.1	1,983,891.7
BIC	1,982,642.1	1,984,605.9

\(~\)

Figure 5.10: Coefficient plot, L2PAN additive models for SGP absolute bias at the grade/content area level, grades 5 through 8

Figure 5.11: Fixed effects estimates, additive model

Figure 5.12: Fixed effects estimates, interaction model

5.2.2 School Level

Table 5.5: L2PAN: Fixed-effect regression models for SGP absolute bias at the school level

	Additive Model	Interaction Model
N	-1.227*** (0.0077)	-0.7295*** (0.0068)
I(I(N^2))	0.1352*** (0.0011)
Percent_Missing	1.155*** (0.0071)	0.9759*** (0.0144)
MISS_TYPESTATUS_w_DEMOG	0.6157*** (0.0146)	0.6131*** (0.0147)
MISS_TYPESTATUS_w_GROWTH	0.5616*** (0.0135)	0.5486*** (0.0135)
N x Percent_Missing		-0.4233*** (0.0051)
N x MISS_TYPESTATUS_w_DEMOG		-0.2233*** (0.0107)
N x MISS_TYPESTATUS_w_GROWTH		-0.1150*** (0.0104)
Percent_Missing x MISS_TYPESTATUS_w_DEMOG		0.1822*** (0.0155)
Percent_Missing x MISS_TYPESTATUS_w_GROWTH		0.1208*** (0.0163)
Fixed-Effects:	——————	——————-
AMP_N	Yes	Yes
________________________________________	__________________	___________________
S.E.: Clustered	by: AMP_N	by: AMP_N
R2	0.25154	0.24458
Within Adj. R2	0.25135	0.24435
AIC	492,190.6	493,165.7
BIC	492,716.2	493,729.5

\(~\)

Figure 5.13: Coefficient plot, L2PAN additive models for SGP absolute bias at the school level

Figure 5.14: Fixed effects estimates, additive model

Figure 5.15: Fixed effects estimates, interaction model

5.3 Random Effects Models

We introduce a random intercept for school number to account for the hierarchical nature of the data.

## Linear mixed model fit by REML ['lmerMod']
## Formula: abs(SS_Raw_Bias) ~ N + Percent_Missing + MISS_TYPE + GRADE +  
##     CONTENT_AREA + (1 | SCHOOL_NUMBER)
##    Data: l2pan.GC.Eval
## 
## REML criterion at convergence: 79957.5
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -3.0141 -0.6006 -0.1106  0.4787 12.1002 
## 
## Random effects:
##  Groups        Name        Variance Std.Dev.
##  SCHOOL_NUMBER (Intercept)  1.622   1.274   
##  Residual                  19.232   4.385   
## Number of obs: 13725, groups:  SCHOOL_NUMBER, 231
## 
## Fixed effects:
##                          Estimate Std. Error t value
## (Intercept)               4.40647    0.13526  32.577
## N                        -0.37465    0.06686  -5.604
## Percent_Missing           2.68961    0.04180  64.342
## MISS_TYPESTATUS_w_DEMOG   2.04493    0.09169  22.301
## MISS_TYPESTATUS_w_GROWTH  5.87448    0.09171  64.056
## GRADE4                   -0.30414    0.11115  -2.736
## GRADE5                   -4.94715    0.11252 -43.967
## GRADE6                   -5.21227    0.13222 -39.420
## GRADE7                   -5.51638    0.19175 -28.769
## GRADE8                   -5.39903    0.19711 -27.391
## CONTENT_AREAMATHEMATICS   0.20905    0.07487   2.792
## 
## Correlation of Fixed Effects:
##                    (Intr) N      Prcn_M MISS_TYPESTATUS__D MISS_TYPESTATUS__G
## N                  -0.052                                                    
## Prcnt_Mssng         0.004  0.008                                             
## MISS_TYPESTATUS__D -0.339  0.000 -0.008                                      
## MISS_TYPESTATUS__G -0.339  0.000 -0.019  0.500                               
## GRADE4             -0.413 -0.003 -0.003  0.000              0.000            
## GRADE5             -0.405  0.018 -0.001  0.000              0.000            
## GRADE6             -0.404  0.005  0.007  0.000              0.000            
## GRADE7             -0.369  0.112  0.015  0.000              0.000            
## GRADE8             -0.363  0.130  0.009  0.000              0.000            
## CONTENT_ARE        -0.277  0.000  0.000  0.000              0.000            
##                    GRADE4 GRADE5 GRADE6 GRADE7 GRADE8
## N                                                    
## Prcnt_Mssng                                          
## MISS_TYPESTATUS__D                                   
## MISS_TYPESTATUS__G                                   
## GRADE4                                               
## GRADE5              0.496                            
## GRADE6              0.424  0.419                     
## GRADE7              0.294  0.292  0.409              
## GRADE8              0.284  0.284  0.403  0.528       
## CONTENT_ARE         0.000  0.001  0.000  0.000  0.000

## Linear mixed model fit by REML ['lmerMod']
## Formula: abs(SGP_Raw_Bias) ~ N + Percent_Missing + MISS_TYPE + GRADE +  
##     CONTENT_AREA + (1 | SCHOOL_NUMBER)
##    Data: l2pan.GC.Eval
## 
## REML criterion at convergence: 35173.7
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -3.2323 -0.5735 -0.0542  0.4228  8.6537 
## 
## Random effects:
##  Groups        Name        Variance Std.Dev.
##  SCHOOL_NUMBER (Intercept) 1.439    1.200   
##  Residual                  6.332    2.516   
## Number of obs: 7407, groups:  SCHOOL_NUMBER, 223
## 
## Fixed effects:
##                          Estimate Std. Error t value
## (Intercept)               2.19733    0.10855  20.243
## N                        -0.58897    0.06916  -8.516
## Percent_Missing           1.39706    0.03298  42.358
## MISS_TYPESTATUS_w_DEMOG   0.98556    0.07163  13.758
## MISS_TYPESTATUS_w_GROWTH  1.27717    0.07165  17.824
## GRADE6                    0.27095    0.08332   3.252
## GRADE7                   -0.17876    0.12897  -1.386
## GRADE8                   -0.11115    0.13255  -0.839
## CONTENT_AREAMATHEMATICS   0.09786    0.05849   1.673
## 
## Correlation of Fixed Effects:
##                    (Intr) N      Prcn_M MISS_TYPESTATUS__D MISS_TYPESTATUS__G
## N                  -0.103                                                    
## Prcnt_Mssng         0.008  0.013                                             
## MISS_TYPESTATUS__D -0.330  0.000 -0.021                                      
## MISS_TYPESTATUS__G -0.330  0.000 -0.031  0.500                               
## GRADE6             -0.317  0.013  0.014  0.000              0.000            
## GRADE7             -0.317  0.276  0.031 -0.001             -0.001            
## GRADE8             -0.313  0.304  0.022  0.000             -0.001            
## CONTENT_ARE        -0.268  0.002  0.000  0.000              0.000            
##                    GRADE6 GRADE7 GRADE8
## N                                      
## Prcnt_Mssng                            
## MISS_TYPESTATUS__D                     
## MISS_TYPESTATUS__G                     
## GRADE6                                 
## GRADE7              0.457              
## GRADE8              0.450  0.651       
## CONTENT_ARE        -0.002 -0.001  0.000

6 References and Helpful R Resources

• Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1-48. https://doi.org/10.18637/jss.v067.i01.
  
• Berge, L. (2018). Efficient estimation of maximum likelihood models with multiple fixed-effects: the R package FENmlm. CREA Discussion Papers.
• Berge, L. (2021, March 29). Fast fixed-effects estimation: Short introduction. Retrieved from https://cran.r-project.org/web/packages/fixest/vignettes/multiple_estimations.html
• DataMentor R Plot Tutorial
• Miri, H. H., Hassanzadeh, J., Khaniki, S. H., Akrami, R., & Sirjani, E. (2020). Accuracy of five multiple imputation methods in estimating prevalence of Type 2 diabetes based on STEPS surveys. Journal of Epidemiology and Global Health, 10(1), 36-41. https://doi.org/10.2991/jegh.k.191207.001
  
• Qi, L., Wang, Y.-F., & He, Y. (2010). A comparison of multiple imputation and fully augmented weighted estimators for Cox regression with missing covariates. Statistics in Medicine, 29(25), 2592-2604. https://doi.org/10.1002/sim.4016