Comparing Multiple Imputation with Basic Generalized Linear Models when Incorporating a Simulated COVID-19 Impact
Allie Cooperman, Adam Van Iwaarden, and Damian Betebenner
June 17, 2021
1 Vignette Overview
This vignette extends the multiple imputation (MI) summary vignette, “Comparing Multiple Imputation Methods when Simulating an Academic Impact of COVID-19.” Here, we run a series of fixed-effects regression models to better understand the relationships among different indices of MI efficacy and both data and school characteristics (e.g., missingness type, school size, etc.). For a more detailed recap of the MI simulation, please see the summary vignette. In this simulation, a COVID-19 impact was incorporated into the synthetic data.
Our primary research questions can be stated as follows:
- Is MI an appropriate method for creating “adjusted” scale scores and SGPs when data are missing?
- If MI is indeed appropriate, which method is most effective and in what data contexts?
In this vignette, we fit linear or logistic fixed-effect regression models. Specifically, we use the fixest function (Berge, 2018) for fixed effects models. Fixed effects for grade and content area are included when appropriate. We also include fixed effects for the amputation number (AMP_N).
2 Raw Bias
In the following models, the dependent variable is the raw bias for either the scale scores or student growth percentiles (SPGs). Raw bias is calculated as the average difference between the true and imputed values.
2.1 Scale Scores
2.1.1 Grade and Content Area Level
## NOTE: 11,300 observations removed because of NA values (LHS: 11,300).
## NOTE: 11,300 observations removed because of NA values (LHS: 11,300).| Additive Model | Interaction Model | |
|---|---|---|
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG | -4.211*** (0.7999) | -4.230*** (0.7950) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN | -4.731*** (0.9200) | -4.751*** (0.9151) |
| i(var=IMP_METHOD_REF,ref=TRUE)RQ | -4.782*** (0.9357) | -4.802*** (0.9308) |
| i(var=IMP_METHOD_REF,ref=TRUE)RF | -4.168*** (0.8905) | -4.187*** (0.8855) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PMM | -3.833*** (0.6511) | -3.853*** (0.6460) |
| i(var=IMP_METHOD_REF,ref=TRUE)PMM | -4.878*** (0.9830) | -4.898*** (0.9781) |
| N | 0.0598 (0.0835) | 0.2640** (0.0795) |
| Percent_Missing | 2.817*** (0.5010) | -0.0825 (0.0572) |
| MISS_TYPESTATUS_w_DEMOG | 3.711*** (0.5871) | 3.782*** (0.5861) |
| MISS_TYPESTATUS_w_GROWTH | 7.742*** (1.497) | 7.789*** (1.492) |
| N x Percent_Missing | -0.1359 (0.1548) | |
| N x MISS_TYPESTATUS_w_DEMOG | -0.2174 (0.1345) | |
| N x MISS_TYPESTATUS_w_GROWTH | -0.3001 (0.2234) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 2.312*** (0.3605) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 4.889*** (0.8985) | |
| Fixed-Effects: | —————— | —————— |
| CONTENT_AREA^GRADE | Yes | Yes |
| AMP_N | Yes | Yes |
| ________________________________________ | __________________ | __________________ |
| S.E.: Clustered | by: CON.^GRA. | by: CON.^GRA. |
| R2 | 0.32943 | 0.37060 |
| Within Adj. R2 | 0.28489 | 0.32879 |
| AIC | 32,859,670.1 | 32,552,651.9 |
| BIC | 32,860,621.1 | 32,553,669.8 |
\(~\)
Figure 2.1: Coefficient plot, additive models for scale score raw bias at the grade/content area level
Figure 2.2: Fixed effects estimates, additive model
Figure 2.3: Fixed effects estimates, interaction model
We can also subset the data to only observations from grades 5 through 8.
## NOTE: 4,460 observations removed because of NA values (LHS: 4,460).
## NOTE: 4,460 observations removed because of NA values (LHS: 4,460).| Additive Model | Interaction Model | |
|---|---|---|
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG | -6.383*** (0.7735) | -6.389*** (0.7720) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN | -7.345*** (0.2733) | -7.351*** (0.2720) |
| i(var=IMP_METHOD_REF,ref=TRUE)RQ | -7.435*** (0.2411) | -7.441*** (0.2399) |
| i(var=IMP_METHOD_REF,ref=TRUE)RF | -6.708*** (0.2410) | -6.714*** (0.2396) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PMM | -5.681*** (0.2664) | -5.687*** (0.2649) |
| i(var=IMP_METHOD_REF,ref=TRUE)PMM | -7.660*** (0.2225) | -7.666*** (0.2213) |
| N | 0.2181** (0.0601) | 0.2304** (0.0551) |
| Percent_Missing | 1.432*** (0.1402) | -0.0328 (0.0496) |
| MISS_TYPESTATUS_w_DEMOG | 2.086*** (0.1610) | 2.130*** (0.1651) |
| MISS_TYPESTATUS_w_GROWTH | 3.536*** (0.3705) | 3.560*** (0.3719) |
| N x Percent_Missing | 0.1932* (0.0651) | |
| N x MISS_TYPESTATUS_w_DEMOG | 0.1033 (0.0705) | |
| N x MISS_TYPESTATUS_w_GROWTH | 0.0967 (0.0824) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 1.281*** (0.0837) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 2.352*** (0.2630) | |
| Fixed-Effects: | —————— | —————— |
| CONTENT_AREA^GRADE | Yes | Yes |
| AMP_N | Yes | Yes |
| ________________________________________ | __________________ | __________________ |
| S.E.: Clustered | by: CON.^GRA. | by: CON.^GRA. |
| R2 | 0.20535 | 0.21959 |
| Within Adj. R2 | 0.20221 | 0.21650 |
| AIC | 17,164,361.6 | 17,116,941.2 |
| BIC | 17,165,217.8 | 17,117,861.4 |
\(~\)
Figure 2.4: Coefficient plot, additive models for scale score raw bias at the grade/content area level, grades 5 through 8
Figure 2.5: Fixed effects estimates, additive model, grades 5 through 8
Figure 2.6: Fixed effects estimates, interaction model, grades 5 through 8
2.1.2 School Level
## NOTE: 26 observations removed because of NA values (LHS: 26).
## NOTE: 26 observations removed because of NA values (LHS: 26).| Additive Model | Interaction Model | |
|---|---|---|
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG | -4.773*** (0.0082) | -4.773*** (0.0082) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN | -5.254*** (0.0093) | -5.254*** (0.0093) |
| i(var=IMP_METHOD_REF,ref=TRUE)RQ | -5.332*** (0.0099) | -5.332*** (0.0099) |
| i(var=IMP_METHOD_REF,ref=TRUE)RF | -4.701*** (0.0096) | -4.701*** (0.0096) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PMM | -4.367*** (0.0121) | -4.367*** (0.0121) |
| i(var=IMP_METHOD_REF,ref=TRUE)PMM | -5.423*** (0.0105) | -5.423*** (0.0105) |
| N | -0.5926*** (0.0048) | 0.1078*** (0.0055) |
| Percent_Missing | 2.480*** (0.0107) | -0.0128 (0.0134) |
| MISS_TYPESTATUS_w_DEMOG | 3.741*** (0.0152) | 3.781*** (0.0150) |
| MISS_TYPESTATUS_w_GROWTH | 7.806*** (0.0217) | 7.805*** (0.0212) |
| N x Percent_Missing | -0.4824*** (0.0075) | |
| N x MISS_TYPESTATUS_w_DEMOG | -0.7213*** (0.0106) | |
| N x MISS_TYPESTATUS_w_GROWTH | -1.610*** (0.0101) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 1.932*** (0.0140) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 4.276*** (0.0196) | |
| Fixed-Effects: | ——————- | ——————- |
| AMP_N | Yes | Yes |
| ________________________________________ | ___________________ | ___________________ |
| S.E.: Clustered | by: AMP_N | by: AMP_N |
| R2 | 0.53237 | 0.61398 |
| Within Adj. R2 | 0.53228 | 0.61391 |
| AIC | 4,174,395.7 | 4,034,251.8 |
| BIC | 4,175,085.9 | 4,034,999.4 |
\(~\)
Figure 2.7: Coefficient plot, additive models for scale score raw bias at the school level
Figure 2.8: Fixed effects estimates, additive model
Figure 2.9: Fixed effects estimates, interaction model
2.2 SGPs
2.2.1 Grade and Content Area Level
## NOTE: 4,460 observations removed because of NA values (LHS: 4,460).
## NOTE: 4,460 observations removed because of NA values (LHS: 4,460).| Additive Model | Interaction Model | |
|---|---|---|
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG | -0.0047 (0.2384) | -0.0037 (0.2385) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN | -0.1062 (0.1935) | -0.1053 (0.1937) |
| i(var=IMP_METHOD_REF,ref=TRUE)RQ | -0.3006 (0.2453) | -0.2997 (0.2454) |
| i(var=IMP_METHOD_REF,ref=TRUE)RF | -0.2234 (0.1960) | -0.2224 (0.1961) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PMM | -0.4355. (0.2193) | -0.4345. (0.2194) |
| i(var=IMP_METHOD_REF,ref=TRUE)PMM | -0.4186 (0.2476) | -0.4177 (0.2476) |
| N | 0.4248*** (0.0634) | 0.4516*** (0.0550) |
| Percent_Missing | 0.2849** (0.0759) | -0.1453** (0.0384) |
| MISS_TYPESTATUS_w_DEMOG | -0.0502 (0.0610) | -0.0386 (0.0730) |
| MISS_TYPESTATUS_w_GROWTH | 0.0228 (0.0826) | 0.0495 (0.0958) |
| N x Percent_Missing | 0.3217** (0.0639) | |
| N x MISS_TYPESTATUS_w_DEMOG | 0.1192 (0.0812) | |
| N x MISS_TYPESTATUS_w_GROWTH | 0.0421 (0.0911) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 0.4724*** (0.0799) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 0.5323** (0.1001) | |
| Fixed-Effects: | —————— | —————— |
| CONTENT_AREA^GRADE | Yes | Yes |
| AMP_N | Yes | Yes |
| ________________________________________ | __________________ | __________________ |
| S.E.: Clustered | by: CON.^GRA. | by: CON.^GRA. |
| R2 | 0.00489 | 0.00678 |
| Within Adj. R2 | 0.00418 | 0.00608 |
| AIC | 17,491,972.0 | 17,486,977.8 |
| BIC | 17,492,828.3 | 17,487,898.0 |
\(~\)
Figure 2.10: Coefficient plot, additive models for SGP raw bias at the grade/content area level, grades 5 through 8
Figure 2.11: Fixed effects estimates, additive model
Figure 2.12: Fixed effects estimates, interaction model
2.2.2 School Level
## NOTE: 299 observations removed because of NA values (LHS: 299).
## NOTE: 299 observations removed because of NA values (LHS: 299).| Additive Model | Interaction Model | |
|---|---|---|
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG | -0.0996*** (0.0100) | -0.0985*** (0.0100) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN | -0.1757*** (0.0085) | -0.1746*** (0.0085) |
| i(var=IMP_METHOD_REF,ref=TRUE)RQ | -0.5214*** (0.0089) | -0.5203*** (0.0089) |
| i(var=IMP_METHOD_REF,ref=TRUE)RF | -0.3249*** (0.0091) | -0.3238*** (0.0091) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PMM | -0.5987*** (0.0092) | -0.5976*** (0.0092) |
| i(var=IMP_METHOD_REF,ref=TRUE)PMM | -0.6331*** (0.0095) | -0.6320*** (0.0095) |
| N | 0.2648*** (0.0043) | 0.2936*** (0.0057) |
| Percent_Missing | 0.2152*** (0.0092) | -0.1814*** (0.0112) |
| MISS_TYPESTATUS_w_DEMOG | -0.0963*** (0.0085) | -0.0791*** (0.0083) |
| MISS_TYPESTATUS_w_GROWTH | 0.0618*** (0.0074) | 0.0779*** (0.0073) |
| N x Percent_Missing | 0.2754*** (0.0077) | |
| N x MISS_TYPESTATUS_w_DEMOG | 0.1023*** (0.0104) | |
| N x MISS_TYPESTATUS_w_GROWTH | -0.0119 (0.0089) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 0.4013*** (0.0171) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 0.6431*** (0.0158) | |
| Fixed-Effects: | ——————- | ——————- |
| AMP_N | Yes | Yes |
| ________________________________________ | ___________________ | ___________________ |
| S.E.: Clustered | by: AMP_N | by: AMP_N |
| R2 | 0.00674 | 0.01127 |
| Within Adj. R2 | 0.00669 | 0.01122 |
| AIC | 4,442,686.2 | 4,439,355.6 |
| BIC | 4,443,376.3 | 4,440,103.2 |
\(~\)
Figure 2.13: Coefficient plot, additive models for SGP raw bias at the school level
Figure 2.14: Fixed effects estimates, additive model
Figure 2.15: Fixed effects estimates, interaction model
2.3 Random Effects Models
We introduce a random intercept for school number to account for the hierarchical nature of the data. We begin with a relatively simple model using only additive fixed effects. The mixed-effects models are run with the lme4 package (Bates et al., 2015).
## Linear mixed model fit by REML ['lmerMod']
## Formula: SS_Raw_Bias ~ IMP_METHOD_REF + N + Percent_Missing + MISS_TYPE +
## GRADE + CONTENT_AREA + (1 | SCHOOL_NUMBER)
## Data: MIsummary.GC.Eval
##
## REML criterion at convergence: 605454.9
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -6.9722 -0.5481 -0.0391 0.4934 12.1680
##
## Random effects:
## Groups Name Variance Std.Dev.
## SCHOOL_NUMBER (Intercept) 5.344 2.312
## Residual 29.481 5.430
## Number of obs: 97146, groups: SCHOOL_NUMBER, 232
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 6.75710 0.16562 40.798
## IMP_METHOD_REFL2PAN_LONG -4.26681 0.06518 -65.461
## IMP_METHOD_REFL2PAN -4.78725 0.06518 -73.446
## IMP_METHOD_REFRQ -4.83797 0.06518 -74.224
## IMP_METHOD_REFRF -4.22370 0.06518 -64.800
## IMP_METHOD_REFL2PMM -3.88882 0.06518 -59.662
## IMP_METHOD_REFPMM -4.93396 0.06518 -75.696
## N -0.12241 0.04250 -2.880
## Percent_Missing 2.53218 0.01993 127.061
## MISS_TYPESTATUS_w_DEMOG 3.72206 0.04267 87.225
## MISS_TYPESTATUS_w_GROWTH 7.76963 0.04268 182.041
## GRADE4 -0.37738 0.05179 -7.286
## GRADE5 -3.92084 0.05207 -75.295
## GRADE6 -5.22147 0.06384 -81.792
## GRADE7 -6.33051 0.10345 -61.195
## GRADE8 -6.43708 0.10346 -62.218
## CONTENT_AREAMATHEMATICS 0.08046 0.03484 2.309##
## Correlation matrix not shown by default, as p = 17 > 12.
## Use print(x, correlation=TRUE) or
## vcov(x) if you need it## Linear mixed model fit by REML ['lmerMod']
## Formula: SGP_Raw_Bias ~ IMP_METHOD_REF + N + Percent_Missing + MISS_TYPE +
## GRADE + CONTENT_AREA + (1 | SCHOOL_NUMBER)
## Data: MIsummary.GC.Eval
##
## REML criterion at convergence: 313236.8
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -9.3759 -0.4987 0.0036 0.4952 8.3462
##
## Random effects:
## Groups Name Variance Std.Dev.
## SCHOOL_NUMBER (Intercept) 11.65 3.413
## Residual 22.26 4.718
## Number of obs: 52542, groups: SCHOOL_NUMBER, 227
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 0.499190 0.238216 2.096
## IMP_METHOD_REFL2PAN_LONG 0.002207 0.077011 0.029
## IMP_METHOD_REFL2PAN -0.099370 0.077011 -1.290
## IMP_METHOD_REFRQ -0.293745 0.077011 -3.814
## IMP_METHOD_REFRF -0.216519 0.077011 -2.812
## IMP_METHOD_REFL2PMM -0.428608 0.077011 -5.566
## IMP_METHOD_REFPMM -0.411747 0.077011 -5.347
## N 0.196208 0.081309 2.413
## Percent_Missing 0.049037 0.023734 2.066
## MISS_TYPESTATUS_w_DEMOG -0.041924 0.050423 -0.831
## MISS_TYPESTATUS_w_GROWTH 0.036095 0.050445 0.716
## GRADE6 -0.759336 0.062262 -12.196
## GRADE7 -2.339532 0.111801 -20.926
## GRADE8 -2.432753 0.112073 -21.707
## CONTENT_AREAMATHEMATICS 0.049679 0.041164 1.207##
## Correlation matrix not shown by default, as p = 15 > 12.
## Use print(x, correlation=TRUE) or
## vcov(x) if you need it3 Absolute Bias
Here, the dependent variable is absolute bias, computed by taking the absolute value of the raw bias (for either the scale scores or SGPs). In the additive models, a quadratic term for grade/content area or school size (N) is added based on previous figures indicating a curved relationship between bias and N.
3.1 Scale Scores
3.1.1 Grade and Content Area Level
The first set of models includes all grades (i.e., grade 3 through 8).
## NOTE: 11,300 observations removed because of NA values (LHS: 11,300).
## NOTE: 11,300 observations removed because of NA values (LHS: 11,300).| Additive Model | Interaction Model | |
|---|---|---|
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG | -3.480*** (0.4123) | -3.494*** (0.4090) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN | -4.614*** (0.7488) | -4.629*** (0.7458) |
| i(var=IMP_METHOD_REF,ref=TRUE)RQ | -4.300*** (0.6550) | -4.315*** (0.6520) |
| i(var=IMP_METHOD_REF,ref=TRUE)RF | -4.384*** (0.7765) | -4.399*** (0.7735) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PMM | -3.853*** (0.5425) | -3.868*** (0.5394) |
| i(var=IMP_METHOD_REF,ref=TRUE)PMM | -4.291*** (0.6693) | -4.306*** (0.6663) |
| N | -0.9817*** (0.1089) | -0.8172*** (0.0870) |
| I(I(N^2)) | 0.1685*** (0.0286) | |
| Percent_Missing | 3.587*** (0.4346) | 1.802*** (0.1058) |
| MISS_TYPESTATUS_w_DEMOG | 1.829*** (0.2527) | 1.834*** (0.2519) |
| MISS_TYPESTATUS_w_GROWTH | 5.310*** (1.108) | 5.260*** (1.104) |
| N x Percent_Missing | -0.4978** (0.1179) | |
| N x MISS_TYPESTATUS_w_DEMOG | 0.1187 (0.0731) | |
| N x MISS_TYPESTATUS_w_GROWTH | 0.3223. (0.1741) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 1.168*** (0.1815) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 3.342*** (0.7368) | |
| Fixed-Effects: | ——————- | ——————- |
| CONTENT_AREA^GRADE | Yes | Yes |
| AMP_N | Yes | Yes |
| ________________________________________ | ___________________ | ___________________ |
| S.E.: Clustered | by: CON.^GRA. | by: CON.^GRA. |
| R2 | 0.41474 | 0.44544 |
| Within Adj. R2 | 0.38271 | 0.41510 |
| AIC | 30,660,076.0 | 30,398,910.0 |
| BIC | 30,661,040.3 | 30,399,927.9 |
\(~\)
Figure 3.1: Coefficient plot, additive models for scale score absolute bias at the grade/content area level
Figure 3.2: Fixed effects estimates, additive model
Figure 3.3: Fixed effects estimates, interaction model
We can also subset the data to only observations from grades 5 through 8.
## NOTE: 4,460 observations removed because of NA values (LHS: 4,460).
## NOTE: 4,460 observations removed because of NA values (LHS: 4,460).| Additive Model | Interaction Model | |
|---|---|---|
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG | -4.628*** (0.3246) | -4.633*** (0.3239) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN | -6.725*** (0.1889) | -6.730*** (0.1887) |
| i(var=IMP_METHOD_REF,ref=TRUE)RQ | -6.123*** (0.2065) | -6.128*** (0.2063) |
| i(var=IMP_METHOD_REF,ref=TRUE)RF | -6.585*** (0.1624) | -6.590*** (0.1620) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PMM | -5.357*** (0.2498) | -5.362*** (0.2493) |
| i(var=IMP_METHOD_REF,ref=TRUE)PMM | -6.148*** (0.2124) | -6.153*** (0.2121) |
| N | -0.9734*** (0.1581) | -0.7821*** (0.0720) |
| I(I(N^2)) | 0.1645** (0.0426) | |
| Percent_Missing | 2.398*** (0.1299) | 1.660*** (0.1169) |
| MISS_TYPESTATUS_w_DEMOG | 1.144*** (0.0265) | 1.140*** (0.0352) |
| MISS_TYPESTATUS_w_GROWTH | 2.229*** (0.0828) | 2.178*** (0.0833) |
| N x Percent_Missing | -0.3079** (0.0759) | |
| N x MISS_TYPESTATUS_w_DEMOG | 0.1397* (0.0530) | |
| N x MISS_TYPESTATUS_w_GROWTH | 0.3138** (0.0710) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 0.6422*** (0.0315) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 1.260*** (0.0762) | |
| Fixed-Effects: | ——————- | ——————- |
| CONTENT_AREA^GRADE | Yes | Yes |
| AMP_N | Yes | Yes |
| ________________________________________ | ___________________ | ___________________ |
| S.E.: Clustered | by: CON.^GRA. | by: CON.^GRA. |
| R2 | 0.33926 | 0.34496 |
| Within Adj. R2 | 0.33494 | 0.34068 |
| AIC | 15,668,441.0 | 15,645,705.8 |
| BIC | 15,669,310.1 | 15,646,626.0 |
\(~\)
Figure 3.4: Coefficient plot, additive models for scale score absolute bias at the grade/content area level, grades 5 through 8
Figure 3.5: Fixed effects estimates, additive model, grades 5 through 8
Figure 3.6: Fixed effects estimates, interaction model, grades 5 through 8
3.1.2 School Level
## NOTE: 26 observations removed because of NA values (LHS: 26).
## NOTE: 26 observations removed because of NA values (LHS: 26).| Additive Model | Interaction Model | |
|---|---|---|
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG | -4.446*** (0.0081) | -4.446*** (0.0081) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN | -5.209*** (0.0098) | -5.209*** (0.0098) |
| i(var=IMP_METHOD_REF,ref=TRUE)RQ | -4.932*** (0.0107) | -4.932*** (0.0107) |
| i(var=IMP_METHOD_REF,ref=TRUE)RF | -4.698*** (0.0093) | -4.698*** (0.0093) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PMM | -4.227*** (0.0114) | -4.227*** (0.0114) |
| i(var=IMP_METHOD_REF,ref=TRUE)PMM | -4.971*** (0.0111) | -4.971*** (0.0111) |
| N | -1.004*** (0.0072) | -0.3040*** (0.0029) |
| I(I(N^2)) | 0.0962*** (0.0010) | |
| Percent_Missing | 2.680*** (0.0087) | 0.7608*** (0.0062) |
| MISS_TYPESTATUS_w_DEMOG | 2.479*** (0.0127) | 2.506*** (0.0128) |
| MISS_TYPESTATUS_w_GROWTH | 6.358*** (0.0201) | 6.346*** (0.0197) |
| N x Percent_Missing | -0.5357*** (0.0057) | |
| N x MISS_TYPESTATUS_w_DEMOG | -0.3496*** (0.0070) | |
| N x MISS_TYPESTATUS_w_GROWTH | -1.019*** (0.0080) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 1.259*** (0.0119) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 3.513*** (0.0180) | |
| Fixed-Effects: | —————— | ——————- |
| AMP_N | Yes | Yes |
| ________________________________________ | __________________ | ___________________ |
| S.E.: Clustered | by: AMP_N | by: AMP_N |
| R2 | 0.56441 | 0.63224 |
| Within Adj. R2 | 0.56433 | 0.63217 |
| AIC | 3,985,182.4 | 3,861,490.0 |
| BIC | 3,985,884.1 | 3,862,237.6 |
\(~\)
Figure 3.7: Coefficient plot, additive models for scale score absolute bias at the school level
Figure 3.8: Fixed effects estimates, additive model
Figure 3.9: Fixed effects estimates, interaction model
3.2 SGPs
3.2.1 Grade and Content Area Level
## NOTE: 4,460 observations removed because of NA values (LHS: 4,460).
## NOTE: 4,460 observations removed because of NA values (LHS: 4,460).| Additive Model | Interaction Model | |
|---|---|---|
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG | 0.3374. (0.1435) | 0.3340. (0.1436) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN | -0.9210*** (0.1394) | -0.9243*** (0.1393) |
| i(var=IMP_METHOD_REF,ref=TRUE)RQ | 0.0892 (0.3153) | 0.0858 (0.3151) |
| i(var=IMP_METHOD_REF,ref=TRUE)RF | -0.9116** (0.2016) | -0.9149** (0.2015) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PMM | 0.2093 (0.3296) | 0.2060 (0.3294) |
| i(var=IMP_METHOD_REF,ref=TRUE)PMM | 0.0873 (0.3179) | 0.0839 (0.3177) |
| N | -1.647*** (0.2902) | -1.040*** (0.1012) |
| I(I(N^2)) | 0.2647** (0.0666) | |
| Percent_Missing | 2.187*** (0.0752) | 2.037*** (0.1035) |
| MISS_TYPESTATUS_w_DEMOG | 0.5630*** (0.0238) | 0.5694*** (0.0280) |
| MISS_TYPESTATUS_w_GROWTH | 0.5948*** (0.0487) | 0.5773*** (0.0538) |
| N x Percent_Missing | -0.5447** (0.1196) | |
| N x MISS_TYPESTATUS_w_DEMOG | -0.1652. (0.0828) | |
| N x MISS_TYPESTATUS_w_GROWTH | -0.1185 (0.1118) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 0.1717* (0.0520) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 0.2767** (0.0788) | |
| Fixed-Effects: | ——————- | ——————- |
| CONTENT_AREA^GRADE | Yes | Yes |
| AMP_N | Yes | Yes |
| ________________________________________ | ___________________ | ___________________ |
| S.E.: Clustered | by: CON.^GRA. | by: CON.^GRA. |
| R2 | 0.24939 | 0.24848 |
| Within Adj. R2 | 0.24509 | 0.24417 |
| AIC | 15,053,235.1 | 15,056,428.9 |
| BIC | 15,054,104.1 | 15,057,349.1 |
\(~\)
Figure 3.10: Coefficient plot, additive models for SGP absolute bias at the grade/content area level, grades 5 through 8
Figure 3.11: Fixed effects estimates, additive model
Figure 3.12: Fixed effects estimates, interaction model
3.2.2 School Level
## NOTE: 299 observations removed because of NA values (LHS: 299).
## NOTE: 299 observations removed because of NA values (LHS: 299).| Additive Model | Interaction Model | |
|---|---|---|
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG | 0.7479*** (0.0081) | 0.7467*** (0.0081) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN | -0.2217*** (0.0087) | -0.2229*** (0.0087) |
| i(var=IMP_METHOD_REF,ref=TRUE)RQ | 0.8704*** (0.0103) | 0.8692*** (0.0103) |
| i(var=IMP_METHOD_REF,ref=TRUE)RF | -0.0033 (0.0092) | -0.0045 (0.0092) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PMM | 0.9477*** (0.0089) | 0.9465*** (0.0089) |
| i(var=IMP_METHOD_REF,ref=TRUE)PMM | 0.8830*** (0.0098) | 0.8818*** (0.0098) |
| N | -1.316*** (0.0056) | -0.7466*** (0.0046) |
| I(I(N^2)) | 0.1443*** (0.0008) | |
| Percent_Missing | 1.457*** (0.0054) | 1.376*** (0.0098) |
| MISS_TYPESTATUS_w_DEMOG | 0.5648*** (0.0101) | 0.5512*** (0.0102) |
| MISS_TYPESTATUS_w_GROWTH | 0.5472*** (0.0095) | 0.5285*** (0.0094) |
| N x Percent_Missing | -0.4856*** (0.0047) | |
| N x MISS_TYPESTATUS_w_DEMOG | -0.2797*** (0.0071) | |
| N x MISS_TYPESTATUS_w_GROWTH | -0.2190*** (0.0067) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 0.0240* (0.0111) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 0.0420*** (0.0116) | |
| Fixed-Effects: | ——————- | ——————- |
| AMP_N | Yes | Yes |
| ________________________________________ | ___________________ | ___________________ |
| S.E.: Clustered | by: AMP_N | by: AMP_N |
| R2 | 0.24380 | 0.24017 |
| Within Adj. R2 | 0.24373 | 0.24010 |
| AIC | 3,786,128.6 | 3,789,633.9 |
| BIC | 3,786,830.1 | 3,790,381.5 |
\(~\)
Figure 3.13: Coefficient plot, additive models for SGP absolute bias at the school level
Figure 3.14: Fixed effects estimates, additive model
Figure 3.15: Fixed effects estimates, interaction model
3.3 Random Effects Models
We introduce a random intercept for school number to account for the hierarchical nature of the data.
## Linear mixed model fit by REML ['lmerMod']
## Formula: abs(SS_Raw_Bias) ~ IMP_METHOD_REF + N + Percent_Missing + MISS_TYPE +
## GRADE + CONTENT_AREA + (1 | SCHOOL_NUMBER)
## Data: MIsummary.GC.Eval
##
## REML criterion at convergence: 577609.8
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -7.0316 -0.5745 -0.0719 0.4538 13.3585
##
## Random effects:
## Groups Name Variance Std.Dev.
## SCHOOL_NUMBER (Intercept) 3.161 1.778
## Residual 22.145 4.706
## Number of obs: 97146, groups: SCHOOL_NUMBER, 232
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 7.03863 0.13008 54.11
## IMP_METHOD_REFL2PAN_LONG -3.40709 0.05649 -60.31
## IMP_METHOD_REFL2PAN -4.25093 0.05649 -75.25
## IMP_METHOD_REFRQ -3.60612 0.05649 -63.83
## IMP_METHOD_REFRF -3.78637 0.05649 -67.03
## IMP_METHOD_REFL2PMM -3.27434 0.05649 -57.96
## IMP_METHOD_REFPMM -3.56943 0.05649 -63.18
## N -0.21698 0.03647 -5.95
## Percent_Missing 3.09985 0.01726 179.56
## MISS_TYPESTATUS_w_DEMOG 2.63184 0.03698 71.16
## MISS_TYPESTATUS_w_GROWTH 6.53136 0.03699 176.56
## GRADE4 -0.49435 0.04489 -11.01
## GRADE5 -2.90312 0.04513 -64.33
## GRADE6 -3.39308 0.05526 -61.40
## GRADE7 -3.69397 0.08924 -41.40
## GRADE8 -3.35159 0.08925 -37.55
## CONTENT_AREAMATHEMATICS 0.36670 0.03020 12.14##
## Correlation matrix not shown by default, as p = 17 > 12.
## Use print(x, correlation=TRUE) or
## vcov(x) if you need it## Linear mixed model fit by REML ['lmerMod']
## Formula:
## abs(SGP_Raw_Bias) ~ IMP_METHOD_REF + N + Percent_Missing + MISS_TYPE +
## GRADE + CONTENT_AREA + (1 | SCHOOL_NUMBER)
## Data: MIsummary.GC.Eval
##
## REML criterion at convergence: 270887.4
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -3.8165 -0.5757 -0.0685 0.4428 13.0049
##
## Random effects:
## Groups Name Variance Std.Dev.
## SCHOOL_NUMBER (Intercept) 2.526 1.589
## Residual 9.969 3.157
## Number of obs: 52542, groups: SCHOOL_NUMBER, 227
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 2.00377 0.11637 17.219
## IMP_METHOD_REFL2PAN_LONG 0.96007 0.05154 18.628
## IMP_METHOD_REFL2PAN 0.05552 0.05154 1.077
## IMP_METHOD_REFRQ 1.68236 0.05154 32.642
## IMP_METHOD_REFRF 0.37734 0.05154 7.321
## IMP_METHOD_REFL2PMM 1.67852 0.05154 32.567
## IMP_METHOD_REFPMM 1.67868 0.05154 32.570
## N -0.42208 0.05091 -8.291
## Percent_Missing 1.75076 0.01586 110.385
## MISS_TYPESTATUS_w_DEMOG 0.70191 0.03375 20.800
## MISS_TYPESTATUS_w_GROWTH 0.91930 0.03376 27.230
## GRADE6 0.67067 0.04145 16.179
## GRADE7 0.36990 0.07273 5.086
## GRADE8 0.92720 0.07291 12.716
## CONTENT_AREAMATHEMATICS 0.42525 0.02755 15.436##
## Correlation matrix not shown by default, as p = 15 > 12.
## Use print(x, correlation=TRUE) or
## vcov(x) if you need it4 Percent Bias
In the following models, percent bias is dichotomized such that 1 indicates an observation with percent bias less than five, and 0 otherwise. Then, a series of logistic regression models are fit, examining the factors associated with a percent bias value lower than the 5% threshold (Miri et al., 2020; Qi et al., 2010).
4.1 Scale Scores
4.1.1 Grade and Content Area Level
The first set of models includes all grades (i.e., grade 3 through 8).
## NOTE: 11,300 observations removed because of NA values (LHS: 11,300).
## NOTE: 11,300 observations removed because of NA values (LHS: 11,300).| Additive Model | Interaction Model | |
|---|---|---|
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG | 1.608*** (0.1583) | 1.613*** (0.1597) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN | 2.230*** (0.4161) | 2.232*** (0.4122) |
| i(var=IMP_METHOD_REF,ref=TRUE)RQ | 2.226*** (0.4040) | 2.229*** (0.4002) |
| i(var=IMP_METHOD_REF,ref=TRUE)RF | 2.078*** (0.4139) | 2.082*** (0.4108) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PMM | 2.085*** (0.3639) | 2.088*** (0.3612) |
| i(var=IMP_METHOD_REF,ref=TRUE)PMM | 2.209*** (0.4076) | 2.211*** (0.4040) |
| N | 0.3524*** (0.0326) | 2.236*** (0.2634) |
| Percent_Missing | -2.811*** (0.1287) | -4.065*** (0.1652) |
| MISS_TYPESTATUS_w_DEMOG | -1.045*** (0.1108) | -2.813*** (0.3098) |
| MISS_TYPESTATUS_w_GROWTH | -3.118*** (0.1845) | -4.728*** (0.3591) |
| N x Percent_Missing | -0.6216*** (0.0789) | |
| N x MISS_TYPESTATUS_w_DEMOG | -0.6421*** (0.1704) | |
| N x MISS_TYPESTATUS_w_GROWTH | -1.287*** (0.2033) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 1.313*** (0.1837) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 1.154*** (0.2217) | |
| Fixed-Effects: | —————— | ——————- |
| CONTENT_AREA^GRADE | Yes | Yes |
| AMP_N | Yes | Yes |
| ________________________________________ | __________________ | ___________________ |
| S.E.: Clustered | by: CON.^GRA. | by: CON.^GRA. |
| Convergence | TRUE | TRUE |
| AIC | 707,746.7 | 700,729.1 |
| BIC | 708,697.7 | 701,747.1 |
\(~\)
Figure 4.1: Coefficient plot, additive model for scale score percent bias classification at the grade/content area level
Figure 4.2: Fixed effects estimates, additive model
Figure 4.3: Fixed effects estimates, interaction model
We can also subset the data to only observations from grades 5 through 8.
## NOTE: 4,460 observations removed because of NA values (LHS: 4,460).
## NOTE: 4,460 observations removed because of NA values (LHS: 4,460).| Additive Model | Interaction Model | |
|---|---|---|
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG | 1.808*** (0.2566) | 1.789*** (0.2545) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN | 5.363*** (0.4232) | 5.312*** (0.4200) |
| i(var=IMP_METHOD_REF,ref=TRUE)RQ | 4.897*** (0.5608) | 4.847*** (0.5565) |
| i(var=IMP_METHOD_REF,ref=TRUE)RF | 4.826*** (0.2592) | 4.777*** (0.2573) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PMM | 3.785*** (0.3947) | 3.741*** (0.3895) |
| i(var=IMP_METHOD_REF,ref=TRUE)PMM | 4.909*** (0.5817) | 4.860*** (0.5772) |
| N | 0.4647*** (0.0759) | 2.166*** (0.2198) |
| Percent_Missing | -2.489*** (0.1103) | -3.665*** (0.3235) |
| MISS_TYPESTATUS_w_DEMOG | -1.201*** (0.1678) | -2.335*** (0.4236) |
| MISS_TYPESTATUS_w_GROWTH | -2.298*** (0.1936) | -3.947*** (0.5961) |
| N x Percent_Missing | -0.4191* (0.1793) | |
| N x MISS_TYPESTATUS_w_DEMOG | -0.8481*** (0.2309) | |
| N x MISS_TYPESTATUS_w_GROWTH | -1.439*** (0.2607) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 0.9304*** (0.2159) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 1.291*** (0.3081) | |
| Fixed-Effects: | —————— | ——————- |
| CONTENT_AREA^GRADE | Yes | Yes |
| AMP_N | Yes | Yes |
| ________________________________________ | __________________ | ___________________ |
| S.E.: Clustered | by: CON.^GRA. | by: CON.^GRA. |
| Convergence | TRUE | TRUE |
| AIC | 199,419.3 | 197,537.9 |
| BIC | 200,275.5 | 198,458.0 |
\(~\)
Figure 4.4: Coefficient plot, additive model for scale score percent bias classification at the grade/content area level, grades 5 through 8
Figure 4.5: Fixed effects estimates, additive model, grades 5 through 8
Figure 4.6: Fixed effects estimates, interaction model, grades 5 through 8
4.1.2 School Level
## NOTE: 26 observations removed because of NA values (LHS: 26).
## NOTE: 26 observations removed because of NA values (LHS: 26).| Additive Model | Interaction Model | |
|---|---|---|
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG | 2.951*** (0.0383) | 2.969*** (0.0375) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN | 4.121*** (0.0389) | 4.135*** (0.0385) |
| i(var=IMP_METHOD_REF,ref=TRUE)RQ | 4.006*** (0.0398) | 4.020*** (0.0392) |
| i(var=IMP_METHOD_REF,ref=TRUE)RF | 3.530*** (0.0391) | 3.545*** (0.0382) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PMM | 3.447*** (0.0396) | 3.464*** (0.0395) |
| i(var=IMP_METHOD_REF,ref=TRUE)PMM | 4.054*** (0.0410) | 4.068*** (0.0411) |
| N | 0.7901*** (0.0248) | -0.0537 (0.0451) |
| Percent_Missing | -3.859*** (0.0411) | 0.0009 (0.0043) |
| MISS_TYPESTATUS_w_DEMOG | -14.19*** (0.0506) | -11.92*** (0.1276) |
| MISS_TYPESTATUS_w_GROWTH | -17.52*** (0.0183) | -14.82*** (0.0478) |
| N x Percent_Missing | -0.3807*** (0.0610) | |
| N x MISS_TYPESTATUS_w_DEMOG | 2.365*** (0.1168) | |
| N x MISS_TYPESTATUS_w_GROWTH | 1.268*** (0.0495) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | -3.540*** (0.0709) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | -4.085*** (0.0353) | |
| Fixed-Effects: | —————— | ——————- |
| AMP_N | Yes | Yes |
| ________________________________________ | __________________ | ___________________ |
| S.E.: Clustered | by: AMP_N | by: AMP_N |
| Convergence | TRUE | TRUE |
| AIC | 37,627.8 | 37,419.9 |
| BIC | 38,317.9 | 38,167.5 |
\(~\)
Figure 4.7: Coefficient plot, additive model for scale score percent bias classification at the school level
Figure 4.8: Fixed effects estimates, additive model
Figure 4.9: Fixed effects estimates, interaction model
4.2 SGPs
4.2.1 Grade and Content Area Level
## NOTE: 4,460 observations removed because of NA values (LHS: 4,460).
## NOTE: 4,460 observations removed because of NA values (LHS: 4,460).| Additive Model | Interaction Model | |
|---|---|---|
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG | -0.3293*** (0.0402) | -0.3299*** (0.0401) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN | 0.0864*** (0.0235) | 0.0861*** (0.0235) |
| i(var=IMP_METHOD_REF,ref=TRUE)RQ | -0.3564*** (0.1066) | -0.3570*** (0.1066) |
| i(var=IMP_METHOD_REF,ref=TRUE)RF | 0.1112* (0.0525) | 0.1108* (0.0525) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PMM | -0.4385*** (0.1117) | -0.4393*** (0.1117) |
| i(var=IMP_METHOD_REF,ref=TRUE)PMM | -0.3562** (0.1117) | -0.3568** (0.1117) |
| N | 0.4209*** (0.0726) | 0.4354*** (0.0730) |
| Percent_Missing | -0.7939*** (0.0123) | -0.7552*** (0.0116) |
| MISS_TYPESTATUS_w_DEMOG | -0.0578* (0.0268) | -0.0621* (0.0258) |
| MISS_TYPESTATUS_w_GROWTH | -0.0849*** (0.0197) | -0.0823*** (0.0230) |
| N x Percent_Missing | -0.1052*** (0.0169) | |
| N x MISS_TYPESTATUS_w_DEMOG | 0.0004 (0.0244) | |
| N x MISS_TYPESTATUS_w_GROWTH | -0.0706** (0.0259) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | -0.0434** (0.0161) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | -0.0374 (0.0271) | |
| Fixed-Effects: | ——————- | ——————- |
| CONTENT_AREA^GRADE | Yes | Yes |
| AMP_N | Yes | Yes |
| ________________________________________ | ___________________ | ___________________ |
| S.E.: Clustered | by: CON.^GRA. | by: CON.^GRA. |
| Convergence | TRUE | TRUE |
| AIC | 3,143,574.3 | 3,141,265.4 |
| BIC | 3,144,430.5 | 3,142,185.5 |
\(~\)
Figure 4.10: Coefficient plot, additive model for SGP percent bias classification at the grade/content area level, grades 5 through 8
Figure 4.11: Fixed effects estimates, additive model
Figure 4.12: Fixed effects estimates, interaction model
4.2.2 School Level
## NOTE: 299 observations removed because of NA values (LHS: 299).
## NOTE: 299 observations removed because of NA values (LHS: 299).| Additive Model | Interaction Model | |
|---|---|---|
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN_LONG | -0.5914*** (0.0075) | -0.5913*** (0.0075) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PAN | -0.0033 (0.0073) | -0.0037 (0.0072) |
| i(var=IMP_METHOD_REF,ref=TRUE)RQ | -0.6874*** (0.0065) | -0.6876*** (0.0065) |
| i(var=IMP_METHOD_REF,ref=TRUE)RF | -0.1445*** (0.0070) | -0.1445*** (0.0069) |
| i(var=IMP_METHOD_REF,ref=TRUE)L2PMM | -0.7487*** (0.0086) | -0.7491*** (0.0086) |
| i(var=IMP_METHOD_REF,ref=TRUE)PMM | -0.6888*** (0.0072) | -0.6890*** (0.0072) |
| N | 0.6187*** (0.0033) | 0.6639*** (0.0071) |
| Percent_Missing | -0.7776*** (0.0030) | -0.8655*** (0.0078) |
| MISS_TYPESTATUS_w_DEMOG | -0.1319*** (0.0072) | -0.1295*** (0.0073) |
| MISS_TYPESTATUS_w_GROWTH | -0.1812*** (0.0076) | -0.1871*** (0.0074) |
| N x Percent_Missing | -0.1122*** (0.0047) | |
| N x MISS_TYPESTATUS_w_DEMOG | -0.0094 (0.0088) | |
| N x MISS_TYPESTATUS_w_GROWTH | -0.0621*** (0.0092) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 0.0822*** (0.0081) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 0.0896*** (0.0095) | |
| Fixed-Effects: | ——————- | ——————- |
| AMP_N | Yes | Yes |
| ________________________________________ | ___________________ | ___________________ |
| S.E.: Clustered | by: AMP_N | by: AMP_N |
| Convergence | TRUE | TRUE |
| AIC | 869,254.8 | 868,226.8 |
| BIC | 869,944.8 | 868,974.4 |
\(~\)
Figure 4.13: Coefficient plot, additive model for SGP percent bias classification at the school level
Figure 4.14: Fixed effects estimates, additive model
Figure 4.15: Fixed effects estimates, interaction model
5 Cross-Sectional L2PAN
For the final set of models, we examine factors influencing absolute scale score and SGP bias only among data imputed using cross-sectional L2PAN.
5.1 Scale Scores
5.1.1 Grade and Content Area Level
The first set of models includes all grades (i.e., grade 3 through 8).
| Additive Model | Interaction Model | |
|---|---|---|
| N | -1.036*** (0.1441) | -0.7433*** (0.0937) |
| I(I(N^2)) | 0.1824*** (0.0374) | |
| Percent_Missing | 2.926*** (0.5607) | 1.444*** (0.1367) |
| MISS_TYPESTATUS_w_DEMOG | 1.163** (0.3127) | 1.177** (0.3135) |
| MISS_TYPESTATUS_w_GROWTH | 4.365** (1.325) | 4.319** (1.314) |
| N x Percent_Missing | -0.5175** (0.1457) | |
| N x MISS_TYPESTATUS_w_DEMOG | -0.0038 (0.0851) | |
| N x MISS_TYPESTATUS_w_GROWTH | 0.1322 (0.2021) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 0.8259** (0.2249) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 2.935** (0.8798) | |
| Fixed-Effects: | —————— | ——————- |
| CONTENT_AREA^GRADE | Yes | Yes |
| AMP_N | Yes | Yes |
| ________________________________________ | __________________ | ___________________ |
| S.E.: Clustered | by: CON.^GRA. | by: CON.^GRA. |
| R2 | 0.42955 | 0.46409 |
| Within Adj. R2 | 0.34403 | 0.38374 |
| AIC | 4,171,923.8 | 4,128,599.4 |
| BIC | 4,172,679.5 | 4,129,400.9 |
\(~\)
Figure 5.1: Coefficient plot, L2PAN additive models for scale score absolute bias at the grade/content area level
Figure 5.2: Fixed effects estimates, additive model
Figure 5.3: Fixed effects estimates, interaction model
We can also subset the data to only observations from grades 5 through 8.
| Additive Model | Interaction Model | |
|---|---|---|
| N | -1.054*** (0.1940) | -0.7101*** (0.0704) |
| I(I(N^2)) | 0.1785** (0.0499) | |
| Percent_Missing | 1.390*** (0.1059) | 1.161*** (0.0870) |
| MISS_TYPESTATUS_w_DEMOG | 0.3229*** (0.0448) | 0.3184*** (0.0457) |
| MISS_TYPESTATUS_w_GROWTH | 0.6910*** (0.1101) | 0.6643*** (0.1097) |
| N x Percent_Missing | -0.2824* (0.0938) | |
| N x MISS_TYPESTATUS_w_DEMOG | 0.0109 (0.0539) | |
| N x MISS_TYPESTATUS_w_GROWTH | 0.0663 (0.0856) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 0.1674** (0.0457) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 0.4485** (0.0833) | |
| Fixed-Effects: | —————— | ——————- |
| CONTENT_AREA^GRADE | Yes | Yes |
| AMP_N | Yes | Yes |
| ________________________________________ | __________________ | ___________________ |
| S.E.: Clustered | by: CON.^GRA. | by: CON.^GRA. |
| R2 | 0.25909 | 0.25820 |
| Within Adj. R2 | 0.24588 | 0.24497 |
| AIC | 1,808,290.1 | 1,808,750.9 |
| BIC | 1,808,961.9 | 1,809,466.1 |
\(~\)
Figure 5.4: Coefficient plot, L2PAN additive models for scale score absolute bias at the grade/content area level, grades 5 through 8
Figure 5.5: Fixed effects estimates, additive model, grades 5 through 8
Figure 5.6: Fixed effects estimates, interaction model, grades 5 through 8
5.1.2 School Level
| Additive Model | Interaction Model | |
|---|---|---|
| N | -1.229*** (0.0071) | -0.3834*** (0.0037) |
| I(I(N^2)) | 0.1118*** (0.0009) | |
| Percent_Missing | 2.051*** (0.0092) | 0.5970*** (0.0075) |
| MISS_TYPESTATUS_w_DEMOG | 1.515*** (0.0129) | 1.528*** (0.0130) |
| MISS_TYPESTATUS_w_GROWTH | 4.961*** (0.0193) | 4.939*** (0.0189) |
| N x Percent_Missing | -0.6365*** (0.0053) | |
| N x MISS_TYPESTATUS_w_DEMOG | -0.4403*** (0.0078) | |
| N x MISS_TYPESTATUS_w_GROWTH | -1.273*** (0.0088) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 0.7713*** (0.0136) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 2.815*** (0.0195) | |
| Fixed-Effects: | —————— | ——————- |
| AMP_N | Yes | Yes |
| ________________________________________ | __________________ | ___________________ |
| S.E.: Clustered | by: AMP_N | by: AMP_N |
| R2 | 0.50375 | 0.59597 |
| Within Adj. R2 | 0.50360 | 0.59583 |
| AIC | 530,476.4 | 509,020.8 |
| BIC | 531,001.9 | 509,584.6 |
\(~\)
Figure 5.7: Coefficient plot, L2PAN additive models for scale score absolute bias at the school level
Figure 5.8: Fixed effects estimates, additive model
Figure 5.9: Fixed effects estimates, interaction model
5.2 SGPs
5.2.1 Grade and Content Area Level
| Additive Model | Interaction Model | |
|---|---|---|
| N | -1.603** (0.2975) | -1.038*** (0.0956) |
| I(I(N^2)) | 0.2617** (0.0748) | |
| Percent_Missing | 1.733*** (0.0694) | 1.598*** (0.0918) |
| MISS_TYPESTATUS_w_DEMOG | 0.4835*** (0.0740) | 0.4818*** (0.0765) |
| MISS_TYPESTATUS_w_GROWTH | 0.6581*** (0.1056) | 0.6333*** (0.1072) |
| N x Percent_Missing | -0.4414* (0.1340) | |
| N x MISS_TYPESTATUS_w_DEMOG | -0.0816 (0.0806) | |
| N x MISS_TYPESTATUS_w_GROWTH | -0.0258 (0.1059) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 0.1295 (0.0855) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 0.2678* (0.0978) | |
| Fixed-Effects: | —————— | —————— |
| CONTENT_AREA^GRADE | Yes | Yes |
| AMP_N | Yes | Yes |
| ________________________________________ | __________________ | __________________ |
| S.E.: Clustered | by: CON.^GRA. | by: CON.^GRA. |
| R2 | 0.24245 | 0.23847 |
| Within Adj. R2 | 0.23687 | 0.23285 |
| AIC | 2,011,334.0 | 2,013,311.4 |
| BIC | 2,012,005.8 | 2,014,026.5 |
\(~\)
Figure 5.10: Coefficient plot, L2PAN additive models for SGP absolute bias at the grade/content area level, grades 5 through 8
Figure 5.11: Fixed effects estimates, additive model
Figure 5.12: Fixed effects estimates, interaction model
5.2.2 School Level
| Additive Model | Interaction Model | |
|---|---|---|
| N | -1.331*** (0.0066) | -0.7850*** (0.0067) |
| I(I(N^2)) | 0.1472*** (0.0009) | |
| Percent_Missing | 1.164*** (0.0068) | 1.093*** (0.0119) |
| MISS_TYPESTATUS_w_DEMOG | 0.5164*** (0.0132) | 0.5031*** (0.0132) |
| MISS_TYPESTATUS_w_GROWTH | 0.6416*** (0.0133) | 0.6236*** (0.0132) |
| N x Percent_Missing | -0.4590*** (0.0053) | |
| N x MISS_TYPESTATUS_w_DEMOG | -0.2140*** (0.0094) | |
| N x MISS_TYPESTATUS_w_GROWTH | -0.1685*** (0.0087) | |
| Percent_Missing x MISS_TYPESTATUS_w_DEMOG | 0.0057 (0.0136) | |
| Percent_Missing x MISS_TYPESTATUS_w_GROWTH | 0.0428** (0.0138) | |
| Fixed-Effects: | —————— | ——————- |
| AMP_N | Yes | Yes |
| ________________________________________ | __________________ | ___________________ |
| S.E.: Clustered | by: AMP_N | by: AMP_N |
| R2 | 0.26371 | 0.25437 |
| Within Adj. R2 | 0.26353 | 0.25416 |
| AIC | 494,973.8 | 496,297.2 |
| BIC | 495,499.4 | 496,861.0 |
\(~\)
Figure 5.13: Coefficient plot, L2PAN additive models for SGP absolute bias at the school level
Figure 5.14: Fixed effects estimates, additive model
Figure 5.15: Fixed effects estimates, interaction model
5.3 Random Effects Models
We introduce a random intercept for school number to account for the hierarchical nature of the data.
## Linear mixed model fit by REML ['lmerMod']
## Formula: abs(SS_Raw_Bias) ~ N + Percent_Missing + MISS_TYPE + GRADE +
## CONTENT_AREA + (1 | SCHOOL_NUMBER)
## Data: l2pan.GC.Eval
##
## REML criterion at convergence: 79181
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -5.3986 -0.5932 -0.1025 0.4763 11.4303
##
## Random effects:
## Groups Name Variance Std.Dev.
## SCHOOL_NUMBER (Intercept) 3.009 1.735
## Residual 16.871 4.107
## Number of obs: 13878, groups: SCHOOL_NUMBER, 232
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 4.26178 0.15128 28.171
## N -0.28436 0.07147 -3.979
## Percent_Missing 2.54639 0.03945 64.554
## MISS_TYPESTATUS_w_DEMOG 1.88806 0.08541 22.107
## MISS_TYPESTATUS_w_GROWTH 5.50547 0.08542 64.449
## GRADE4 -0.56714 0.10355 -5.477
## GRADE5 -4.45360 0.10408 -42.791
## GRADE6 -4.73850 0.12520 -37.848
## GRADE7 -5.14284 0.19121 -26.896
## GRADE8 -5.00958 0.19125 -26.194
## CONTENT_AREAMATHEMATICS 0.22402 0.06973 3.213
##
## Correlation of Fixed Effects:
## (Intr) N Prcn_M MISS_TYPESTATUS__D MISS_TYPESTATUS__G
## N -0.069
## Prcnt_Mssng 0.005 0.006
## MISS_TYPESTATUS__D -0.282 0.000 -0.008
## MISS_TYPESTATUS__G -0.282 0.000 -0.021 0.500
## GRADE4 -0.337 0.013 -0.004 0.000 0.000
## GRADE5 -0.337 0.020 -0.003 0.000 0.000
## GRADE6 -0.341 0.036 0.002 0.000 0.000
## GRADE7 -0.314 0.185 0.012 0.000 0.000
## GRADE8 -0.314 0.183 0.008 0.000 0.000
## CONTENT_ARE -0.230 0.000 0.000 0.000 0.000
## GRADE4 GRADE5 GRADE6 GRADE7 GRADE8
## N
## Prcnt_Mssng
## MISS_TYPESTATUS__D
## MISS_TYPESTATUS__G
## GRADE4
## GRADE5 0.492
## GRADE6 0.407 0.408
## GRADE7 0.262 0.270 0.421
## GRADE8 0.262 0.269 0.421 0.575
## CONTENT_ARE 0.000 0.000 0.000 0.000 0.000## Linear mixed model fit by REML ['lmerMod']
## Formula: abs(SGP_Raw_Bias) ~ N + Percent_Missing + MISS_TYPE + GRADE +
## CONTENT_AREA + (1 | SCHOOL_NUMBER)
## Data: l2pan.GC.Eval
##
## REML criterion at convergence: 35507.7
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -4.0776 -0.5689 -0.0569 0.4210 9.1465
##
## Random effects:
## Groups Name Variance Std.Dev.
## SCHOOL_NUMBER (Intercept) 1.510 1.229
## Residual 6.207 2.491
## Number of obs: 7506, groups: SCHOOL_NUMBER, 227
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 2.34118 0.10840 21.597
## N -0.51924 0.06995 -7.423
## Percent_Missing 1.41083 0.03260 43.275
## MISS_TYPESTATUS_w_DEMOG 0.81196 0.07045 11.526
## MISS_TYPESTATUS_w_GROWTH 1.18504 0.07048 16.815
## GRADE6 0.37593 0.08279 4.541
## GRADE7 0.01595 0.12906 0.124
## GRADE8 0.08622 0.12938 0.666
## CONTENT_AREAMATHEMATICS 0.02165 0.05751 0.376
##
## Correlation of Fixed Effects:
## (Intr) N Prcn_M MISS_TYPESTATUS__D MISS_TYPESTATUS__G
## N -0.108
## Prcnt_Mssng 0.009 0.020
## MISS_TYPESTATUS__D -0.325 0.000 -0.017
## MISS_TYPESTATUS__G -0.325 -0.001 -0.033 0.500
## GRADE6 -0.308 0.032 0.006 0.000 0.000
## GRADE7 -0.303 0.306 0.022 0.000 -0.001
## GRADE8 -0.304 0.306 0.016 0.000 -0.001
## CONTENT_ARE -0.265 0.000 0.000 0.000 0.000
## GRADE6 GRADE7 GRADE8
## N
## Prcnt_Mssng
## MISS_TYPESTATUS__D
## MISS_TYPESTATUS__G
## GRADE6
## GRADE7 0.458
## GRADE8 0.460 0.655
## CONTENT_ARE 0.000 0.000 0.0006 References and Helpful R Resources
- Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1-48. https://doi.org/10.18637/jss.v067.i01.
- Berge, L. (2018). Efficient estimation of maximum likelihood models with multiple fixed-effects: the R package FENmlm. CREA Discussion Papers.
- Berge, L. (2021, March 29). Fast fixed-effects estimation: Short introduction. Retrieved from https://cran.r-project.org/web/packages/fixest/vignettes/multiple_estimations.html
- DataMentor R Plot Tutorial
- Miri, H. H., Hassanzadeh, J., Khaniki, S. H., Akrami, R., & Sirjani, E. (2020). Accuracy of five multiple imputation methods in estimating prevalence of Type 2 diabetes based on STEPS surveys. Journal of Epidemiology and Global Health, 10(1), 36-41. https://doi.org/10.2991/jegh.k.191207.001
- Qi, L., Wang, Y.-F., & He, Y. (2010). A comparison of multiple imputation and fully augmented weighted estimators for Cox regression with missing covariates. Statistics in Medicine, 29(25), 2592-2604. https://doi.org/10.1002/sim.4016