About

Each year, professional staff at the Center for Assessment identify a topic which is outside of our current areas of expertise, but is likely to have a significant impact on the field and our work in the near future. Over the course of the year, Center associates gather information on the topic, identify and interact with experts in the field, and develop a set of critical ideas/questions to be addressed. The culmination of this effort is a two-day meeting in June in which experts are convened to share information and engage in discussions with Center staff.

Colloquium 2019, Educational Assessment Enhanced by Analytics, Machine Learning (ML), and Artificial Intelligence (AI), examines how educational assessment is already being enhanced by the use of analytics, machine learning, and artificial intelligence. Applications include (put “automated” in front of each of the following terms) item/dialog generation, computer adaptive testing (CAT), essay scoring; assessment of difficult constructs including collaboration and leadership; showing correspondences between competencies and evidence across different contexts; development of student models; formative assessment through dialogic language and other natural and dynamic inputs; content generation; summarization of text, recommendations of instructional materials; tutoring; recommendations of course options and career pathways; data analysis for monitoring and improving programs.

As we gather together in Portland, Oregon, our goals is to jointly explore how the affordances provided by analytics, machine learning and artificial intelligence will shape educational assessments in the next three to five years and the implications to stakeholders and assessment professionals.

Here is a welcome letter from Drs. Scott Marion and Chris Domaleski, Executive Director and Associate Executive Director of the Center.

Location

Colloquium 2019 will be held at the The Nines, 525 SW Morrison, Portland, Oregon. Beautiful Downtown Portland is about a 12 miles from Portland International Airport (PDX). Participants may take a cab service, Uber/Lyft, or the MAX Light Rail to and from the airport.

Experts

Center Staff

Schedule

Time Topic Description
Tuesday, June 4th
5:30pm Dinner (Optional) Join us at Forktown Food Tours to experience some of the best food in Portland on a 1.5-hour culinary walking tour of the vibrant downtown.
Wednesday, June 5th
7:30am Check-in / Breakfast
8:15am Scott Marion Welcome Center for Assessment Executive Director Dr. Scott Marion welcomes participants on behalf of the Center to Colloquium 2019. Center staff introductions.
8:30am Brian Gong Overview Center for Assessment Dr. Brian Gong provides an overview of the Colloquium.
8:45am Experts Introduction Experts introduce themselves and describe one difference in education they envision and what they are working on that involves data analytics, ML, or AI.
9:15am Leslie Keng Group Discussion #1 Center for Assessment Current at-scale application for data analytics, ML and AI in education. What compelling applications in education are available now at scale (e.g., commercially, as in, a state or district might be able to adopt for operational use)? (Working definitions)
10:15am Break
10:45am Nathan Dadey Group Discussion #2 Center for Assessment Promising paths and limitations for analytics, ML, AI applications: Applications that largely replace current assessment technologies and uses. What are key capacities that will likely be available in 3-5 years? 5-10 years? What commonly promoted or promised applications will likely not to be operational at scale in the next 10 years? Why not? What are promising paths for adoption and implementation? What underlying work models and capacities do these applications require? How different are they from how things are currently done?
12:00pm Lunch
1:00pm Erika Landl Group Discussion #3 Center for Assessment Promising paths and limitations for analytics, ML, AI applications: Comprehensive assessment systems. What are key capacities that will likely be available in 3-5 years? 5-10 years? What commonly promoted or promised applications will likely not to be operational at scale in the next 10 years? Why not? What are promising paths for adoption and implementation? What underlying work models and capacities do these applications require? How different are they from how things are currently done?
2:30pm Break
3:00pm Joseph Martineau Group Discussion #4 Center for Assessment Promising paths and limitations for analytics, ML, AI applications: Intelligent systems. What are key capacities that will likely be available in 3-5 years? 5-10 years? What commonly promoted or promised applications will likely not to be operational at scale in the next 10 years? Why not? What are promising paths for adoption and implementation? What underlying work models and capacities do these applications require? How different are they from how things are currently done?
4:30pm Adjourn for the day
5:30pm Dinner Join us for dinner at Higgins.
Thursday, June 6th
7:30am Breakfast
8:00am Synthesis Synthesis and reflections from Day 1.
8:30am Carla Evans Group Discussion #5 Center for Assessment Challenges to Quality. What are known challenges to quality use (e.g., where do data come from; detecting biased data; dealing with variation and exceptions; privacy—similar issues could be identified for infrastructure, literacy, etc.), and how may those challenges be addressed?
10:00am Break
10:30am Juan D'Brot Group Discussion #6 Center for Assessment Evaluating Validity and Utility. What evidence should an end-user ask for when evaluating the validity and utility of an application? How is that evidence similar to and differ from traditional evaluation of tests?
12:00pm Lunch
1:00pm Brian Gong Group Discussion #7 Center for Assessment The Center's Role. What role can the Center play?
2:30pm Chris Domaleski Wrapup Center for Assessment Associate Director Dr. Chris Domaleski provides final reflections and thanks all participants on behalf of the Center.
3:00pm Adjourn

Explore

Downtown Portland and the surrounding area provide an abundance of things to do.

  • Columbia River Gorge
    • Take a Segway tour of downtown Portland (rated #4 of Tours to do in Portland on TripAdvisor)

    • Visit the Columbia River Gorge

    • Enjoy one or more of the parks or gardens that Portland is famous for (110 listed on Trip Advisor including massive Washington Park, the International Rose Test Garden, the Portland Japanese Garden, and the Lan Su Chinese Garden)

    • Join the Beerquest Haunted Pub Walking Tour