Ignite Sessions
Ignite Sessions are a high energy and innovative way to present content in a quick and concise format. Twenty slides automatically advance at 15-second intervals for a total presentation time of five minutes in length. Join us for these quick presentations on hot topics that also include an additional five minutes for Q&A following each presentation.
Assessing 21st Century Skills: Practical Considerations and Approaches
21st century skills are increasingly important for success in our rapidly changing, digital society. These skills include, for example, digital literacy, creativity, problem solving and cooperation. They differ from traditional academic skills, since they are not primarily content knowledge based, which makes it particularly hard to assess 21st century skills using traditional types of assessment. In the field of educational measurement, the question arises how assessments for 21st century skills should be designed.
This session will focus on practical considerations and approaches for assessing digital literacy, which has to do with how individuals access, understand, communicate and create digital content, and use digital devices, tools and applications. First, we will introduce the challenges in assessing 21st century skills. Then, we will elaborate on the particular characteristics of digital literacy. Finally, it will be illustrated how an eye-tracking device can be used to support validity research of a digital literacy assessment.
Speaker: Marie-Anne Keizer-Mittelhaeuser, Cito
Feedback on Feedback – Development of a Feedback Instrument and Program, Implications, and Relationship to the Big 5 Assessment
Over the past year and half, AERE has developed a validated instrument that measures how individuals deliver and receive feedback called the Straehle Feedback Inventory. Feedback in the workplace has been a continually discussed topic in various business publications such as Harvard Business Review, Inc., Fortune, and other periodicals. It is believed that organizations lose substantial amounts of money who employ individuals who don’t deliver and receive feedback effectively.
This session will focus on the development of the instrument including the use of Amazon’s Mechanical Turk to recruit over 500 participants. In addition, the presenter will share some interesting results such as one-fifth of the respondents don’t always give facts when delivering feedback. The session will discuss the relationship of the Straehle Feedback Inventory and the Big 5 assessment. Finally, it will discuss how AERE has developed a training and assessment program that can be incorporated to advance human capital anywhere in the world.
Speaker: Manny Straehle, Assessment, Education, and Research Experts, LLC
Garbage In, Garbage Out: Feed AI Translatable Items
Good news for multilingual testing: neural machine translation (NMT) is progressing fast, and multiple language versions of assessment item banks will (soon) be produced faster and at a lesser cost. Caveat: this only works when the master version is up to par. The source needs to be translatable. Ambiguities need to be removed. The machine cannot infer from the context whether a person actually teaches Dutch or is merely a Dutch citizen who teaches. In multilingual testing, as AI takes over a larger chunk of the downstream production process, the need for multidisciplinary upstream work increases; test publishers begin to organise thorough reviews of their assessment item banks by ad hoc panels before commencing translation. Such panels typically include psychometricians, domain specialists and expert linguists. The methodology is in place. There is a robust framework to categorize translatability issues in source items. There are databases of solutions that effectively circumvented translatability issues. If you get your master version right from the outset, you are equipped to surf the wave of new advances in AI and to conquer the global assessment market.
The IT majors have understood this. Their tests and questionnaires now undergo a thorough translatability assessment before they are sent to translation. In the long run, they will be translated by AI and post-edited by translators who caught the train and adapted to their new role.
Speaker: Steve Dept, cApStAn Linguistic Quality Control
PIE Assessment
The highly innovative and research-based PIE method (Preselected Items Evaluation) allows an evaluator to easily and efficiently rank a large number of open questions or text productions. An evaluation is reliable when it is judged accurate, objective, stable and reproducible. Using the PIE method (Preselected Items Evaluation, Kockaert & Segers 2014), you can evaluate the quality of a translation by evaluating only a limited number of items, not the entire text. The PIE method addresses the issue of reliability and the different types of validity (construct, content, ecological, face, pragmatic, and predictive validity).
After automatically correcting and scoring the translations, EvaluationQ calculates two psychometric values (p-value or difficulty level of the item and d-index, discriminability index) which can eliminate invalid or non-discriminative items. The final result allows the evaluator to rank the translations accurately and objectively. All answers are treated with the same criteria.The method has been tested with medical, legal and even literature texts. The advanced algorithms allow for complex key phrases to be recognised and handled.
Again, this method allows for ranking candidates, not for analysing full texts as such. The method is operational in pilot studies and real use cases can be shown.
Speakers: Bert Wylin, Televic Education; Winibert Segers, KU Leuven
For a complete list of sessions, visit our full programme.
Visit our full programme
Grand Hotel Huis ter
Duin Noordwijk, the Netherlands Grand Hotel Huis ter
Duin Noordwijk, The Netherlands