0 SIGCSE - PM Sessions - Concept Inventories, Web Services for CS1

Developing a Validated Assesment of Fundamental CS1 Concepts
Allison Elliott Tew and Mark Guzdial, Georgia Tech (allison@cc.gatech.edu)

• Educators need to be able to measure student learning.
• Related studies - McCracken, et al 2001 (explored students basic programming abilities using simple calculator program); Lister, et al, 2004 (Assessed student code comprehension and tacing ability using multiple choice questions about arrays & iteration); Tew, et al, 2005 (Investigated impact of different CS1 courses on students' conceptual knowledge at the end of the course using MCQs)
• Content validity - are the topics a reasonable operationalization of the area
• Construct validity - accurately measuring knowledge of a topic.
• We currently do not have the tools to assess CS1 concepts. Why? We are a young field.
• Method for developing an assessment - define content, expert review of test specification, build test bank of questions, pilot questions, establish validity, establish reliability.
• (From above bullet) We need to add step 4: Verify language independence
• Analyzing widely adopted textbooks, proved unsuccessful. Textbooks covered a greater range of topics than what is covered in a CS1 course.

List of topics

1. Fundamentals (vars, assignment .... )
   
2. Logical operators
3. Selection statment
4. Definite loops (for)
5. Indefinite loops (while)
6. Arrays
7. Function/method parameters
8. Function/method return values
9. Recursion
10. Object oriented basics
    

• 3 types of questions for each concept - definition, tracing, code completion.
• two versions of each question

***********************************************************
A Web Services approach to CS1
Illinois State University

• Statistically significant improvement in final exam performance and course grade; but grading bias and other concerns.
• Hoping to get industry support
• Where do you find these web services? Web services often disappear. You don't have control.
• They are building a library of their own web services.

************************************************************
Student Misconceptions of Programming
Kaczmarcyk, Petrick, East, Herman (Univ of San Diego, Northern Iowa, Illinois at Urbana)

• focuses on concepts that are important and difficult
• concrete & immediately actionable
• not comprehensive - short!
• 32 important & difficult concepts in CS1 - Goldman et al. in press; Goldman et al. 2008
• Their goal was to confirm (or not) Delphi results
• Identify widely held misconceptions
• Used modified think-aloud interviews, problems both code & non code based
• 89 consisten, reliably repeated misconceptions: memory usage & allocation, array construction, primitive confusion
• Validated (with 2 exceptions) the Delphi results - basic loop operations stump many students, poor to NO conception of objects; NO conception of inheritance
• no conception == no misconceptions
• many of the problems were caused because the students applied a real world concept to a problem. "all of the meats will be in the meats array, and all of the cheese will automatically be in the cheese array." WRONG.