IS4800 – Empirical Research Methods in Information Science

[Syllabus] [Schedule] [Homework] [Projects] [Bibliography] [Resources] [Directory] [Grads]

Graduate Empirical Research Methods Course (CS6350)

Extra requirements for graduate students: In addition to the undergraduate course requirements, graduate students must each design one additional publication-quality empirical research study and attend one extra one-hour seminar meeting per week during the semester. The study must be on a topic and use methods approved by the instructor. Graduate students must submit a five-page proposal on the study and analysis techniques to be used prior to the midterm, and apply for Northeastern IRB approval by the end of the semester. In addition to project presentations, the seminar meetings will be used to critique study designs from the current HCI and health informatics literature.

Seminar Schedule: We will spend the first half of the semester reviewing and critiquing studies from the HCI and health informatics literature. During each of these classes two students will present the work (15 minutes each) and their critique. Thereafter, all students will discuss and critique it. Following the midterm, study proposals (including IRB applications) will be presented and discussed, in addition to advanced study designs and statistical analysis techniques, with one student assigned to give a 30 minute tutorial on a particular technique.

Meeting: TBD

Date	Topics & Readings
Week
1/12	Introduction, Plan logistics Homework: find good/bad study examples
1/19	Past study critique : Sean, Elin
1/26	Past study critique : Britton, Alex
2/2	Past study critique : Lin, Sean
2/9	Past study critique : Elin, Britton
2/16	All: present preliminary project ideas (5 min - purpose, hypotheses, basic design, measures)
2/23	Past study critique : Alex, Lin
3/2	Five-page project proposals due. Present & discuss: all.
	SPRING BREAK
3/16	Advanced study design & analysis: Alex Project proposal review.
3/23	Advanced study design & analysis: Britton
3/30	Advanced study design & analysis: Elin
4/6	Advanced study design & analysis: Lin
4/13	Advanced study design & analysis: Sean
4/20	Wrapup. Final discussion of projects.

Grades will be based on the following:

Quizzes (10%).
Class participation (10%), including presentations.
Individual homework (20% divided equally among assignments).
Midterm exam (15%).
Final exam (15%).
Individual Term Project (30% study design & IRB application).

Homework alternatives to IS4800 assignments (deadlines remain the same):

Individual Homework #2 – Research Plan Review & Library Research: Draft Proposal Section B (Background & Significance) related to your area of interest (2-3 pages).
Individual Homework #3 – Ethics in Dealing with Human Subjects: Draft the Human Subjects and Data Safety and Monitoring Plan Sections related to your area of interest (2-3 pages). This will require you to make a lot of assumptions about the study you might do. Document these assumptions in a short paragraph. Also, sometime soon you will be required to to take the NIH human subjects training course (before you can submit your own protocol to the NU IRB, or be added to an existing IRB). If you have not already done this, this would be an excellant time to get it out of the way. Takes about 2-3 hours. Please email me a screen shot of your completion certificate.
Individual Homework #4 – Ethnography: Do the 4800 exercise (brief 1-hour observational study) but in a context related to your research area of interest.
Individual Homework #5 – Designing Research Models: Design 3 different research models for studies related to your area of interest. For each, include a box-and-arrow diagram and text describing the purpose of the study, the measures you plan to use, and what the results would be useful for. For any experimental studies, identify a possible confounding variable and how you would control it. 1-2 pages per design.
Individual Homework #6 – Objective Measures & Descriptive Statistics in R: just do the 4800 exercise.
Individual Homework #7 – Usability Performance Test: just do the 4800 exercise, but better if you can relate it to your area of interest.
Individual Homework #8 – Behavioral Measures : do the 4800 exercise in pairs.
Individual Homework #9 – Designing a Composite Measure for a Questionnaire: do the 4800 exercise, but make the subject matter of the survey related to your area of interest.
Individual Homework #10 – Chi^2 and Correlation Testing in R: interpret the 4800 exercise as "find a meaningful excuse to use chi-square goodness of fit and pearson correlation stats on your (appropriate) survey data".
Individual Homework #11 – Designing and Analyzing an Experiment: Your five-page project proposals are due at this time, so I'll take those in lieu of this homework.

Resources:

Suggested papers for review:

[Good example / Modeling] Paul M. Fitts (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, volume 47, number 6, June 1954, pp. 381-391. (Reprinted in Journal of Experimental Psychology: General, 121(3):262--269, 1992). [PDF]
[Good example(?) / Modeling] Nielsen, J. and Landauer, T. K. 1993. A mathematical model of the finding of usability problems. In Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems (Amsterdam, The Netherlands, April 24 - 29, 1993). CHI '93. [PDF]
[Good example(?), Chi-Sq & correlation] Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? Proceedings of the SIGCHI conference on Human factors in computing systems [ACM]
[Bad example(?), uses several unusual stats] Aesthetics and apparent usability: empirically assessing cultural and methodological issues [ACM]
[Bad example] Cassell, J. & Thórisson, K. R. (1999). The Power of a Nod and a Glance: Envelope vs. Emotional Feedback in Animated Conversational Agents. International Journal of Applied Artificial Intelligence, 13(4-5): 519-538. [PDF]
[Bad example] Tim Bickmore, Justine Cassell "Social Dialogue with Embodied Conversational Agents" In J. van Kuppevelt, L. Dybkjaer, and N. Bernsen (eds.), Natural, Intelligent and Effective Interaction with Multimodal Dialogue Systems. New York: Kluwer Academic. [PDF]
[Good example, meta-analysis] Yee, N., Bailenson, J. N., Rickertsen, K. (2007). A meta-analysis of the impact of the inclusion and realism of human-like faces on user experiences in interfaces. Nominated for Best Paper Award in Proceedings of the Conference on Computer-Human Interaction (CHI). April 28 - May 3, California, USA [PDF]
[Good example?, ANOVA, some advanced methods] Okita, S.Y., Bailenson, J., Schwartz, D. L. (2007). The mere belief of social interaction improves learning, In Proceedings of the Twenty-ninth Meeting of the Cognitive Science Society [PDF]
[Good example, ANOVA] Rickenberg, R. and Reeves, B. 2000. The effects of animated characters on anxiety, task performance, and evaluations of user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (The Hague, The Netherlands, April 01 - 06, 2000). CHI '00. [ACM]
Communication studies by Cliff Nass, Jeremy Bailenson and their students at Stanford are usually great examples, as are most recent long CHI papers.
The Journal of Contemporary Clinical Trials has many examples of study designs, published prior to study completion.

Suggested papers for Advanced Design discussions:

Baron, R. and Kenny, D., The Moderator-Mediator Variable Distinction in Social Psychological Research: Conceptual, Strategic, and Statistical Considerations, Journal of Personality and Social Psychology 51, 6 (1986) 1173-1182 (Tim has)
Rosenthal and DiMatteo, Meta-Analysis: Recent Developments in Quantitative Methods in Literature Reviews, Annual Review of Psychology 52 (2001) 59-82.
Longitudinal Data Analysis:
- Fitmaurice, Laird & Ware, "Applied Longitudinal Analysis" (Tim has book)
- Zeger & Liang, "Longitudinal data anlaysis for discrete and continuos outcomes", Biometrics, 1986, 42:121-30.
- Hardin & Hilbe, "Generalized estimating equations", Chapman & Hall, 2003.
- Davis, "Program for regression analysis of repeated measures using generalized estimating equations", Comput Methods Programs Biomed, 1993, 40:15-31.
Gottman & Roy, "Sequential analysis: a guide for behavioral researchers" (Tim has book)
Kenny, Kashy & Cook, "Dyadic Data Analysis" (Tim has book)
Causal Path Analysis aka Structural Equation Modeling
- Bollen, 1989, "Structural equations with latent variables", Wiley
- Bollen & Long, 1993, "Testing structural equation models", Sage
- "Identification in Structural Equation Modeling" intro in Chapter 6 "Data Analysis in Social Psychology", "The Handbook of Social Psychology" (Tim has book)
Multilevel data analysis
- Intro in Chapter 6 "Data Analysis in Social Psychology", "The Handbook of Social Psychology" (Tim has book)
- Hardin & Hilbe, "Generalized estimating equations", Chapman & Hall, 2003.
Logistic regression(?)