IS4800 - Empirical Research Methods in Information Science
[Syllabus] [Schedule]
[Homework] [Projects]
[Bibliography] [Resources]
[Directory] [Grads]
Graduate Empirical Research Methods Course (CS6350
& CS6351)
Extra requirements for graduate students:
In addition to the undergraduate course requirements, graduate students must
each design one additional publication-quality empirical research study and
attend one extra one-hour seminar meeting per week during the semester. The
study must be on a topic and use methods approved by the instructor. Graduate
students must submit a seven-page proposal on the study and analysis techniques
to be used, along with full application materials for Northeastern IRB approval by the
end of the semester. In addition to project presentations, the seminar meetings
will be used to critique study designs from the current HCI and health
informatics literature.
Project proposals: Preliminary 5-page study proposals must be submitted
prior to spring break. Final study proposals must be submitted by the end of the semester following the NIH R21
format (1 page Specific Aims plus 6 page Research Strategy sections only,
with the latter organized into Significance, Innovation, and Approach
sections).
Homework: The homework in the table below is in lieu
of the IS4800 homeworks, and you do not need to do
the IS4800 team projects.
IRB application & Study protocol: By the end of
the semester, you must turn in a full Northeastern IRB application
to the instructor, including any consent forms and other materials the IRB
would require. You must also turn in a full study protocol, which includes
step-by-step instructions for running the study that would enable a layperson
to run the study.
Seminar Schedule: We will spend the first half
of the semester reviewing and critiquing studies from the HCI and health
informatics literature. During each of these classes three students will
each present the work and their critique (10 minutes each). Thereafter, all students
will discuss and critique it. Following the midterm, students will present advanced
study designs and statistical analysis techniques, with two students assigned
to each give a 20 minute
tutorial on a particular technique. Students assigned to present
(critiques ro tutorials) must obtain approval from the instructor on
the topic and paper to present, and send the paper to the class (cs6350@ccs.neu.edu) at least 3 days before class so all can read it.
Seminar Meeting (CS6351): Mondays 4:35-5:35, BK007
Date
|
Topics & Readings
|
Week
|
|
|
1/8
|
Introduction, Plan logistics
Homework: find good/bad study examples to present. Start
thinking about project topic (do not turn in).
|
|
1/15
|
HOLIDAY: NO CS6350 MEET
Homework: Complete the NIH human subjects
training course and send me your completion certificate (due 1/19).
|
|
1/22
|
Past study critique: Teresa, Mengyue
Homework: Do the 4800 exercise I2
(brief 1-hour observational study) but in a context related to your research
area of interest (due 1/30).
|
|
1/29
|
Past study critique: Kelsey, Janice
Homework: Do the 4800 exercise I3
(usability test), but ideally on a system related to your interest. You can
work in pairs. (due 2/6)
|
|
2/5
|
Past study critique: Bhargavi, Yash
Homework: Do the 4800 exercise I4 (questionnaire) on a
topic related to your interest. You can work in pairs. (due 2/16)
|
|
2/12
|
All: Present preliminary project ideas (5 min - purpose, hypotheses,
basic design, measures)
Homework: Draft Specific Aims for your proposal (1
page). (due 2/19)
|
|
2/19
|
HOLIDAY: NO CS6350 MEET
Homework: Draft the Significance section of your project
proposal (1-2 pages). (due 2/26)
|
|
2/26
|
Five-page project proposals due (3/2)
Present & discuss: all.
|
|
|
SPRING BREAK
|
|
3/12
|
Past study critiques: 1) Zhe; 2) Akash
|
|
3/19
|
Advanced study design &
analysis: 1) Ryan ; 2) Mansi
|
|
3/26
|
Advanced study design &
analysis: 1) Shruthi ; 2) Teresa & Kelsey
|
|
4/2
|
Advanced study design &
analysis: 1) Nathan; 2) Mengyue
|
|
4/9
|
Wrapup. Final discussion of
projects.
|
|
4/16
|
NO SEMINAR MEET
4/13 Final project presentations: Teresa, Mengyue, Kelsey, Janice, Ryan, Shruthi
4/17 Final project presentations: Bhargavi, Yash, Zhe, Akash, Mansi, Nathan
|
Grades will be based on the following:
- Quizzes
(10%).
- Class
participation (10%), including presentations.
- Individual
homework (20% divided equally among assignments).
- Midterm
exam (15%).
- Final
exam (15%).
- Individual
Term Project (30% study proposal & IRB application).
- Proposal - "A" is recommend for funding as NIH R21 application.
- IRB - "A" is if the Northeastern IRB would approve without modification.
- Study protocol - "A" is if a freshman could run the study without any additional information.
Resources:
Some
suggested papers for review (but better for you to find your own related to
your research interests):
- Most recent long
CHI papers are generally good examples.
- [Good example / Modeling]
Paul M. Fitts (1954). The information capacity of
the human motor system in controlling the amplitude of movement. Journal
of Experimental Psychology, volume 47, number 6, June 1954, pp. 381-391.
(Reprinted in Journal of Experimental Psychology: General, 121(3):262--269, 1992). [PDF]
- [Good example / Modeling]
Nielsen, J. and Landauer, T. K. 1993. A
mathematical model of the finding of usability problems. In Proceedings
of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing
Systems (Amsterdam, The Netherlands, April 24 - 29, 1993). CHI '93. [PDF]
- [Good example Chi-Sq & correlation] Measuring usability: are effectiveness, efficiency, and satisfaction really
correlated? Proceedings of the SIGCHI conference on Human factors in
computing systems [ACM]
- [Bad example, uses several
unusual stats] Aesthetics and apparent usability: empirically assessing
cultural and methodological issues [ACM]
- [Bad example] Cassell, J. & Thorisson,
K. R. (1999). The Power of a Nod and a Glance: Envelope vs. Emotional
Feedback in Animated Conversational Agents. International Journal of
Applied Artificial Intelligence, 13(4-5): 519-538. [PDF]
- [Bad example] Tim Bickmore,
Justine Cassell "Social Dialogue with
Embodied Conversational Agents" In J. van Kuppevelt,
L. Dybkjaer, and N. Bernsen
(eds.), Natural, Intelligent and Effective Interaction with Multimodal
Dialogue Systems. New York: Kluwer Academic. [PDF]
- [Good example, meta-analysis]
Yee, N., Bailenson, J. N., Rickertsen,
K. (2007). A meta-analysis of the impact of the inclusion and realism of
human-like faces on user experiences in interfaces. Nominated for Best
Paper Award in Proceedings of the Conference on Computer-Human Interaction
(CHI). April 28 - May 3, California, USA [PDF]
- [Good example, ANOVA, some
advanced methods] Okita, S.Y., Bailenson, J., Schwartz, D. L. (2007). The mere belief
of social interaction improves learning, In Proceedings of the
Twenty-ninth Meeting of the Cognitive Science Society [PDF]
- [Good example, ANOVA] Rickenberg, R. and Reeves, B. 2000. The effects of
animated characters on anxiety, task performance, and evaluations of user
interfaces. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (The Hague, The Netherlands, April 01 - 06, 2000). CHI
'00. [ACM]
- Communication studies by Cliff
Nass, Jeremy
Bailenson and their students at Stanford are
usually great examples.
- The Journal of
Contemporary Clinical Trials has many examples of study designs,
published prior to study completion.
Some
papers for Advanced Design discussions:
- Baron, R. and Kenny, D., The
Moderator-Mediator Variable Distinction in Social Psychological Research:
Conceptual, Strategic, and Statistical Considerations, Journal of Personality
and Social Psychology 51, 6 (1986) 1173-1182
- Rosenthal and DiMatteo, Meta-Analysis: Recent Developments in
Quantitative Methods in Literature Reviews, Annual Review of Psychology 52
(2001) 59-82.
- Longitudinal Data Analysis:
- Fitmaurice,
Laird & Ware, "Applied Longitudinal Analysis"
- Zeger
& Liang, "Longitudinal data anlaysis
for discrete and continuos outcomes",
Biometrics, 1986, 42:121-30.
- Hardin & Hilbe, "Generalized estimating equations",
Chapman & Hall, 2003.
- Davis, "Program
for regression analysis of repeated measures using generalized estimating
equations", Comput Methods Programs
Biomed, 1993, 40:15-31.
- Time-series Analysis
- Gottman
& Roy, "Sequential analysis: a guide for behavioral
researchers"
- Velicer, W. F., & Fava, J. L. (2003).
Time Series Analysis. In J. Schinka & W. F.
Velicer (Eds.), Research Methods in Psychology(581-606). Volume 2, Handbook of Psychology
(I. B. Weiner, Editor-in- Chief.). New York: John Wiley & Sons.
- Kenny, Kashy
& Cook, "Dyadic Data Analysis"
- Causal Path Analysis aka
Structural Equation Modeling
- Bollen,
1989, "Structural equations with latent variables", Wiley
- Bollen
& Long, 1993, "Testing structural equation models", Sage
- "Identification
in Structural Equation Modeling" intro in Chapter 6 "Data
Analysis in Social Psychology", "The Handbook of Social
Psychology"
- Multilevel data analysis
- Intro in Chapter 6
"Data Analysis in Social Psychology", "The Handbook of
Social Psychology"
- Hardin & Hilbe, "Generalized estimating equations",
Chapman & Hall, 2003.
- Logistic regression
- Peng,
Chao-Ying Joanne, Kuk Lida
Lee, and Gary M. Ingersoll. "An introduction to logistic regression
analysis and reporting." The Journal of Educational Research 96.1
(2002): 3-14.
- Meta-Analysis
- Rosenthal and DiMatteo, Meta-Analysis: Recent Developments in
Quantitative Methods in Literature Reviews, Annual Review of Psychology
52 (2001) 59-82.
- R topics that have been
presented in the past include: