Research methods Experimental design 16.422 Human Supervisory Control April 2004
Research Methods & Experimental Design 16.422 Human Supervisory Control April 2004
Research methods Qualitative vs. quantitative Understanding the relationship between objectives(research question and variables is critical Information≠Data Information =data analysis a Planning in advance is a must To include how data will be analyzed
Research Methods Qualitative vs. quantitative Understanding the relationship between objectives (research question) and variables is critical Information ≠ Data Information=data + analysis Planning in advance is a must To include how data will be analyzed
Qualitative Research Methods Social cultural phenomenon Case studies Focus groups Observations Usability testing Can be quantitative Interviews Questionnaires
Qualitative Research Methods Social & cultural phenomenon Case studies Focus groups Observations Usability testing Can be quantitative Interviews Questionnaires
Quantitative Research Methods Natural phenomenon Mathematical modeling Experiments a optimization Game theory a Surveys a Bottom line statistics are a must
Quantitative Research Methods Natural phenomenon Mathematical modeling Experiments Optimization Game theory Surveys Bottom line – statistics are a must
Project Assignment Design and conduct an experiment in which you explore some measure of human performance through testing, analyze the results, and discuss the broader implications Design an actual display that uses automation for decision support.. While formal experimental testing is not required, a small group of users should be used to identify problems with the design to include functionality evaluation as well as recommendation for future improvements and systems integration
Project Assignment Design and conduct an experiment in which you explore some measure of human performance through testing, analyze the results, and discuss the broader implications. Design an actual display that uses automation for decision support… While formal experimental testing is not required, a small group of users should be used to identify problems with the design to include functionality evaluation as well as recommendation for future improvements and systems integration
The Experimental design Process Research Question (Hypothesis) Design Experiment Collect data Analyze data Draw Conclusions
The Experimental Design Process Research Question (Hypothesis) Design Experiment Collect Data Analyze Data Draw Conclusions
Experimental Design Design of Experiments (doe defined a theory concerning the minimum number of experiments necessary to develop an empirica/model of a research question and a methodology for setting up the necessary experiments a parsimony model Human subject VS object experimentation other doe constraints Time Money
Experimental Design Design of Experiments (DOE) defined: A theory concerning the minimum number of experiments necessary to develop an empirical model of a research question and a methodology for setting up the necessary experiments. A parsimony model Human subject vs. object experimentation Other DOE Constraints Time Money
Experimental Design Basics TWo kinds of data gathering methodologies observation Cant prove cause effect but can establish associations Hawthorne effect social facilitation Experimental Cause effect Variables of interest- factors vs treatments Independent variable Treatment manipulations of variables of interest Treatment VS control group Dependent variable is what you are measuring
Experimental Design Basics Two kinds of data gathering methodologies Observation Can’t prove cause & effect but can establish associations. Hawthorne effect, social facilitation Experimental Cause & effect Variables of interest – factors vs. treatments Independent variable Treatment – manipulations of variables of interest Treatment vs. control group Dependent variable is what you are measuring
More basics Confounds Randomization concerns Randomization prevents experimental bias Assignment by experimenter Counterbalancing Statistical assumptions a requirement for statistical tests of significance Why would you use the observation methodology instead of experiments?
More Basics Confounds Randomization Concerns Randomization prevents experimental bias Assignment by experimenter Counterbalancing Statistical assumptions A requirement for statistical tests of significance Why would you use the observation methodology instead of experiments?
DOE Terminology Replications Independent observations of a single treatment. a Variance The measuring stick that compares different treatments Internal validity The extent to which an experiment accomplishes its goals) Reproducibility Given the appropriate information, the ability of others to replicate the experiment
DOE Terminology Replications Independent observations of a single treatment. Variance The measuring stick that compares different treatments. Internal validity The extent to which an experiment accomplishes its goal(s). Reproducibility Given the appropriate information, the ability of others to replicate the experiment
