Moment objectives Experimental psychology fter the moment the participants shall March 22 and 29. 2006 ogical and psychological variables are lEcturer included, which entails that they shall be able to identify different orientations within psychology Bo Schenkman be able to apply parts of experimental psychology to issues of Human- Computer Interaction. oncerning design, selection of analysis and quality, which entails that brief overview of experimental use a number of different concepts in experimental design gy and a background to experimentation in ch as dependent and independent variables, reliability, validity, factor 3. present the outlines of an own experimental study in the field of Human-Computer Interaction Carson, N, Buskist, w.& Martin, G. N. (2000). Psychology: the Science of ehaviour: European Adaptation. Chap I and 2(pp 2-63) Stencil What I shall speak about homson/Wadsworth, 6th ed, Chap 5, 7, 8, 11, 12 and 14 Stencil. Experimental psychology winer, BJ, Brown, D R.& Michaels, K M.(1991). Statistical Principles in Research methods in psychology Experimental Design. 3rd ed. Boston: McGraw-Hill. Chap 1. Introduction to design,pp1-11.Stencil Optional literature Experimental environment Sarle,wS(1996).Measurement theory: Frequently asked questions. URL: Accuracy: Reliability and validity ftp: //ftp. sas. com/pub/neural/measurement. htm Measurement: Scales Skitka L 1.& Sargis, E.G. (2006). The Internet as psychological boratory. Annual Review of Psycholog. Vol. 57, 529-555 Stencil Stemberg, R.(2005)Cognitive Psychology. Thomson/wadsworth, 4th ed Experiments in daily life Experimental psychology Research methods in psychology Experimental designs Experimental environment ccuracy: Reliability and validity Measurement: Scales
1 Experimental psychology March 22 and 29, 2006 Lecturer: Bo Schenkman Email: bosch@nada.kth.se Purpose: To give a brief overview of experimental psychology and a background to experimentation in HCI Moment objectives After the moment the participants shall 1. be able to design a basic experiment in the field of Human-Computer Interaction, where both technological and psychological variables are included,which entails that they shall be able to identify different orientations within psychology be able to apply parts of experimental psychology to issues of HumanComputer Interaction. 2. be able to analyse studies within Human-Computer Interaction concerning design, selection of analysis and quality, which entails that they shall, be able to use a number of different concepts in experimental design such as dependent and independent variables, reliability, validity, factor and control. 3. present the outlines of an own experimental study in the field of Human-Computer Interaction. Mandatory literature Carson, N., Buskist, W. & Martin, G. N. (2000). Psychology: the Science of Behaviour: European Adaptation. Chap 1 and 2 (pp 2-63). Stencil. McBurney, D.H. & White, T.L. (2004). Research methods. Thomson/Wadsworth, 6th ed. Chap 5, 7, 8, 11, 12 and 14 Stencil. Winer, B.J, Brown, D.R. & Michaels, K.M. (1991). Statistical Principles in Experimental Design. 3rd ed. Boston: McGraw-Hill. Chap 1. Introduction to design, pp. 1-11. Stencil. Optional literature Sarle, W.S. (1996). Measurement theory: Frequently asked questions. URL: ftp://ftp.sas.com/pub/neural/measurement.html Skitka; L. J. & Sargis, E.G. (2006). The Internet as psychological laboratory. Annual Review of Psychology. Vol. 57, 529-555. Stencil. Sternberg, R.J. (2005) Cognitive Psychology. Thomson/Wadsworth, 4th ed. Chap 1, pp 1 – 27. Stencil. What I shall speak about Experimentation Experimental psychology Research methods in psychology Experimental designs Experimental environment Accuracy: Reliability and Validity Measurement: Scales Experimentation Experimental psychology Research methods in psychology Experimental designs Experimental environment Accuracy: Reliability and Validity Measurement: Scales Experiments in daily life
Philosophy of science Determining causality relations Earlier: focus on theories, e.g. Hempel, Popper and Kuhn Based on common sense absence of method e. g Feyerabend We all do experiments all the time Today: emphasis on the role of experiment, e.g. Deborah Keeping some conditions controlled Is especially for scientific knowledge and its growth Essential for the scientific knowledge cf only theory based data Statistics follows logically, not an end in itself Ronald Fisher Requirements of a good experiment of m Calculation of" insecurity C make a clear and convincing test of a hypothesis created the foundations for modern statistical science (McBurney White) Some psychological pioneers psycholog gn Experimental environment ccuracy: Reliability and validity Measurement: Scales William James Sigmund Freud James Gibson B.F. Skinne
2 Experimentation Determining causality relations Based on common sense We all do experiments all the time Keeping some conditions controlled Verifying the results Essential for the scientific knowledge cf. only theory based data Statistics follows logically, not an end in itself Philosophy of science Earlier: focus on theories, e.g. Hempel, Popper and Kuhn or absence of method, e.g Feyerabend. Today: emphasis on the role of experiment, e.g. Deborah Mayo - especially for scientific knowledge and its growth Ronald Fisher “…created the foundations for modern statistical science." Requirements of a good experiment Absence of systematic errors Precision “Validity” – the capacity of making generalization Simplicity Calculation of “insecurity” (Cox) The elegant experiment : the simplest experiment that will make a clear and convincing test of a hypothesis (McBurney & White) Experimentation Experimental psychology Research methods in psychology Experimental design Experimental environment Accuracy: Reliability and Validity Measurement: Scales Some psychological pioneers William James Sigmund Freud James Gibson B.F. Skinner
Notions and relations of psychology Image colour phenomenon-induction Human Factors and ergonomics Experimental vs non-experimental psychology Divisions Aerospace Systems Human Factors and Society for General pschology Ergonomics Society etiseo engineering and Internet 5. Evaluation, Measurement, and Statistics 6. Behavioral Neuroscience and Comparative Computer Systems Perception and Performance Product Design Developmental Psychology 17. Society of Counseling Psychology Safety 19. Society for Military Psychology Forensics Professional for the Psychological Study of Social 20. Adult Development and Aging (SPSSI) System Development 21. Applied Experimental and Engineering Society for the Psychology of Aesthetics, Psychology Human Performance Modeling Test and evaluation Creativity and the Arts 12. Socety of Clinical Psychology 55. American Society for the Advancem Virtual Environments 13. Society of Consulting Psychology Object of study Scientific perspectives in modern psycholo Technical system or psychological processes? Psychodynamic Difference Human Factors- Psychology Cognitive Determines choice of and analysis of variable Socio cultural
3 Notions and relations of psychology Theoretical psychology Applied psychology Human Factors and Ergonomics Human Computer Interaction Experimental vs non-experimental psychology Clinical psychology Image colour phenomenon- induction Divisions 1. Society for General Psychology 2. Society for the Teaching of Psychology 3. Experimental Psychology 5. Evaluation, Measurement, and Statistics 6. Behavioral Neuroscience and Comparative Psychology 7. Developmental Psychology 8. Society for Personality and Social Psychology 9. Society for the Psychological Study of Social Issues (SPSSI) 10. Society for the Psychology of Aesthetics, Creativity and the Arts 12. Society of Clinical Psychology 13. Society of Consulting Psychology 14. Society for Industrial and Organizational Psychology 15. Educational Psychology 16. School Psychology 17. Society of Counseling Psychology 18. Psychologists in Public Service 19. Society for Military Psychology 20. Adult Development and Aging 21. Applied Experimental and Engineering Psychology 22………….. 55. American Society for the Advancement of Pharmacotherapy Aerospace Systems Aging Cognitive Engineering and Decision Making Communications Computer Systems Education Environmental Design Forensics Professional Health Care Human Performance Modeling Individual Differences in Performance Industrial Ergonomics Internet Macroergonomics Perception and Performance Product Design Safety Surface Transportation System Development Test and Evaluation Training Virtual Environments Object of study Technical system or psychological processes? Difference Human Factors - Psychology Determines choice of and analysis of variable Scientific perspectives in modern psychology Biological Psychodynamic Behaviouristic Cognitive Humanist Evolutionary Socio cultural
Pavlov and dog Freuds coach. now in the Freud museum. London Functional Magnetic Resonance Imaging(fMRI) HMI: Car simulator Research methods in modern psychology ital psychology Controlled laboratory experiment ychobiological research; e. g EEG, fMRI esearch methods in psychology Self-reports, verbal protocols, self-ratings, diaries Naturalistic observations; e. g in home, at work ccuracy: Reliability and validity Computer simulations Measurement: Scales Each has its advantages and disadva 4
4 Pavlov and dog Skinner box Freud´s coach, now in the Freud museum, London Functional Magnetic Resonance Imaging (fMRI) 2.6 m Night vision camera projector Test driver Screen Screen Screen Ambient world projectors HMI: Car simulator Experimentation Experimental psychology Research methods in psychology Experimental design Experimental environment Accuracy: Reliability and Validity Measurement: Scales Research methods in modern psychology Controlled laboratory experiment Psychobiological research; e.g EEG, fMRI Self-reports; verbal protocols, self-ratings, diaries Case studies Naturalistic observations; e.g in home, at work Computer simulations Each has its advantages and disadvantages
Chosing a research method in psychology Experimentation Experimental psychology xperimental control of independent variables Sample size Research methods in psychology Experimental design Ecological validity Experimental environment Information about individual differences Accuracy: Reliability and Validity Measurement: Scales Experimental Design xperimental sources oferror Design is the planning of the generation of data, Premeasurement error, e.g nervousness ction error, e.g. sensitivity chang Maturation, e.g. hunger, tiredness the aim is to obtain correct information with History, e.g. political, social change: Instrumentation acceptable accuracy, at a cost that will not supercede the value of the information Selection of test Crucial: error reduction and control Measurement occasion wrong De Treatment Dependent variables Supplementary(nuisance)variables Condition tation: ability to con a systematic manner In HCI: Which are the dependent and which
5 Chosing a research method in psychology Random assignment of subjects Experimental control of independent variables Sample size Sample representativeness Ecological validity Information about individual differences Strengths Weaknesses Experimentation Experimental psychology Research methods in psychology Experimental design Experimental environment Accuracy: Reliability and Validity Measurement: Scales Experimental Design Design is the planning of the generation of data, where the aim is to obtain correct information with acceptable accuracy, at a cost that will not supercede the value of the information. Crucial: error reduction and control Experimental sources of error Premeasurement error, e.g. nervousness Interaction error, e.g. sensitivity change Maturation, e.g. hunger, tiredness History, e.g. political, social changes Instrumentation Selection of test persons Mortality – missing cases Reactive error, e.g. artificiality of test Measurement occasion wrong Surrogate situation Design terminology Many from agriculture: Treatment Plot Condition Level Block Variables Independent variables Dependent variables Supplementary (nuisance) variables Experimentation: ability to control/manipulate variables in a systematic manner In HCI: Which are the dependent and which are the independent variables?
means to rule out threats to the validity of research eneral rules for a god Experimental and control group 2. Randomization Subjects as their own control Randomization and random assignment Matching Test procedure designs 1B=pretest, X=treatment, MA=posttest, R=randomization Allocation of subjects-distinction fter onh Y MA tween-subjects design: each subject does only get one Before-After MB X MA f the experimental conditions Before-After with a control R MB,X MA requirement: random allocation of subjects to groups R MB,X MA Simulated Before-After RRRR X MA Within-subjects design: each subject gets every ly with a control Y Md experimental condition Solomon four-group R MB X MA order effects, position of condition affecting result R MB,X MA X MA sequence effects, condition affecting other conditions X MA Completely Randomized design(Between-subjects design Statistical designs b4 1. Completely Randomized (Between-subjects design 3. Latin Square S39 S10|S20 6
6 Control Two general rules for a good experiment: 1. Control conditions or control group 2. Randomization Control Experimental and control group Between vs Within-subjects design Subjects as their own control Randomization and random assignment Matching Statistical control ”means to rule out threats to the validity of research” Test procedure designs MB= pretest, X=treatment, MA=posttest, R=randomization After only X MA Before - After MB X MA Before - After with a control R MB1 X MA1 R MB2 X MA2 Simulated Before - After R MB R X MA After - only with a control R X MA1 R MA2 Solomon four - group R MB1 X MA1 R MB2 X MA2 R X MA3 R X MA4 Allocation of subjects - distinction Between-subjects design: each subject does only get one of the experimental conditions requirement: random allocation of subjects to groups Within-subjects design: each subject gets every experimental condition risks: order effects; position of condition affecting result sequence effects; condition affecting other conditions Statistical designs 1. Completely Randomized (Between-subjects design) 2. Randomized Block (Within-subjects design) 3. Latin Square Completely Randomized design (Between-subjects design) S31 S39 S40 S11 ….. S20 S1 S2 S3 S10 b1 b2 b3 b4 CR-k CR-4 Treatment levels
Randomized block design (within-subjects design) Latin Square design RB-k LS-K RB-4 Treatment levels Nuisance variable 1:columns Nuisance variable 2: rows 105 Latin Square design: Example Factorial designs More than one factor is investigated at the same time b4b3 b2 Joint effects of variables is studied Interaction estimates car al B C DA Can become too complex and time-consumir a2 C DAB levels=4 gum mixtur Time saving. but no estimate of interactions Completely Randomized Factorial design( Between-subjeets Randomized Block Factorial design (Within-subjects -p4 SI SI SI S4 S5S6 7
7 Randomized Block design (Within-subjects design) 4 4 2 3 6 1 3 5 5 S1 S2 …. S10 Subjects b1 b2 b3 b4 Treatment levels RB-k RB-4 Latin Square design LS-k C A B B C A A B C D A B C C D A B B C D A A B C D LS-3 LS-4 Nuisance variable 1: columns Nuisance variable 2: rows Latin Square design: Example a4 a2 a1 a3 A B C D D A B C C D A B B C D A b4 b3 b1 b2 wheel position car a: no rows = 4 cars b: no columns = 4 wheel positions c: treatment levels = 4 gum mixtures Time saving, but no estimate of interactions Factorial designs More than one factor is investigated at the same time Joint effects of variables is studied Interaction estimates Often more efficient than simple-factor studies Can become too complex and time-consuming Completely Randomized Factorial design (Between-subjects) CRF-pq CRF-pqr CRF-23 S4 S5 S6 S1 S2 S3 b1 b2 b3 a1 a2 S5 S6 S7 S8 S1 S2 S3 S4 CRF-222 b1 b1 b2 b2 c1 c2 c1 c2 a1 a2 Randomized Block Factorial design (Within-subjects) b1 b2 b3 a1 a2 S1 S1 S1 S1 S1 S1 RBF-pq RBF-23
Split-Plot (Factorial)design Mixed desig bI al Within-subjects desi No interaction when the effect of one independent variable depends on the level of another independent Interactions
8 Split-Plot (Factorial) design = Mixed design b1 b2 b3 a1 a2 S2 S2 S2 S1 S1 S1 RBF-pq RBF-23 Within-subjects design Between-subjects design Hierarchical designs b1 b2 b3 b4 a1 a2 Treatments Crossed Nested a1 a2 b1 b2 b3 b4 Interaction: “when the effect of one independent variable depends on the level of another independent variable” Interactions No interactions http://www.upa.pdx.edu/IOA/newsom/pa551/lectur12.htm Interactions Example of interactions 200 m 100 m 50 m 25 m Camera on r oof Scale: 1:1 1:2 0 2 4 6 8 10 12 14 16 TTC (s) Camera on g rill Scale: 1:1 1:2
Single case experiments Quasi experiments One subject is studied, often n=l ue experiments: the scientist has complete control over If more than one subject, each is analysed separately all aspects of the experiment, i.e the who, what, when quasi experiments: "the scientist must select subjects Suitable e.g. for medicine, psychophysics and others for different conditions from preexisting groups HCI sometimes fully adequate not less scientific One sub class: meta analysis Experimentation Experimental psychology Laboratory experiments control Research methods in psychology reactive sources of erro Experimental designs Field studies Experimental environment realism Accuracy: Reliability and validity ow control measurement: Scales Human Computer Interaction Experimentation Experimental psychology Research methods in psychology perimental environment ccuracy:Reliability and Validity
9 Single case experiments One subject is studied, often n=1 If more than one subject, each is analysed separately Case studies Suitable e.g. for medicine, psychophysics and others. HCI? Quasi experiments true experiments: the scientist has complete control over all aspects of the experiment, i.e the who, what, when, where and how. quasi experiments : “the scientist must select subjects for different conditions from preexisting groups” • sometimes fully adequate • not less scientific One sub class: meta analysis Experimentation Experimental psychology Research methods in psychology Experimental designs Experimental environment Accuracy: Reliability and Validity Measurement: Scales Experimental environment Laboratory experiments control reactive sources of error Field studies realism low control Surveys Simulations Human Computer Interaction Usability testing at OCLC CS, Southern Illinois University Edwardsville Experimentation Experimental psychology Research methods in psychology Experimental designs Experimental environment Accuracy: Reliability and Validity Measurement: Scales
A cy of Components of the measurement value L R leasure = True value plus 2. Validity measurement error Longtime stabile properties hort time properties ation properties Measuring process Measuring instrument Rorschach-test Reliability -Validity ⊙ Reliability Validity Test-retest reliability same test results on two occasions 1. Construct validity: if a test measures the theoretical instruct it is assumed to test 2. Alternate Forms 2. Face validity: if a test appears to measure what it supposed to test ty: if a test relates to other measures of the 3. Split Half reliability 4. Inter-rater Reliability 4. Convergent validity Intermal consistency 5. Discriminant validity 10
10 Accuracy of the measure 1. Reliability 2. Validity Components of the measurement value Measure = True value plus measurement error: Longtime stabile properties Short time properties Situation properties Measuring process Measuring instrument Analysis errors Rorschach-test Reliability - Validity http://www.socialresearchmethods.net/kb/rel&val.htm Reliability 1. Test-retest reliability: same test results on two occasions 2. Alternate Forms 3. Split Half reliability 4. Inter-rater Reliability 5. Internal consistency Validity 1. Construct validity: if a test measures the theoretical construct it is assumed to test 2. Face validity : if a test appears to measure what it is supposed to test 3. Criterion validity: if a test relates to other measures of the same concept 4. Convergent validity 5. Discriminant validity