Qualitative Evaluation
Click here to download a PDF version of Assignment 2.
Weight: 25% of your final grade
Due:after Unit 4 in Week 13
This assignment is designed to enhance your ability to perform a comprehensive qualitative evaluation. It will help you build expertise in the area of usability studies using five strategies:
- the silent observer
- think aloud
- constructive interaction
- questionnaire
- interview
You will perform a critical analysis of the usability of a well-known software system such as Microsoft Word, a website such as Amazon.com, or a device such as the iPad. The immediate purpose of this assignment is to give you experience at
- preparing the pre-usability study conditions.
- working with subjects to make them familiar with the purpose of the study and the role that they play.
- conducting and moderating a usability session based on various evaluation strategies.
- analyzing and compiling the results of a usability study, and preparing an informative report for decision makers.
Readings
- Chapter 14 ofInteraction Design, 5th Edition
- Chapter 15 ofInteraction Design, 5th Edition
- Chapter 16 ofInteraction Design, 5th Edition
Deliverables
You will deliver a technical report written to the imaginary vice president of the company that has requested the usability study. Assume that you are a third-party company that has been hired to evaluate the product of this company, and the report you provide will impact the decisions made on the final outcome of the product. Therefore, this report must be detailed and needs to include your observations, the findings, the major problems detected, and some design recommendations for the product.
As mentioned above, the selection of the system to be evaluated is up to you. The only restriction is that the system, be it hardware or software, will need to be widely used, and a wide range of familiar users for it must be found in the general population. If you are not sure whether your system would qualify as such a system, you can discuss it with your instructor.
The report that you provide will need to include the following sections. In total it should be at least 10 pages (~2500 words) and at most 15 pages (~3750 words) long in Times New Roman 12-point font.
Section 1. Scenario
Weight:10% of the assignment mark
Give a very brief reminder to the VP of what the system is and why the system is being evaluated. Clarify the purpose and scope of the study.
Section 2. Methodology
Weight:30% of the assignment mark
Explain in detail the process that you performed in order to complete the usability studies. This will need to include every step of the process that was performed from the pre-evaluation steps, including approaching the subjects through to the details of how the gathered observation data were analyzed. Provide a copy of the questionnaire you used, and justify how you defined the questions.
Section 3: Observations
Weight:20% of the assignment mark
Provide a summary of the observations made through the evaluation process. This needs to be different from your subjective judgment and interpretation of the results.
Section 4: Interpretation
Weight:25% of the assignment mark
Identify the pros and cons of the system, and justify your conclusions using the observations made. This section will include your subjective interpretation of the observations.
Section 5: Suggested Improvements
Weight:10% of the assignment mark
Describe the five required changes that you would make in the product. Back these up with the information provided in Sections 3 and 4.
Section 6: Conclusion
Weight:5% of the assignment mark
Summarize what you found and provide your recommendations.
Appendix 1: Raw Data
The assignment submission is unacceptable without this data.All original observations/recordings, questionnaire details, etc., need to be included at the end of the report.
METHOD
Step 1.Identify the target system of interest that you will evaluate. Create a list of important points: why do you think this system is worth evaluating? Formulate the scope of the usability study, e.g., if you are performing the usability study for Microsoft Word you can define the scope of the study to evaluating the “document reviewing tools” of this office application. Identify further the purpose of the study. The purpose and scope of the study need to be clear so that they can be communicated to both the stakeholders and the study participants.
Step 2.Identify four subjects who are willing to take part in your usability study. If you do not have suitable people who can help you complete the tasks, you can communicate with other students taking this course to form a group. This way you can help each other in completing the usability evaluation process. Clearly explain the purpose of study and its scope for the participants.
Step 3.Think ahead about the details of the usability study and decide on the type of information that you would like to collect. Develop questions for the questionnaire strategy. Create a list of tasks that will be provided to the subjects during the evaluation process. These will be the tasks that the subjects will perform for observation. Think about what details you will observe.
Step 4.Use each of the five following strategies to evaluate the usability of the system. Each of the participants will participate in only one of the first three evaluation strategies, while they will all need to participate in strategies 4 and 5:
The silent observer strategy.The experimenter and the subject who is performing the usability test are not allowed to speak with each other.
Think aloud strategy. This is similar to the silent observer strategy except that the subject is asked to say aloud what she is doing while she is doing it.
Constructive interaction strategy. This will include two subjects working together on performing the required tasks; they will have natural communication with each other. The experimenter will only observe the process and take notes.
Questionnaire strategy. Provide all of the subjects with the usability questionnaire that you have already developed, and ask them to complete it.
Interview strategy. The experimenter should interview all of the subjects about their perception of the system in terms of its benefits, disadvantages, pros, cons, and other weaknesses.
Step 5.Gather all of the information, consider your observations, analyze the collected data, and compile them into a usability analysis report with the structure shown in the Deliverables section.