Software (Usability Studies)
GitHub recently redesigned its Pull Request feature, aiming to improve efficient collaboration and accessibility. I tested the usability of the new Pull Request feature regarding comments, discussions, and merging requests to facilitate code modifications, and provided design recommendations .
The Alpha feature is expected to be released in July 2024, so the findings & UI cannot be revealed as part of this case study. Please contact me or check back in August for the complete writeup.
GitHub is a web-based platform that provides version control and collaboration features for software development projects. A GitHub pull request allows proposing changes to a repository and collaborating with others by asking them to review and merge changes.
Github wanted to see how effective their new UI would be and if it had any usability issues regarding accessibility.
I worked in a team of five students from the University of Washington to collaborate with Github's Pull Request team and UX Researcher.
The Primary Objectives of this Study were to…
Identify success measures of the new pull request feature's user interface (UI) compared to the previous UI.
Identify usability issues in locating and resolving comments as well as merging pull requests.
Assess the intuitiveness and mergeability of the new UI for existing users.
We conducted a remote moderated usability test involving participants interacting with both the old and new user interfaces to compare participants' experiences between both UIs effectively.
We established four testing environments to accommodate the two user interfaces and two distinct user roles: Code Author and Code Reviewer. The varied order of exposure to the UIs helped mitigate potential biases, such as increased preference for the first interface encountered.
The main portion, which was the usability tasks, involved interacting with Pull-request comments and code suggestions, viewing files changed, and merging Pull-requests. After testing each environment, I followed up with a short questionnaire (SUS, adjectives) and interview to delve into their personal thoughts on the intuitiveness and mergeability of the new features.
We collected both qualitative and quantitative data to measure various success rates and gather insights.
The tasks that presented the most difficulty to participants were finding and resolving comments and suggestions.
Overall, participants were able to complete the tasks on both interfaces and had overall positive compliments of the appearance and usability of the new UI. However, as expected, participants had a learning curve to the newer interface, and some bias towards the old interface due to familiarity. Ultimately, the old UI had a significantly higher system usability scale (SUS) score than the new UI.
Please contact me for further details!
Collaborating with Github was an incredible experience ☺
After our study, we presented our findings to the Github Pull Request team that is currently working on improvements of the Alpha feature. We received positive feedback that our findings were very helpful, and shed light on accessibility issues.
Through this collaborative study, I learned a lot in designing research studies, staying flexible and proactive during setbacks, and discussing with stakeholders to align study goals and product development.
Areas for improvements in our study…
Try testing exclusively with the New UI. This would have provided additional time for interview questions, allowing for more in-depth qualitative data collection.
Implement stricter participant screening methods to ensure the study is conducted with genuine participants, avoiding spending time with individuals solely motivated by financial gain.participant screening methods to ensure the study is conducted with genuine participants, avoiding spending time with individuals solely motivated by financial gain.
