Quiz
Once participants in all conditions complete their respective tasks and the timer finishes, participants should continue to the next page and complete the five-question quiz on the topic they researched to the best of their ability based on the results or outputs they produced.
Post-Survey Procedures
After the quiz, Condition 2-4 will be asked to paste the first three searches or prompts they used to understand the quiz topic better
Participant will be asked if more than three searches were used, and the total number of searches used if there were more than three
After all conditions, participants will be presented with a post-quiz survey that asks questions to measure their feelings of engagement, focus, and enjoyment with the study
Results
Innovativeness & Technology Avoidance Scores
Control: 3.86
Google Search Group: 3.79
ChatGPT w/o prompt-coaching: 3.70
ChatGPT w prompt-coaching: 3.91
Quiz Scores
Control: 2.86
Google Search Group: 2.91
ChatGPT w/o prompt-coaching: 3.55
ChatGPT w prompt-coaching: 3.77
Average: 3.25
Statistical Significance in Quiz Scores
ChatGPT w prompt-coaching & control: p = 0.032
ChatGPT w prompt-coaching & Google Search group: p = 0.106
ChatGPT w prompt-coaching & w/o: p = .930
Average: 3.25
Internet Self-Efficacy Scores
Control: 5.28
Google Search Group: 5.21
ChatGPT w/o prompt-coaching: 5.06
ChatGPT w prompt-coaching: 5.70
Discussion
What is Searching For Refinement?
Overview
My Role
Searching For Refinement is a research study about determining whether research using artificial intelligence technology can improve the user’s learning and experience in comparison to traditional search engines, such as Google. Researchers want to understand if guided-searching using artificial intelligence technology, such as ChatGPT, can lead to more refined results.
I am the main researcher for my UCR University Honors Capstone Project, Searching For Refinement, with the assistance of my faculty advisor, Professor Rich Yueh. My role is to conduct UX research through this study by developing an IRB-approved survey and quiz backed up by artificial intelligence technology research for participants, and analyze the participant data. After, I wrote a literature review with a report of my findings and results.
Awarded Mini-Grants for Undergraduate Research & Creative Activities
What will my participants do during the study?
Depending on the randomly generated condition they are placed in, participation in this study will involve following a series of instructions that may ask them to:
Respond to Questionnaires
Complete a training module
Take a scored quiz
Research a presented topic
Engage in a game activity
Hypotheses
H1a: ChatGPT users will perform better on the quiz than the control group.
H1b: ChatGPT users will perform better on the quiz than the Google-only users.
H1c: ChatGPT users with prompt coaching will perform better on the quiz than ChatGPT users without prompt coaching.
H2a: ChatGPT users will perceive the quiz to be easier than the control group.
H2b: ChatGPT users will perceive the quiz to be easier than the Google-only users.
H2c: ChatGPT users with prompt coaching will perceive the quiz to be easier than ChatGPT users without prompt coaching.
H3a: ChatGPT users will perceive a higher sense of flow than control groups during their search.
H3b: ChatGPT users will perceive a higher sense of flow than Google-only users during their search.
H3c: ChatGPT users with prompt coaching will perceive a higher sense of flow than ChatGPT users without prompt coaching.
Methods
Participants
Procedures
Materials
Pre-Survey
Internet Self-Efficacy Scale
Innovative and Technology Avoidance Scale
Condition 1
Control group
Play Wordle for 5 minutes.
Report the amount of Wordles they were able to solve correctly within the given time
Condition 2
Google search-engine group
Provided with an instructions page
Question with five options asking the participant how often they use Google for research–never, rarely, sometimes, often, and always
Randomly assigned one of the three quiz topics to research on Google for two minutes
Condition 3
ChatGPT without prompt-coaching group
Presurvey: asked whether they have used ChatGPT before, how many times they have used ChatGPT in the past, and how familiar they feel using the AI tool
Tutorial on creating and utilizing ChatGPT for searching information
Instructed to prompt-search about a randomly assigned topic they are provided for two minutes using ChatGPT
Guidelines to assist the participants with their prompt-search are provided along with the two-minute timer that begins counting down
Condition 4
ChatGPT with prompt-coaching group
Presurvey: asked whether they have used ChatGPT before, how many times they have used ChatGPT in the past, and how familiar they feel using the AI tool
Tutorial on creating and utilizing ChatGPT for searching information
Pre-assessed by answering a question on how they would choose to prompt-search information on ChatGPT
Provided with a guide for prompt-coaching tips that teaches the user how to utilize ChatGPT to its full potential as well as write prompts that will refine and produce optimal outputs
Same pre-assessment question will be asked again along with the correct answer and an explanation after tips
Participant will be instructed to prompt-search about a randomly assigned topic they are provided for two minutes using ChatGPT with guidelines
Internet Self-Efficacy Scale Analysis:
Marginal significant difference found between ChatGPT w/ prompt-coaching and w/o prompt-coaching, which means there is a slight noticeable difference between confidence and comfortability levels in using technology
Average rating was found to be higher in the ChatGPT w/ prompt-coaching group, which means Condition 4 participants felt more familiarity
No other statistically significant differences found, meaning confident and comfortability levels of other participants all fall within the same range
Quiz Performance:
ChatGPT groups scored higher than other conditions but no statistically significant differences, which implies H1 a-c, but cannot make the determination that ChatGPT proves to be more efficient and useful for searching over traditional methods
Statistically significant difference between Control group and ChatGPT with prompt-coaching group, demonstrating a search enhancement in ChatGPT
Learning and knowledge of Control group remained stagnant while ChatGPT with prompt-coaching users had the opportunity to increase their understanding levels with the AI chatbot
Conclusion and Future Direction
Participants demonstrated marginal to no significant differences in technology avoidance, familiarity, and self-efficacy
Most rated comfortability in technology usage higher
ChatGPT with prompt-coaching performed higher than other conditions & had statistical difference compared to control
Enhanced learning from searches
Failed to measure and induce participants into flow
Lack of enjoyment & satisfaction
Future → multiple surveying rounds, revisions to flow scale, diversify and increase sample size, replicate study
Reach me at josalynchow@gmail.com.
Innovativeness & Technology Avoidance Scale Analysis:
No differences among the four conditions, which meant participants felt similarly about technology avoidance and usage
Opinions were not significantly strong, which was the result we wanted to achieve to ensure consistency among participants
Average rating was higher than the midpoint, which meant participants had more favorable attitudes towards technological advancements