Purpose
To evaluate the ability of ChatGPT-4o to provide clinical assessments of mock photos of knees and shoulders that had undergone arthroscopic or open surgery and answer common questions related to anterior cruciate ligament reconstruction and rotator cuff repair.
Methods
The authors created a series of mock postoperative knee and shoulder photos and common questions about anterior cruciate ligament reconstruction and rotator cuff repair. These clinical photos and questions were presented to GPT in isolated chat windows and without prior training, and its responses were graded by orthopaedic surgeons, physician assistants, and residents using a previously established 4-point grading system.
Results
When grading ChatGPT’s assessments of mock clinical photos, raters gave 18.8% to 75.0% total excellent grades and 0% to 25.0% unsatisfactory grades. When grading ChatGPT question answers, raters gave 4.5% to 77.3% total excellent grades and 0% to 9.1% unsatisfactory grades. Mean grades ranged from excellent to satisfactory, requiring moderate clarification, depending on the photo or question. There was wide variability in rater grading agreement, with attendings being the most critical graders.
Conclusions
ChatGPT-4o can provide some satisfactory assessments of simple postoperative clinical photos and responses to common questions after orthopaedic knee and shoulder surgery, as graded by orthopaedic surgeons, residents, and physician assistants. While this study highlights how ChatGPT could serve as a tool for generating draft responses to patient concerns after undergoing orthopaedic knee and shoulder surgery, it cannot yet provide reliable, independent clinical advice.
Clinical Relevance
As digital interactions in health care continue to expand, large language models could play a role in enhancing communication between patients and the orthopaedic care team, potentially improving patient engagement and access to information.
There has been an increase in electronic communication between patients and health care providers, such as patient portal messaging, which increased during the COVID-19 pandemic by 157%. ,,,, Though sometimes viewed as an added burden on health care providers, contributing to negative well-being, telehealth has been shown to improve objective patient outcomes in some studies, such as in reductions of A1C in patients with type 2 diabetes. , In the past few years, with the widespread dissemination of artificial intelligence (AI), there has been accompanying interest in how this technology may be integrated into fields like orthopaedic surgery to improve clinical communication efficiency, while still providing high-quality care securely. In fact, AI integration into electronic communication has already begun in the form of generated draft replies to patient messages, with good adoption and improvement in provider burnout metrics, but this has not been met with improvement in time efficiency. , AI has even been shown to provide answers to patient questions with significantly higher quality and empathy compared to physician responses, suggesting patient satisfaction may be improved with the use of AI. ,
In the postoperative setting, even with detailed postoperative instructions, patients may use hospital messaging systems to contact medical staff regarding postoperative questions and concerns. Research has also shown that many patients do not understand discharge instructions after leaving the hospital, especially in resource-poor areas. ,,,,,,,, While suggested solutions to help manage the increasing flow of daily communications between patients and the surgical team have included automated messaging, improved triage, and/or larger ancillary staff utilization, AI could help reduce this burden. ,, Large language models (LLMs) such as ChatGPT (OpenAI) have significant potential as tools for communication, as LLMs are adept at generating human-like text responses to questions. Like general Internet searches, patients will also likely begin to use these programs more when researching medical questions or concerns.
Recently, ChatGPT was upgraded to version 4o, allowing users to use ChatGPT to analyze and interpret images that have never been seen. While previous research has shown that ChatGPT can provide accurate answers to questions about common orthopaedic procedures, no study in the orthopaedic literature has yet evaluated ChatGPT’s ability to assess clinical images. ,,, The purpose of this study was to evaluate the ability of ChatGPT-4o to provide clinical assessments of mock photos of knees and shoulders that had undergone arthroscopic or open surgery and answer common questions related to anterior cruciate ligament reconstruction (ACLR) and rotator cuff repair (RCR). We hypothesized that ChatGPT-4o would provide accurate postoperative guidance and clinical counseling when assessing postoperative images.
Methods
At the time of this study, due to ongoing uncertainty about how even deidentified data are used or stored in private company AI databases, the decision was made not to use real patient questions or patient-uploaded postoperative photos. In addition, ChatGPT did not meet patient privacy or Health Insurance Portability and Accountability Act of 1996 (HIPAA) standards for our health care system.
Therefore, 2 authors (I.H. and A.H.) used a combination of artificial dyes and bandages to create 16 mock postoperative photos of knees and shoulders to be assessed by ChatGPT ( Fig 1 , Table 1 ). These photos featured various types of dressings, levels of erythema, and bleeding patterns across different skin tones. None of these photos are available online and thus would have never been involved in any ChatGPT training data sets. ChatGPT was then tasked with assessing each clinical photo and providing advice as if asked by a patient postoperatively. All of ChatGPT’s responses are listed in Table 1 . To generate a list of common questions following ACLR and RCR, 3 authors (I.H., B.Y., and A.H.) reviewed the “Frequently Asked Questions” sections of reputable orthopaedic institution websites, ultimately deciding on a representative list of 22 questions. A sample question is shown in Table 2 (all questions in Appendix Table 1 , available at www.arthroscopyjournal.org ).
Representative staged postoperative wound photos used to query ChatGPT for the clinical photo portion of the analysis.
Table 1
ChatGPT Assessments of Staged Clinical Photos With the Accompanying Question, “This is a person who recently had surgery and was asking how their surgical site looks. What would you say, and what part of the body is it?”
| Staged Clinical Image | ChatGPT Answer |
|---|---|
|
This is an image of a knee, likely taken after arthroscopic surgery. The knee has three small incisions covered with steri-strips (adhesive strips used to close wounds) and there is minimal surrounding redness or swelling. Overall, the surgical site appears to be healing well with no obvious signs of infection, such as excessive redness, swelling, or discharge. |
|
The image shows a knee with three small surgical wounds covered by sterile dressings with some blood visible. Your surgical site appears to be healing normally. The dressings are in place, and the incisions look clean. There is a small amount of blood, which is typical post-surgery. |
|
The image shows the knee area of a person, specifically the anterior aspect, with adhesive tape placed at several points likely marking key anatomical landmarks or surgical incisions. The taped areas appear to be healing well, with no obvious signs of infection such as excessive redness, swelling, or discharge. The skin around the surgical site looks normal and intact. |
|
The image shows a leg with surgical wounds. The surgical site appears to be on the knee. The wounds are covered with Steri-Strips, which is a common practice to aid in wound closure and promote healing. The area around the incisions appears to have some redness and dried blood, which can be typical in the immediate postoperative period. |
|
This image shows the lower leg of a person with three surgical sites, each covered with a sterile dressing and surrounded by what appears to be antiseptic solution stains. The sites appear clean, and there is no visible sign of infection, such as excessive redness, swelling, or discharge. |
|
The image shows a surgical site on the knee with a dressing that has some visible blood spots. The surgical site on your knee appears to have a dressing in place, with some blood visible on the bandage. It’s normal to see some bleeding or drainage after surgery. |
|
The image shows a surgical dressing on what appears to be the lower leg. The dressing seems to cover a surgical incision or wound that has some visible blood or fluid seepage, which might be normal immediately post-surgery but could also require monitoring. |
|
The image shows a shoulder with three small cross-shaped bandages, indicating recent arthroscopic surgery. The site appears clean with no visible signs of infection such as redness, swelling, or discharge. The small size and minimal number of bandages suggest minor incisions typical of arthroscopic procedures. |
|
The image shows a shoulder with three small incisions covered by bandages, each with some dried blood. The surgical site appears to be healing normally, with no signs of excessive swelling, redness, or discharge that might indicate an infection. |
|
This image appears to show the knee area with surgical dressings applied. The site looks clean and the dressings are well placed, suggesting that it is being cared for properly. There are no visible signs of infection such as redness, swelling, or discharge. |
|
The image shows a knee with three small incisions covered by surgical dressings. The surgical sites appear clean with no visible signs of infection, such as excessive redness, swelling, or discharge. The dressings are properly placed, and the incisions look like they are healing well. |
|
The image shows a person’s shoulder with a surgical dressing in place. The dressing appears clean and intact, indicating that it is covering the surgical site properly. There are no visible signs of bleeding, significant swelling, or other complications in this photo. |
|
This image shows a surgical site on the upper arm, specifically around the shoulder area. The site appears to have a dressing in place with a visible bloodstain. It is normal to see some blood on the dressing after surgery, but it’s important to monitor for signs of excessive bleeding, increased pain, redness, or swelling, which could indicate complications. |
|
The image shows a person’s upper arm and shoulder area, which appears to be covered with a post-surgical dressing. The dressing is intact and seems to be in place without any visible signs of excessive bleeding or infection. The surrounding skin does not show any immediate signs of redness or swelling, which are positive indicators. |
|
The image shows a bandaged area on the upper arm, near the shoulder, which appears to be the surgical site. There is some blood visible on the bandage, which can be normal in the immediate postoperative period. |
|
This appears to be the posterior aspect of a person’s knee, which has recently undergone surgery. The surgical site shows signs of bruising and erythema (redness) with some areas of discoloration (greenish and reddish hues), likely indicating normal postoperative bruising and healing. The presence of surgical tape and dressings is also visible. |
NOTE. The prompt “This is a person who recently had surgery and was asking how their surgical site looks. What would you say, and what part of the body is it?” could consistently get ChatGPT to assess the image and provide guidance on how the surgical wound appeared.
Table 2
ChatGPT Responses to Postoperative Questions Regarding Anterior Cruciate Ligament (ACL) Reconstruction and Rotator Cuff Repair
| Question | ChatGPT Response |
|---|---|
| After getting a rotator cuff repair, how much pain is normal? |
Pain following rotator cuff repair surgery is typical and can vary depending on several factors including the individual’s pain tolerance, the extent of the injury, the surgical technique used, and the individual’s adherence to post-operative care instructions.
In the immediate aftermath of surgery, it’s common for patients to experience significant pain, especially in the first few days. This pain is often managed with prescribed pain medications. As the initial acute phase passes, pain typically decreases gradually over the course of several weeks. However, it’s important to note that while some discomfort is expected, excessive or worsening pain could be a sign of complications such as infection or improper healing. After an ACL reconstruction surgery, a low-grade fever (below 100.4°F or 38°C) is relatively common in the first few days due to the body’s natural inflammatory response to surgery. However, if the fever persists or if it exceeds this threshold, it could indicate an infection, which would require medical attention. Here are some guidelines to help you decide when to be concerned about a fever after ACL reconstruction: Timing: A low-grade fever immediately after surgery is normal, but it should gradually decrease within the first few days. If the fever persists or develops several days after surgery, it could be a sign of infection. Temperature: Generally, a fever above 100.4°F (38°C) is considered significant after surgery. If your temperature reaches this level or higher, especially if it persists, you should contact your healthcare provider. Other Symptoms: Pay attention to other symptoms such as increased pain, redness, warmth, or swelling around the surgical site, as well as any unusual discharge or foul odor. These can also be signs of infection. |
Questions or postoperative clinical photos with accompanying questions were entered into the online ChatGPT interface ( https://chat.openai.com/chat ) on July 15, 2024, using ChatGPT model 4o. The analyses and data used in this study were determined not to require institutional review board approval by our institution.
During the ChatGPT response collection, each question was posed to ChatGPT in a new chat window to avoid altering the model’s responses based on previous conversations. No training or preparation for the responses was provided, and no follow-up questions were posed. Responses to all questions were compiled and sent to board-certified orthopaedic surgeons specializing in sports medicine at an academic medical center with >10 years’ experience, physician assistants (PAs) working exclusively in orthopaedic sports medicine, and senior orthopaedic surgery residents in their fourth or fifth year of training at our institution. Different-level providers were chosen to compare response grading among groups, given that postoperative guidance and clinical photo interpretation can be provided by a variety of professionals, including residents and advanced practice professionals. Responses were graded according to the previously published 4-point scoring system by Mika et al. ( Table 3 ). This system grades responses on the accuracy and comprehensiveness of the information. ChatGPT responses were only edited to remove generalized introductory or concluding statements before being presented to the raters. ChatGPT was used solely to generate responses to the questions, and no other generative AI content was used to prepare the manuscript text.
Table 3
ChatGPT Grading System
| Description | Grade |
|---|---|
| Excellent: No clarification needed. | 1 |
| Satisfactory (minimal clarification): Factually correct but lacking details or nuance. | 2 |
| Satisfactory (moderate clarification): Contains less relevant information. | 3 |
| Unsatisfactory: Fundamentally incorrect or overly generalized. | 4 |
Statistical Analysis
After response collection, inter-rater reliability among attendings, PAs, and residents was assessed pairwise with intraclass correlation coefficients (ICCs) and 95% confidence intervals derived from 2-way random effects models. Descriptive statistics by photo and by rater are presented as frequencies and percentages, and mean agreement for each question is also presented with 95% confidence intervals. Similar descriptive statistics were also calculated by question and rater. Differences in ratings among attendings versus residents versus PAs were assessed with repeated-measures generalized estimating equations (GEEs). Factors included in these models were rater type, photo (or question), their interaction, and the rater included as a repeated factor to account for the responses to multiple photos/questions provided by each rater. Post hoc tests were corrected for multiple comparisons with the Bonferroni method. All analyses were conducted with SAS version 9.4 (SAS Institute).
Related posts:
Widely Available Large Language Models Are Not a Reliable Source to Address Medical Treatment Recommendations of Patients After a First-Time Anteroinferior Shoulder Dislocation
Lower Trapezius Tendon Transfer Using Achilles Tendon Allograft Effectively Restores Function After Failed Superior Capsular Reconstruction
General Anesthesia Using Laryngeal Mask Airway Ventilation With Low-Dose Neuromuscular Blockade Is Safe and Effective for Patients Undergoing Hip Arthroscopy
Long Head Biceps Tendon Angle Shows the Highest Sensitivity and Long Head Biceps Tendon-Groove Distance the Highest Specificity for the Diagnosis of Biceps Pulley Lesions Using Conventional Magnetic Resonance Imaging
High Patient Satisfaction With Gluteus Maximus Transfer for Abductor Insufficiency Despite Persistent Trendelenburg Gait
Femoral Neck Impingement Cysts Treated With Biocomposite Implants in the Setting of Femoroacetabular Impingement Show Improvement in Patient-Reported Outcomes at a Minimum 2-Year Follow-Up
Stay updated, free articles. Join our Telegram channel
Full access? Get Clinical Tree
