A marketing claims substantiation study
Measuring improvement
During an FDA 510(k) review process, the technical documentation is evaluated, but so too are marketing claims about the medical device’s performance.
Use-Lab conducted a study measuring the effect of new product features to improve workflow during surgical procedures for a manufacturer of mobile x-ray systems.
Preparation
Based on the new product features, initial ideas and high-level claims were made about the desired increase in performance. These high-level claims concerned various topics such as improved communication between surgeons and techs, shortened intervention times or the reduction of unnecessary x-ray images.
Based on these topics, a coding scheme was then used to further define (operationalize) the individual measurement points on which the study design was ultimately based. This resulted in both objective measures (e.g., time to first image, number of attempts to successful image, total time) but also subjective measures (e.g., perceived workload, perceived fitness for purpose).
Based on the new product features, initial ideas and high-level claims were made about the desired increase in performance. These high-level claims concerned various topics such as improved communication between surgeons and techs, shortened intervention times or the reduction of unnecessary x-ray images.
Based on these topics, a coding scheme was then used to further define (operationalize) the individual measurement points on which the study design was ultimately based. This resulted in both objective measures (e.g., time to first image, number of attempts to successful image, total time) but also subjective measures (e.g., perceived workload, perceived fitness for purpose).
During the study, participants were asked to take multiple images with the x-ray system. A wireless camera was installed on the system’s flat detector, which transmitted the images to the x-ray system’s integrated screen. An anatomical replica of the human sacral region served as a model into which metal tubes were inserted in typical projection planes. The aim was to position the camera in such a way that, after triggering the camera, it was possible to look more or less exactly through a tube.
In total, there were seven tubes and thus seven possible positions. Based on this, various sequences were worked out. In total, the participants were asked to approach 25 individual positions. With 30 test sessions, this resulted in up to 750 individual positions for later analysis.
For the study, teams of physicians and techs who did not know each other were invited to participate in order, for example, to be able to evaluate claims regarding reduced miscommunication, and individual techs were asked to evaluate the usability of the overall system.
For comparison, the new features were enabled in some runs and disabled in others. The order of performance with and without features was alternated between individual participants to compensate for any habituation and learning effects that might occur.
Data Recording & Analysis
The individual test sessions were recorded on audio and video for later data analysis. During the sessions, particular attention was paid to the different task completion times, the number of images required, and the number of miscommunications, etc. The final data was analyzed using a statistical program and significance tests were performed for each of the previously established hypotheses.
Results
Based on the data, a majority of the previously created marketing claims were supported with significant data. Using the features led to the following results, among others:
- 45% less movement in the wrong direction on positioning
- 45% fewer instances of miscommunication
- Lower sense of frustration with (re)positioning.
(-48% for physicians, -72% for techs) - 55% fewer images for re-positioning
The corresponding white paper by the manufacturer can be found here.
