Open Access
Table 4
PEMAT-P item scores among artificial intelligence chatbot and web search.
| Artificial Intelligence Chatbot | Web Search | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Normal level | 6th Grade Level | ||||||||||||||||||||
| ChatGPT-3.5 | Copilot | Gemini | ChatGPT-3.5 | Copilot | Gemini | ||||||||||||||||
| n | % | AR | n | % | AR | n | % | AR | n | % | AR | n | % | AR | n | % | AR | n | % | AR | |
| Understandability | |||||||||||||||||||||
| Content | |||||||||||||||||||||
| 1 The material makes its purpose completely evident | 30 | 100.0 | 2.7 | 32 | 100.0 | 2.7 | 29 | 100.0 | 2.6 | 29 | 100.0 | 2.6 | 30 | 96.8 | 2.2 | 26 | 92.9 | 1.5 | 117 | 66.1 | −8.0 |
| 2 The material does not include information or content that distracts from its purpose | 30 | 100.0 | 1.2 | 32 | 100.0 | 1.2 | 29 | 100.0 | 1.2 | 29 | 100.0 | 1.2 | 28 | 90.3 | −1.6 | 25 | 89.3 | −1.8 | 168 | 94.9 | −0.8 |
| Word Choice & Style | |||||||||||||||||||||
| 3 The material uses common, everyday language | 14 | 46.7 | −1.0 | 20 | 62.5 | 0.9 | 19 | 65.5 | 1.2 | 25 | 86.2 | 3.5 | 24 | 77.4 | 2.6 | 27 | 96.4 | 4.6 | 67 | 37.9 | −6.5 |
| 4 Medical terms are defined when they are used | 14 | 46.7 | −0.9 | 16 | 50.0 | −0.5 | 16 | 55.2 | 0.1 | 20 | 69.0 | 1.6 | 25 | 80.6 | 3.1 | 27 | 96.4 | 4.6 | 76 | 42.9 | −4.4 |
| Use of Numbers | |||||||||||||||||||||
| 5 Numbers appearing in the material are clear and easy to understanda | 0 | 0.0 | – | 2 | 20.0 | −1.0 | 1 | 20.0 | −0.7 | 1 | 100.0 | 1.4 | 5 | 35.7 | 0.1 | 1 | 50.0 | 0.5 | 42 | 34.7 | 0.4 |
| 6 The material does not expect the user to perform calculationsa | 30 | 100.0 | 0.9 | 32 | 100.0 | 0.9 | 29 | 100.0 | 0.9 | 29 | 100.0 | 0.9 | 31 | 100.0 | 0.9 | 28 | 100.0 | 0.8 | 169 | 95.5 | −2.9 |
| Organization | |||||||||||||||||||||
| 7 The material breaks or "chunks" information into short section | – | – | – | – | – | – | 66 | 56.9 | 0.0 | ||||||||||||
| 8 The material's sections have informative headers | – | – | – | – | – | – | 81 | 69.2 | 0.0 | ||||||||||||
| 9 The material presents information in a logical sequence | 30 | 100.0 | 0.5 | 32 | 100.0 | 0.5 | 29 | 100.0 | 0.5 | 29 | 100.0 | 0.5 | 30 | 96.8 | −1.5 | 28 | 100.0 | 0.5 | 175 | 98.9 | −0.6 |
| 10 The material provides a summary | – | – | – | – | – | – | 18 | 15.4 | 0.0 | ||||||||||||
| Layout & Design | |||||||||||||||||||||
| 11 The material uses visual cues to draw attention to key points | 5 | 16.7 | −3.3 | 23 | 71.9 | 3.1 | 29 | 100.0 | 6.1 | 2 | 6.9 | −4.4 | 25 | 80.6 | 4.1 | 28 | 100.0 | 6.0 | 51 | 28.8 | −6.4 |
| Use of Visual Aids | |||||||||||||||||||||
| 12 The material uses visual aids whenever they could make content more easily understood | 0 | 0.0 | −3.5 | 0 | 0.0 | −3.6 | 0 | 0.0 | −3.5 | 0 | 0.0 | −3.5 | 0 | 0.0 | −3.6 | 0 | 0.0 | −3.4 | 97 | 55.4 | 11.7 |
| 13 The material's visual aids reinforce rather than distract from the content | – | – | – | – | – | – | 110 | 77.5 | – | ||||||||||||
| 14 The material's visual aids have clear titles or captions | – | – | – | – | – | – | 95 | 66.4 | – | ||||||||||||
| 15 The material uses illustrations and photographs that are clear and uncluttered | – | – | – | – | – | – | 60 | 42.0 | – | ||||||||||||
| 16 The material uses simple tables with short and clear row and column headingsa | – | – | – | – | – | – | 46 | 90.2 | – | ||||||||||||
| Actionability | |||||||||||||||||||||
| 17 The material clearly identifies at least one action the user can take | 2 | 6.7 | 0.7 | 0 | 0.0 | −1.2 | 4 | 13.8 | 2.7 | 1 | 3.4 | −0.2 | 1 | 3.2 | −0.3 | 3 | 10.7 | 1.8 | 4 | 2.3 | −1.8 |
| 18 The material addresses the user directly when describing actions | 0 | 0.0 | −0.9 | 0 | 0.0 | −0.9 | 2 | 6.9 | 1.8 | 0 | 0.0 | −0.9 | 1 | 3.2 | 0.4 | 2 | 7.1 | 1.8 | 3 | 1.7 | −0.7 |
| 19 The material breaks down any action into manageable, explicit steps | 0 | 0.0 | −1.0 | 0 | 0.0 | −1.0 | 4 | 13.8 | 3.7 | 0 | 0.0 | −1.0 | 1 | 3.2 | 0.1 | 1 | 3.6 | 0.3 | 4 | 2.3 | −0.6 |
| 20 The material provides a tangible tool whenever it could help the user take action | 0 | 0.0 | – | 0 | 0.0 | – | 0 | 0.0 | – | 0 | 0.0 | – | 0 | 0.0 | – | 0 | 0.0 | – | 0 | 0.0 | – |
| 21 The material provides simple instructions or examples of how to perform calculationsa | – | – | – | – | – | – | 0 | 0.0 | – | ||||||||||||
| 22 The material explains how to use the charts, graphs, tables, or diagrams to take actionsa | – | – | – | – | – | – | 0 | 0.0 | – | ||||||||||||
| 23 The material uses visual aids whenever they could make it easier to act on the instructions | 0 | 0.0 | −0.3 | 0 | 0.0 | −0.3 | 0 | 0.0 | −0.3 | 0 | 0.0 | −0.3 | 0 | 0.0 | −0.3 | 0 | 0.0 | −0.3 | 1 | 0.6 | 1.0 |
Current usage metrics show cumulative count of Article Views (full-text article views including HTML views, PDF and ePub downloads, according to the available data) and Abstracts Views on Vision4Press platform.
Data correspond to usage on the plateform after 2015. The current usage metrics is available 48-96 hours after online publication and is updated daily on week days.
Initial download of the metrics may take a while.