EDP Sciences logo
Subscriber Authentication Point
Search
Menu
Home All issues Volume 60 / No 1 (January-March 2025) Radioprotection, 60 1 (2025) 109-119 Full HTML
Open Access
Issue
Radioprotection
Volume 60, Number 1, January-March 2025
Page(s) 109 - 119
DOI https://doi.org/10.1051/radiopro/2024023
Published online 14 March 2025

© S. Ito et al., Published by EDP Sciences, 2025

Licence Creative CommonsThis is an Open Access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

1 Introduction

On March 11, 2011, many people were exposed to radiation following the accident at the Tokyo Electric Power Company (TEPCO) Fukushima Daiichi Nuclear Power Station (Tsubokura et al., 2012; United Nations Scientific Committee on the Effects of Atomic Radiation, 2015). In Japan, prefectures and municipalities are required to maintain regional disaster preparedness plans (Cabinet Office and Fire and Disaster Management Agency, 2013). The preparation of these nuclear disaster preparedness manuals, which include basic knowledge of radiation and other hazards and the actions to be taken in the event of a disaster, is expected to encourage prompt protective actions by the general public and protect their lives and health in the event of a disaster (Cabinet Office and Fire and Disaster Management Agency, 2013; Hellier et al., 2014). In the aftermath of a nuclear power plant accident, many people seek accurate and easy-to-understand knowledge about radiation and other related hazards (Goto et al., 2018; Hellier et al., 2014; Ito et al., 2017; Ohno and Endo, 2015). In this context, the importance of presenting informational content using visual design elements in a format appropriate for people with limited literacy skills has been examined (Gauntlett et al., 2019; Rudd et al., 2003). However, to date, no study has evaluated nuclear accident preparedness manuals in terms of ease of understanding, actionability, and the difficulty of the text although communication is a key issue in the recovery phase of a nuclear accident (Thu Zar et al., 2022).

Materials produced by professionals, including healthcare professionals, have been noted to be difficult to understand because of the use of complex professional terminology, which is often beyond the literacy level of the general adult population (Berland et al., 2001; Ito and Furukawa, 2024; McClure et al., 2016; McInnes and Haglund, 2011). While there are no studies evaluating text difficulty for nuclear disaster materials, an evaluation study of patient education materials reported that adult and pediatric radiation therapy patient education materials exceeded recommended reading levels (Al Balushi et al., 2020; Delaney et al., 2021; Hansberry et al., 2014; Prabhu et al., 2016; Yi et al., 2018). It has been reported that people with lower health literacy and literacy skills are more likely to have poorer overall health (Berkman et al., 2011; Svendsen et al., 2020). Additionally, a survey conducted in Fukushima Prefecture after the Fukushima Daiichi Nuclear Power Plant accident by Tokyo Electric Power Company reported that people with lower health literacy were more anxious about radiation (Kuroda et al., 2018; Nakayama et al., 2019). Furthermore, health literacy among the Japanese is lower than in other countries (Nakayama et al., 2015), and in a post-nuclear accident survey, Fukushima residents (Moriyama et al., 2020) were less health literate than the general Japanese population (Goto et al., 2019; Ishikawa et al., 2016). To improve understanding, it is important to assess the difficulty of radiation-related disaster preparedness materials to match the average literacy level of the local population who will be the readers.

To encourage protective actions by the public during a disaster, the information provided must be easy to understand, easy to act upon, and easy to read. The National Workgroup on Cancer and Health (Cotugna et al., 2005), the American Medical Association (Weiss, 2003), and the Centers for Disease Control and Prevention (CDC) (Centers for Disease Control and Prevention (U.S.) et al., 2009) recommend that patient information materials be readable at a 6th to 8th grade level or lower. However, many online medical materials are more difficult than this standard (Hutchinson et al., 2016) and the readability of nuclear emergency preparedness materials is not clear. The purpose of this study is to clarify the current state of nuclear disaster preparedness manuals by evaluating their ease of understandability, actionability, readability, and credibility.

2 Methods

2.1 Website selection

In this study, we conducted a systematic quantitative content analysis of online materials. From November 27 to December 8, 2023, the survey used Google as the search engine, as in previous epidemiological web page reviews (Alwani et al., 2021; Ito and Furukawa, 2024; Shneyderman et al., 2022). Selection criteria were as follows: the web pages were in Japanese, related to nuclear disaster prevention, and aimed at the general public. Videos; academic papers; reports; advertisements; newspaper articles; pages for medical, educational, and public agency personnel; broken links; inappropriate content; sites requiring subscriptions or fees; and complex disaster prevention materials such as those regarding earthquakes or typhoons were excluded. In cases where only evacuation maps and contact information differed between municipalities and other materials consisted of the same content, only one representative case was extracted. This study was not subject to ethical review because it did not involve any human subjects or interventions.

The search keywords were “(radiation OR radioactive material OR nuclear) AND (disaster prevention OR disaster) AND (pamphlet OR handbook OR leaflet OR flyer OR bookmark)”, and all 219 web pages found through this search were reviewed. Of these, the Reconstruction Agency (35 web pages) and the Nuclear Human Resource Development Network (57 web pages) provided a list of web pages related to radiation disaster prevention, so these web pages were also reviewed. An exhaustive manual search was also conducted because more than 10 years had passed since the Great East Japan Earthquake and many materials were difficult to find by keyword search alone.

2.2 Evaluators

In this study, one epidemiologist studying the nuclear accident reviewed the materials on the websites and independently evaluated all but those whose content was clearly inappropriate. Next, an internist who did not specialize in radiation or nuclear disasters evaluated 30 randomly selected websites from among all the websites using the PEMAT-P. The PEMAT-P is an index intended to be evaluated by health care providers and health professionals who provide medical information. This independent evaluation by a person who specialized in radiation and nuclear hazards and a non-specialist was performed to avoid the possibility of information bias in the evaluation of the material. In cases of disagreement, consensus was reached through discussion.

2.3 Understandability and actionability

The understandability and actionability of the contents of each website were evaluated using the Japanese version of the PEMAT-P (Furukawa et al., 2022; Shoemaker et al., 2014). The PEMAT-P Japanese version is a 23-item evaluation tool designed to assess the understandability of health and medical information for patients and consumers, and evaluates items such as “content”, “word choice and style”, “use of numbers”, “organization,” “layout and design,” and “use of visual aids.” The questionnaire items are listed in Table 4. Evaluators were required to respond to each item by selecting either “Disagree = 0” or “Agree = 1”. These scores ranged from 0% to 100%, with higher scores indicating higher levels of perceived comprehensibility and ease of actionability. The cutoff value was set at 70% for both scores.

2.4 Readability

In this study, to quantitatively evaluate the readability of Japanese text on web pages, we used jReadability, a tool that analyzes elements such as grammar, sentence structure, word complexity, and sentence length, with the aim of making the text easier to understand (Lee, 2016; Lee et al., 2023). The index calculates readability based on the average length of a sentence, word difficulty, percentage of grammatical parts of speech, and type of characters per sentence. Scores range from 0.5 to 6.4, with higher scores indicating that the sentences are relatively easy to read: 0.5–1.4, very difficult to read; 1.5–2.4, difficult to read; 2.5–3.4, somewhat difficult to read; 3.5–4.4, neutral; 4.5–5.4, easy to read; and 5.5–6.4, very easy to read). The approximate reading time was calculated based on the amount of text in the material and the average Japanese reading speed of 653 words/minute (Kobayashi and Kawashima, 2018). In addition, a list of place names and addresses of evacuation centers was excluded from the jReadability evaluation. The reason for this was to prevent the difficulty rating of the sentences from becoming too high due to many difficult place names.

2.5 Credibility (JAMA benchmark criteria and GQS)

To evaluate the credibility of web pages, we utilized the Journal of the American Medical Association (JAMA) Benchmark Criteria (Silberg et al., 1997), which consist of authorship (clearly identifiable author and contributors with affiliations and relevant credentials present), attribution (references and sources clearly listed with any copyright information disclosed), currency (clearly identifiable posting date of any content as well as date of any revisions), and disclosure (website ownership clearly disclosed along with any sponsorship, advertising, underwriting, and financial support). Each criterion was assessed using a binary scale of 1 for “met” and 0 for “not met”. We also used the Global Quality Scale (GQS) (Bernard et al., 2007), which is a comprehensive indicator for assessing the quality of online materials. To stratify the quality of the materials, the GQS employs the following five-point rating system: (1) poor quality, poor flow of the site, most information missing, not at all useful for patients; (2) generally poor quality and poor flow, some information listed but many important topics missing, of very limited use to patients; (3) moderate quality, suboptimal flow, some important information adequately discussed but other topics poorly discussed, somewhat useful for patients; (4) good quality and generally good flow, most relevant information listed, but some topics not covered, useful for patients; and (5) excellent quality and excellent flow, very useful for patients).

2.6 Statistical analysis

Three types of statistical analyses were performed in this study. First, within each rating scale, a one-factor analysis of variance using Tukey’s method was conducted to compare means, a chi-square test or Fisher’s exact probability test was used to compare proportions, and a Kruskal-Wallis test was used to compare non-normal distributions. Second, Spearman’s correlation coefficient was calculated to examine the association between the year the sources were updated and each rating scale. Finally, the intraclass correlation coefficient was calculated to compute the agreement between evaluators. A significance level of 0.05 was used for these statistical analyses. In the residual analysis, a significant difference was considered to have occurred when the adjusted residual was greater than or less than 1.96. All analyses were performed using IBM SPSS (Ver. 25.0 for Windows; IBM SPSS Japan, Tokyo, Japan).

3 Results

In this study, 116 materials were included in the evaluation. The breakdown is as follows. First, of the 219 websites found in the keyword search, 160 were excluded based on exclusion criteria (142 with no materials and 18 with duplicates), leaving 59 websites. Of these, 66 materials were extracted from the 59 websites because a single website contained multiple materials, e.g., one for the general public and one for elementary school students. Next, from the list of 35 Reconstruction Agency websites, 12 with different themes were excluded, leaving 23; from the list of 57 Nuclear Human Resource Development Network websites, 54 with different themes were excluded, leaving 3. Finally, 24 materials were added by manual searching. The intraclass correlation coefficient (ICC) was calculated to quantify the degree of agreement between items, with ICC = 0.52 for PEMAT-P’s understandability and ICC = 0.78 for actionability.

The material attributes and the rating scale scores are listed in Table 1. The source of the material was the municipality in 54 cases (46.6%), the topic of the material was nuclear disaster prevention in 76 cases (65.5%), the target audience for the material was general adults in 102 cases (87.9%), and the year the material was last updated was 2023 in 21 cases (18.1%). The number of materials scoring 70 or higher on the PEMAT-P was 71 (61.2%) for understandability and 57 (49.1%) for actionability. Regarding the difficulty level of the texts by jReadability, “Difficult” was the most frequent response with 82 responses (70.7%). We calculated the Spearman’s correlation coefficient between the year the material was updated and each rating scale, and found that the year the material was updated correlated significantly with understandability (rho = 0.21, p = 0.046), GQS (rho = 0.33, p = 0.001), and total number of letters (rho = 0.32, p = 0.002) on the PEMAT-P. No significant differences were found among the other items.

Table 1

Characteristics of Nuclear Emergency Preparedness Materials (n = 116).

3.1 Material themes

The themes of the materials were nuclear disaster prevention in 76 cases and basic knowledge of radiation in 40 cases (Tab. 2). More than 80% of the nuclear disaster prevention materials described indoor and outdoor evacuation methods, means of information transmission, and the definition of a nuclear disaster. In contrast, less than 60% of the materials described the criteria for evacuation and inspection at the time of evacuation. More than 80% of the materials on basic knowledge of radiation explained how to measure radiation, food, and dose, and 45.0% explained the effects of radiation on children.

Table 2

Topics covered in nuclear-related materials by source.

3.2 Understandability and actionability

Regarding the results of group comparisons of the PEMAT-P’s understandability and actionability scale scores, Table 3 and Figure 1 summarize the results. For the understandability scale score, in the comparison by source, municipalities (M = 76.7, SD = 13.9) and prefectures (M = 79.4, SD = 15.8) were significantly higher than Academic Organizations (M = 53.3, SD = 18.5). Additionally, in the comparison by source, municipalities (M = 75.2, SD = 23.2) and prefectures (M = 82.1, SD = 16.2) were significantly higher than the other groups. Furthermore, the percentage of materials scoring 70 or higher on the PEMAT-P is discussed. With respect to actionability, a larger proportion of municipalities (n = 34, 63.0%) and prefectures (n = 21, 77.8%) achieved a score of 70 or higher.

The PEMAT-P item scores are listed in Table 4. With respect to the results for all materials, the following items were applicable to more than 80% of those materials: “No unnecessary info (item 2; 94.0%)”, “No calculations (item 6; 95.4%)”, “Use of informative headers (item 8; 89.7%)”, “Info presentation in logical sequence (item 9; 81.0%)”, “use of visual aids (item 12; 81.9%)”, “use of simple table (item 16; 93.4%)”, and “Clearly identified at least 1 action the user can take (item 17; 84.5%).” The items with less than 50% of applicable materials are as follows: “Number easy to understand (item 5; 49.5%)”, “Providing summary (item 10; 19.8%)”, and “Explained how to use chart, table, etc. (item 22; 43.2%).” For the results by source, the three items that exceeded 80% in all groups were “No unnecessary information (item 2)”, “No calculations (item 6)” and “Use of simple table (item 16)”. Conversely, one item, “Providing summary (Item 10)”, received less than 50% of the responses in all groups.

thumbnail Figure 1

Actionability and Understandability scores by Source. The solid line shows the cutoff points (70%) of PEMAT-P, and the dashed line shows the linear regression line.
Table 3

Comparison of scale scores by Source.

Table 4

PEMAT-P item scores by Source and overall (n = 116).

3.3 Readability

For comparisons by source, municipalities (M = 1.8, SD = 0.5) had more difficulty than National Government/Ministry (M = 2.4, SD = 1.2), General Adult (M = 1.9, SD = 0.6) was more difficult than Elementary School Students (M = 2.6, SD = 1.2) (Tab. 3).

3.4 Credibility

When comparing by source, municipalities (M = 4.5, SD = 0.8) and prefectures (M = 4.6, SD = 0.9) scored significantly higher than National Government/Ministry (M = 3.4, SD = 0.9) and Academic Organizations (M = 3.1, SD = 0.9) (Tab. 3).

4 Discussion

The results showed that 71 (61.2%) of the nuclear disaster prevention materials were “easy to understand,” 57 (49.1%) were “easy to act upon,” 3 (2.6%) had “little or no jargon below neutral,” 41 (35.3%) had “all references and sources are listed,” and 52 (44.8%) were “within 5 years updated” in 41 cases (35.3%). Looking at the PEMAT-P by item, more than 90% of the total number of documents were (1) not deviating from the purpose of the information, (2) not requiring calculations, and (3) having appropriate matrix headings in the tables. On the other hand, less than 50% of all the materials (1) contained easy-to-understand figures and tables, (2) contained a summary or outline of the material, and (3) explained how to act on the figures and tables. With respect to the inter-rater agreement rate for the PEMAT-P, actionability was sufficiently high; with respect to understandability, it was lower than actionability, but there were no clear criteria for ICC values and it was judged to be acceptable.

These results suggest that the nuclear disaster prevention materials have been updated and that the information provided is consistent with the purpose of the materials. On the other hand, the text of the nuclear disaster prevention materials may not be understood because it is beyond the reading comprehension of the average adult. The results of this study were similar to those of previous studies (Al Balushi et al., 2020; Delaney et al., 2021; Hansberry et al., 2014; Prabhu et al., 2016; Yi et al., 2018). The National Work Group on Cancer and Health (Cotugna et al., 2005), the American Medical Association (Weiss, 2003), and the Centers for Disease Control and Prevention (CDC) (Centers for Disease Control and Prevention (U.S.) et al., 2009) recommend that patient information materials be readable at a 6th to 8th grade level or lower. Although no studies have evaluated the readability, understandability, and actionability of nuclear disaster materials, a study of patient education materials on radiation therapy reported that adult and pediatric patient education materials tended to be above the recommended reading level (Al Balushi et al., 2020; Delaney et al., 2021; Hansberry et al., 2014; Prabhu et al., 2016; Yi et al., 2018). In nuclear emergency preparedness materials, authors use easy-to-understand illustrations, definitions and explanations of technical terms, and word substitutions to promote reader understanding, but these explanations are also considered difficult to understand. In addition, numerical values used in figures and tables may also be difficult to understand. When preparing materials for nuclear disaster prevention, it may be necessary to improve the readability level of text and numerical values by using a text difficulty evaluation index such as jReadability, incorporating the opinions of non-specialist users of the materials and summarizing and condensing the content as appropriate. In addition, although the figures and tables are detailed and accurate, it is difficult to understand their meaning and how to apply them to one’s actions and daily life. In addition to detailed versions of these figures and tables, it would be beneficial to create summaries and abstract versions of important information. Thus it would be very beneficial that the International Committee on Radiological Protection (ICRP) future recommendations be easily understood by the average adult (Clement et al., 2021, 2022).

In terms of comparisons among sources, the municipal and prefectural materials were significantly easier to understand, more actionable, and of higher credibility than the academic organizations. The municipalities and prefectures were more likely to provide information on how to deal with specific problems (actionability). Municipalities and prefectures create nuclear disaster preparedness materials based on the Manual for Preparation of Regional Disaster Prevention Plans (Nuclear Disaster Preparedness Chapter) by the Cabinet Office in Japan (Cabinet Office and Fire and Disaster Management Agency, 2013). Therefore, many of the materials describe specific actions to be taken in the event of a disaster. On the other hand, the materials prepared by the national government, ministries, and agencies may not include the actionable items because there is no manual for these organizational bodies to prepare such materials, and the actions that should be taken in the event of a disaster differ depending on the residential area. In addition, national and provincial government materials are considered to contain a lot of specialized information, as their themes vary from basic knowledge of radiation, survey results after the accident at the Fukushima Daiichi Nuclear Power Plant, and information on reconstruction in Fukushima Prefecture. Despite the above, the Japanese text difficulty level for the municipal materials was significantly higher than that for the national materials. In addition, according to the jReadability criteria (Lee, 2016; Lee et al., 2023), a score of 1.5–2.4 is classified in the “difficult” category, and it seems reasonable to interpret all of them as having a high level of text difficulty, as in previous studies (Al Balushi et al., 2020; Delaney et al., 2021; Hansberry et al., 2014; Prabhu et al., 2016; Yi et al., 2018).

The strengths of the present study are as follows. To our knowledge, this is the first study to quantitatively analyze the content of nuclear disaster prevention-related websites. Our search was able to cover all relevant Japanese-language websites related to nuclear disaster prevention because web page materials were collected until saturation was reached. However, the present study also has some limitations. First, since the evaluation covered only Japanese-language websites related to nuclear disaster prevention, it is difficult to generalize the results to websites related to nuclear disaster prevention in other languages, printed materials, smartphone applications, videos, and so on. Second, because this study included only freely accessible websites, it is also difficult to generalize the results to paid websites that may contain more detailed information about nuclear disaster prevention. However, it is likely that the first place a person newly interested in radiation would look for information would be a freely accessible website rather than a paid one. Third, because some of the evaluation indicators on the website, such as the JAMA Benchmark Criteria (Silberg et al., 1997) and the GQS (Bernard et al., 2007), have not been fully examined in terms of reliability and validity, caution is needed in interpreting whether they adequately assess the content that one wishes to evaluate. Fourth, if the amount of text in nuclear disaster preparedness materials is large, the understandability, actionability, and credibility of the materials are likely to be evaluated as high. However, materials with a large amount of information may not be easily accessed from any location, and people may be less inclined to read these materials for pre-disaster preparedness. It has been reported that the percentage of people who save and understand emergency preparedness pamphlets is low (Hellier et al., 2014). In the future, the usefulness of these materials as pamphlets may also need to be evaluated.

In this study, we evaluated the overall quality of existing nuclear accident prevention brochures and were able to identify the characteristics of the materials by source. However, we have not yet been able to create and provide materials adapted to the “average adult” and suggest appropriate search methods. Studies have been conducted on risk communication after a nuclear accident, and the importance of a two-way approach between victims and experts has been emphasized (Bertho et al., 2021; Thu Zar et al., 2022). In the future, it will be important to create materials that are accessible to “average adults" and others.

5 Conclusion

The present study quantitatively evaluated the understandability, actionability, readability, and credibility of nuclear disaster prevention materials for the Japanese public. Municipal and prefectural materials were easy to understand and act upon, and the credibility of the materials was high, but the Japanese text difficulty level for most materials was also high. In creating or revising materials, it is considered important to reduce the difficulty level of the text according to its difficulty rating index, to incorporate the opinions of local residents as users, and to add summaries as appropriate.

Acknowledgments

This study used the “Japanese Sentence Difficulty Discrimination System” (http://jreadability.net), a product of a Grant-in-Aid for Scientific Research (Project No. 25370573).

Funding

This work was supported by JSPS KAKENHI Grant Number JP 24K06389 and the Program of the Network-type Joint Usage/Research Center for Radiation Disaster Medical Science.

Conflicts of interest

The authors declare that they have no conflict of interest.

Data availability statement

The datasets generated and/or analyzed during the current study are available from the corresponding author on reasonable request

Author contribution statement

Ito conceived of the paper, collected and analyzed data, and wrote the paper; Okuhara, Okada and Kiuchi conceived and designed the study and revised the paper; Furukawa conceived of the study, collected data, and revised the paper. In addition, all authors have final review of the manuscript and are responsible for the content of the article of the paper.

References

Cite this article as: Ito S, Furukawa E, Okuhara T, Okada H, Kiuchi T. 2025. Evaluating the overall quality of online information on nuclear power plant accidents in Japanese. Radioprotection 60 (1): 109–119. https://doi.org/10.1051/radiopro/2024023.

All Tables

Table 1

Characteristics of Nuclear Emergency Preparedness Materials (n = 116).

In the text
Table 2

Topics covered in nuclear-related materials by source.

In the text
Table 3

Comparison of scale scores by Source.

In the text
Table 4

PEMAT-P item scores by Source and overall (n = 116).

In the text

All Figures

thumbnail Figure 1

Actionability and Understandability scores by Source. The solid line shows the cutoff points (70%) of PEMAT-P, and the dashed line shows the linear regression line.
In the text

Current usage metrics show cumulative count of Article Views (full-text article views including HTML views, PDF and ePub downloads, according to the available data) and Abstracts Views on Vision4Press platform.

Data correspond to usage on the plateform after 2015. The current usage metrics is available 48-96 hours after online publication and is updated daily on week days.

Initial download of the metrics may take a while.