Menu
Home Explore People Places Arts History Plants & Animals Science Life & Culture Technology
On this page
Progress in International Reading Literacy Study
Progress in International Reading Literacy Study

The Progress in International Reading Literacy Study (PIRLS) is an international study of reading (comprehension) achievement in 9–10 year olds. It has been conducted every five years since 2001 by the International Association for the Evaluation of Educational Achievement (IEA). It is designed to measure children's reading literacy achievement, to provide a baseline for future studies of trends in achievement, and to gather information about children's home and school experiences in learning to read.

Over 60 countries and sub-national, benchmarking entities participated in PIRLS 2021.

Related Image Collections Add Image
We don't have any YouTube videos related to Progress in International Reading Literacy Study yet.
We don't have any PDF documents related to Progress in International Reading Literacy Study yet.
We don't have any Books related to Progress in International Reading Literacy Study yet.
We don't have any archived web articles related to Progress in International Reading Literacy Study yet.

History

PIRLS4 provides internationally comparative data on how well children read by assessing students' reading achievement. PIRLS collects background information on how education systems provide educational opportunities to their students as well as the factors that influence how students use these opportunities. These background data include information about the following: national curriculum policies in reading; how the education system is organized to facilitate learning; students' home environment for learning; school climate and resources; and how instruction actually occurs in classrooms. Studies of reading literacy had been conducted prior to the PIRLS study of 2001, and PIRLS is the successor to IEA studies, such as the Reading Literacy Study, that started in 1970 and continued until 1991.5 The PIRLS study of 2001 started the trend for cyclical testing; PIRLS has a frequency of five years. By administering the test every five years, education systems are able to monitor their children's literacy achievement over time. The current cycle, PIRLS 2016, is the fourth cycle of the IEA PIRLS. Like the previous PIRLS cycles (conducted in 2001, 2006, and 2011), the study will also collect information about home supports for literacy, curriculum and curriculum implementation, instructional practices, and school resources in each participating country.6

Cycles

PIRLS 2021

PIRLS 2021 is the fifth cycle in the PIRLS framework.7 IEA's PIRLS will continue to collect background information from the assessed students, their parents, teachers and school principals on how education systems provide educational opportunities to their students, as well as the factors that influence how students use these opportunities. Trend results across assessments permit countries to monitor the effectiveness of their educational systems in a global context, and PIRLS 2021 marks 20 years of trends.

PIRLS 2021 evolves further from PIRLS 2016 in allowing countries to administer the full PIRLS reading assessment, including both PIRLS Informational and Literary (the previous standard PIRLS assessment), and the ePIRLS Online Informational (the previous ePIRLS), as one digitally based endeavour. Countries may also select from two levels of the PIRLS assessment; providing students with an assessment experience better suited to their reading abilities increases student motivation and provides more accurate assessment data. All results will be reported on the same PIRLS achievement scale.

PIRLS 2021 thus offers three options, enabling participants to select the administration path best suited to assessing their education system:(1) A new fully digital ePIRLS assessment, which integrates all aspects of PIRLS Informational, PIRLS Literary, and the ePIRLS Online Informational assessments; 2) The paper-only version of the PIRLS assessment, which is equivalent to the original pen-and-paper PIRLS standard assessment; and (3) The paper-only version of the PIRLS assessment, taken together with the ePIRLS Online Informational assessment.8

RankCountryAveragescale score9Changeover 5 years
1 Singapore587 11 points
2 Ireland577 10 points
3 Hong Kong573 4 points
4 Russia567 14 points
5 Northern Ireland566 1 point
6 England10558 1 point
7 Croatia557N/A
8 Lithuania552 4 points
9 Finland549 17 points
9 Poland549 16 points
11 United States548 1 point
12 Chinese Taipei544 15 points
12 Sweden544 11 points
14 Australia11540 4 points
14 Bulgaria539 13 points
14 Czech Republic539 4 points
17 Hungary539 15 points
17 Denmark539 8 points
17 Norway12539 20 points
20 Italy537 11 points
21 Macau536 10 points
22 Austria530 11 points
23 Slovakia529 6 points
24 Latvia528 30 points
25 Netherlands527 18 points
26 Germany524 13 points
27 New Zealand521 2 points
27 Spain521 7 points
29 Portugal520 8 points
29 Slovenia520 22 points
30 Malta515 63 points
31 France514 3 points
31 Serbia514N/A
33 Albania513N/A
34 Cyprus511N/A
34 Belgium (Flemish)511 14 points
36 Israel13510 20 points
37 Kazakhstan504 32 points
PIRLS Scale Centerpoint500
38 Turkey496N/A
39 Belgium (French)494 3 points
39 Georgia494 6 points
41 Montenegro487N/A
42 Qatar485 43 points
43 United Arab Emirates483 33 points
44 Bahrain458 12 points
45 Saudi Arabia449 19 points
46 North Macedonia442N/A
47 Azerbaijan440 32 points
48 Uzbekistan437N/A
49 Oman429 11 points
50 Kosovo421N/A
51 Brazil14419N/A
52 Iran15413 15 points
53 Jordan381N/A
54 Egypt378 48 points
55 Morocco372 14 points
56 South Africa16288 32 points
Benchmarking participants
Moscow (Russia)598 14 points
Dubai (United Arab Emirates)552 37 points
Quebec (Canada)551 4 points
Alberta (Canada)539N/A
British Columbia (Canada)535N/A
Newfoundland and Labrador (Canada)523N/A
Abu Dhabi (United Arab Emirates)440 26 points
 South Africa1718384N/A
Table key
Delayed assessment of 4th grade cohort at the beginning of 5th grade

PIRLS 2016

PIRLS 2016 was released on December 5, 2017.19 It also collects extensive information about home supports for literacy, curriculum and curriculum implementation, instructional practices, and school resources in each participating country. In this cycle there were two additional initiatives: (1) the PIRLS Literacy assessment (earlier known as prePIRLS) is equivalent to PIRLS in scope and reflects the same conception of reading as PIRLS. Its purpose is to extend the effective measurement of reading literacy at the lower end of the achievement scale. Countries whose fourth-grade students are still developing fundamental reading skills can participate in the PIRLS Literacy assessment and still have their results reported on the PIRLS achievement scale. The reading passages and questions in common between the PIRLS Literacy and the PIRLS assessments will enable the two assessments to be linked, and their results to be compared. (2) Initiated in 2016, ePIRLS is a computer-based reading assessment of students' ability to acquire and use information when reading online. The assessment encompasses an engaging, simulated internet environment with authentic school-like assignments about science and social studies topics. The ePIRLS online reading achievement scale enables countries to examine their fourth-graders' online reading performance relative to their performance on the PIRLS reading achievement scales.

In terms of trends, the PIRLS results for student achievement by country states that 18 countries had higher average achievement, 13 countries had the same average achievement, and 10 countries had lower average achievement; and girls had higher reading achievement than boys in 48 of the 50 countries.2021

The 2016 PIRLS Encyclopedia has the Education Policy and Curriculum in Reading by country. It describes the structure of each education system, the reading curricula in the primary grades, and overall policies related to reading instruction.22

The ten countries with the highest average reading achievement were: Russian Federation, Singapore, Hong Kong SAR, Ireland, Finland, Poland, Northern Ireland, Norway, Taiwan, and England.23

RankCountryAveragescale scoreChangeover 5 years
1 Russia581 13 points
2 Singapore576 9 points
3 Hong Kong569 2 points
4 Ireland567 15 points
5 Finland566 2 points
6 Poland565 39 points
6 Northern Ireland565 7 points
8 Norway24559 52 points
8 Chinese Taipei559 6 points
8 England559 7 points
11 Latvia558N/A
12 Sweden555 13 points
13 Hungary554 15 points
14 Bulgaria552 20 points
15 United States549 7 points
16 Lithuania548 20 points
16 Italy548 7 points
18 Denmark547 7 points
19 Macau546N/A
20 Netherlands545 1 point
21 Australia544 17 points
22 Czech Republic543 2 points
22 Canada543 5 points
24 Slovenia542 12 points
25 Austria541 12 points
26 Germany537 4 points
27 Kazakhstan536N/A
28 Slovakia535
29 Israel530 11 points
30 Portugal528 13 points
30 Spain528 15 points
32 Belgium (Flemish)525N/A
33 New Zealand523 8 points
34 France511 9 points
International average500
35 Belgium (French)497 9 points
36 Chile494N/A
37 Georgia488
38 Trinidad and Tobago479 8 points
39 Azerbaijan472 10 points
40 Malta452 25 points
41 United Arab Emirates450 11 points
42 Bahrain446N/A
43 Qatar442 17 points
44 Saudi Arabia430
45 Iran428 29 points
46 Oman418 27 points
47 Kuwait393N/A
48 Morocco358 48 points
49 Egypt330N/A
50 South Africa320N/A
Benchmarking participants
Moscow (Russia)612N/A
Madrid (Spain)549N/A
Quebec (Canada)547 9 points
Ontario (Canada)544 8 points
Andalusia (Spain)525 10 points
 Norway25517N/A
Dubai (United Arab Emirates)515 39 points
 Denmark26501N/A
Buenos Aires (Argentina)480N/A
Abu Dhabi (United Arab Emirates)414 10 points
 South Africa (English/Afrikaans/Zulu)27406N/A

Helpful pages

PIRLS 2011

Combining newly developed reading assessment passages and questions for 2011 with a selection of secure assessment passages and questions from 2001 and 2006, the study offered a state-of-the-art assessment of reading comprehension that allowed for measurement of changes since 2001. The international population for PIRLS 2011 consisted of students in the grade that represents four years of schooling, provided that the mean age at the time of testing was at least 9.5 years. In the 2011 cycle, prePIRLS (now known as PIRLS Literacy) was offered to assess basic reading skills as a bridge to PIRLS, for countries where most children are still developing fundamental reading skills at the end of the primary school cycle.28

RankCountryAveragescale scoreChangeover 5 years
1 Hong Kong571 7 points
2 Russia568 3 points
2 Finland568N/A
4 Singapore567 9 points
5 Northern Ireland558N/A
6 United States556 16 points
7 Denmark554 8 points
8 Croatia553N/A
8 Chinese Taipei553 18 points
10 Ireland552N/A
10 England552 13 points
12 Canada548N/A
13 Netherlands546 1 point
14 Czech Republic545N/A
15 Sweden542 7 points
16 Italy541 10 points
16 Germany541 7 points
16 Israel541 29 points
16 Portugal541N/A
20 Hungary539 12 points
21 Slovakia535 3 points
22 Bulgaria532 15 points
23 New Zealand531 1 point
24 Slovenia530 8 points
25 Austria529 9 points
26 Lithuania528 9 points
27 Australia527N/A
28 Poland526 7 points
29 France520 2 points
30 Spain513
31 Norway507 9 points
32 Belgium (French)506 6 points
33 Romania502 13 points
International average500
34 Georgia488 17 points
35 Malta477N/A
36 Trinidad and Tobago471 35 points
37 Azerbaijan462N/A
38 Iran457 36 points
39 Colombia448N/A
40 United Arab Emirates439N/A
41 Saudi Arabia430N/A
42 Indonesia428 23 points
43 Qatar425 72 points
44 Oman391N/A
45 Morocco310 13 points
6th grade participants
 Honduras29450N/A
 Morocco30424N/A
 Kuwait31419N/A
 Botswana32419N/A
Benchmarking participants
Florida (United States)569N/A
Ontario (Canada)552 3 points
Alberta (Canada)548 12 points
Quebec (Canada)538 5 points
Andalusia (Spain)515N/A
Dubai (United Arab Emirates)476N/A
 Malta (Maltese)457N/A
Abu Dhabi (United Arab Emirates)424N/A
 South Africa (English/Afrikaans)33421N/A

PIRLS 2006

PIRLS 2006 assessed a range of reading comprehension strategies for two major reading purposes: literary and informational. The student test of reading comprehension addressed four processes:

  • retrieval of explicitly stated information
  • making straightforward inferences
  • interpreting and integrating ideas and information
  • examination and evaluation of content, language, and textual elements.

PIRLS 2006 assessed students enrolled in the fourth grade.34

RankCountryAveragescale scoreChangeover 5 years
1 Russia565 37 points
2 Hong Kong564 36 points
3 Alberta (Canada)560N/A
4 Singapore558 30 points
4 British Columbia (Canada)558N/A
6 Luxembourg557N/A
7 Ontario (Canada)555N/A
8 Italy551 10 points
8 Hungary551N/A
10 Sweden549 12 points
11 Germany548 9 points
12 Netherlands547 7 points
12 Belgium (Flemish)547N/A
12 Bulgaria547 3 points
15 Denmark546N/A
16 Nova Scotia (Canada)542N/A
17 Latvia541 4 points
18 United States540 2 points
19 England539 14 points
20 Austria538N/A
21 Lithuania537 6 points
22 Chinese Taipei535N/A
23 Quebec (Canada)533N/A
24 New Zealand532 3 points
24 Slovakia532 15 points
26 Scotland527 1 point
27 France522 3 points
27 Slovenia522 20 points
29 Poland519N/A
30 Spain513N/A
31 Israel512 3 points
32 Iceland511 1 point
International average500
33 Moldova500 8 points
33 Belgium (French)500N/A
35 Norway498 1 point
36 Romania489 23 points
37 Georgia471N/A
38 Republic of Macedonia442
39 Trinidad and Tobago436N/A
40 Iran421N/A
41 Indonesia405N/A
42 Qatar353N/A
43 Kuwait330 66 points
44 Morocco323 27 points
45 South Africa302N/A

PIRLS 2001

The IEA Progress in International Reading Literacy Study (PIRLS) 2001 was the first cycle of assessments to measure trends in children's reading literacy achievement, and policy and practices related to literacy. The study examined three aspects of reading literacy: processes of comprehension, purposes for reading, and reading literacy behavior and attitudes. 35 countries took part in the first cycle where students enrolled in the fourth grade were assessed.35

RankCountryAveragescale score
1 Sweden561
2 Netherlands554
3 England553
4 Bulgaria550
5 Latvia545
6 Canada36544
7 Lithuania543
7 Hungary543
9 United States542
10 Italy541
11 Germany539
12 Czech Republic537
13 New Zealand529
14 Scotland528
14 Singapore528
14 Russia528
14 Hong Kong528
18 France525
19 Greece524
20 Slovakia518
21 Iceland512
21 Romania512
23 Israel509
24 Slovenia502
International average500
25 Norway499
26 Cyprus494
27 Moldova492
28 Turkey449
29 Republic of Macedonia442
30 Colombia422
31 Argentina420
32 Iran414
33 Kuwait396
34 Morocco350
35 Belize327

United States results by race and ethnicity

Race201637201138200639200140
ScoreScoreScoreScore
Asian591588567551
Multiracial578
White571575560565
US Average549556540542
Other545573
Hispanic525532518517
Black518522503502
American Indian/Alaska Native468

PIRLS assessment

The PIRLS study consists of a main survey that consists of a written reading comprehension test and a background questionnaire. The PIRLS Reading Development Group (RDG) and National Research Coordinators (NRCs) from the participating countries collaborate to develop the reading assessments. The assessment focuses on three main areas of literacy: process of comprehension, purposes for reading, and reading behaviors and attitudes. The background questionnaire is used to determine the reading behaviors and attitudes. The written test is designed to address the process of comprehension and the purposes for reading. There are two purposes for reading that are examined in this study: reading for literary experience and reading to acquire and use information. Each student receives 80 minutes to complete two passages and then time to complete the survey. There are a total of 8 passage. Four passages are for each purpose of reading. "With eight reading passages in total, but just two to be given to any one student, passages and their accompanying items were assigned to student test booklets according to a matrix sampling plan. The eight passages were distributed across 10 booklets, two per booklet, so that passages were paired together in a booklet in as many different ways as possible."41 The PIRLS target population is the grade that represents four years of schooling, counting from the first year of ISCED Level 1, which corresponds to the fourth grade in most countries. To better match the assessment to the achievement level of students, countries have the option of administering PIRLS or PIRLS Literacy at the fifth or sixth grade.

Background questionnaire

Given to:

  • Home/parents—This questionnaire includes questions about "students' early reading experiences, child-parent literacy interactions, parents' reading habits and attitudes, home-school connections, and demographic and socioeconomic indicators."
  • Students—This questionnaire includes questions about "instructional experiences, self-perception and attitudes towards reading, out-of-school reading habits, computer use, home literacy resources, and basic demographic information."
  • Teachers—This questionnaire includes questions about "characteristics of the class tested, instructional activities for teaching reading, classroom resources, assessment practices, and about their education, training, and opportunities for professional development."
  • Schools—This questionnaire includes questions about "enrollment and school characteristics, school organization for reading instruction, school staffing and resources, home-school connections, and the school environment."42

Participating organizations

Participating countries

CountryYears
 Albania2021
 Argentina2001, 201643
 Australia2011, 2016, 2021
 Austria2006, 2011, 2016, 2021
 Azerbaijan2011, 2016, 2021
 Bahrain2016, 2021
 Belize2001
 Belgium2006,44 2011,45 2016,46 202147
 Botswana201148
 Brazil2021
 Bulgaria2001, 2006, 2011, 2016, 2021
 Canada2001,49 2006,50 2011, 2016, 202151
 Chile2016
 Chinese Taipei2006, 2011, 2016, 2021
 Colombia2001, 2011
 Croatia2011, 2021
 Cyprus2006, 2021
 Czech Republic2001, 2011, 2016, 2021
 Denmark2006, 2011, 2016, 2021
 Egypt2016, 2021
 England2001, 2006, 2011, 2016, 2021
 Finland2011, 2016, 2021
 France2001, 2006, 2011, 2016, 2021
 Georgia2006, 2011, 2016, 2021
 Germany2001, 2006, 2011, 2016, 2021
 Greece2001
 Honduras201152
 Hong Kong2001, 2006, 2011, 2016, 2021
 Hungary2001, 2006, 2011, 2016, 2021
 Iceland2001, 2006
 Indonesia2006, 2011
 Iran2001, 2006, 2011, 2016, 2021
 Ireland2011, 2016, 2021
 Israel2001, 2006, 2011, 2016, 2021
 Italy2001, 2006, 2011, 2016, 2021
 Jordan2021
 Kazakhstan2016, 2021
 Kosovo2021
 Kuwait2001, 2006, 2011,53 2016
 Latvia2001, 2006, 2016, 2021
 Lithuania2001, 2006, 2011, 2016, 2021
 Macau2016, 2021
 Malta2011, 2016, 2021
 Moldova2001, 2006
 Montenegro2021
 Morocco2001, 2006, 2011, 2016, 2021
 Netherlands2001, 2006, 2011, 2016, 2021
 New Zealand2001, 2006, 2011, 2016, 2021
 North Macedonia2001, 2006, 2021
 Northern Ireland2011, 2016, 2021
 Norway2001, 2006, 2011, 2016, 2021
 Oman2011, 2016, 2021
 Poland2021
 Portugal2011, 2016, 2021
 Qatar2006, 2011, 2016, 2021
 Romania2001, 2006, 2011
 Russia2001, 2006, 2011, 2016, 2021
 Saudi Arabia2011, 2016, 2021
 Scotland2001, 2006
 Serbia2021
 Singapore2001, 2006, 2011, 2016, 2021
 Slovakia2001, 2006, 2011, 2016, 2021
 Slovenia2001, 2006, 2011, 2016, 2021
 South Africa2006, 2011,54 2016, 2021
 Spain2006, 2011, 2016, 2021
 Sweden2001, 2006, 2011, 2016, 2021
 Trinidad and Tobago2006, 2011, 2016
 Turkey2001, 2021
 United Arab Emirates2011, 2016, 2021
 United States2001, 2006, 2011, 2016, 2021
 Uzbekistan2021

See also

References

  1. "PIRLS. Progress in International Reading Literacy Study - IEA". www.iea.nl. Archived from the original on 2016-11-21. Retrieved 2016-11-21. https://web.archive.org/web/20161121234828/http://www.iea.nl/pirls-progress-international-reading-literacy-study

  2. "PIRLS 2001 International Report, Appendix A" (PDF). Retrieved 2018-08-10. http://timss.bc.edu/pirls2001i/pdf/P1_IR_A.pdf

  3. "PIRLS 2021 International Results in Reading". pirls2021.org. Retrieved 11 September 2023. https://pirls2021.org/results

  4. "PIRLS. Progress in International Reading Literacy Study - IEA". www.iea.nl. Archived from the original on 2016-11-21. Retrieved 2016-11-21. https://web.archive.org/web/20161121234828/http://www.iea.nl/pirls-progress-international-reading-literacy-study

  5. "Other IEA studies - IEA". www.iea.nl. http://www.iea.nl/other-iea-studies

  6. "PIRLS. Progress in International Reading Literacy Study - IEA". www.iea.nl. Archived from the original on 2016-11-21. Retrieved 2016-11-21. https://web.archive.org/web/20161121234828/http://www.iea.nl/pirls-progress-international-reading-literacy-study

  7. "PIRLS 2021 | IEA.nl". www.iea.nl. Retrieved 2022-11-06. https://www.iea.nl/studies/iea/pirls/2021

  8. "Info" (PDF). www.iea.nl. http://www.iea.nl/sites/default/files/publications/Electronic_versions/PIRLS%202021%20Brochure.pdf

  9. "PIRLS and TIMSS". paises.qedu.org.br. Retrieved 2025-02-19. https://paises.qedu.org.br/en/pirls-and-timss/

  10. Assessed one year later than originally scheduled

  11. Assessed one year later than originally scheduled

  12. 5th grade students

  13. Assessed one year later than originally scheduled

  14. Assessed one year later than originally scheduled

  15. Assessed one year later than originally scheduled

  16. Assessed one year later than originally scheduled

  17. Assessed one year later than originally scheduled

  18. 6th grade students

  19. "PIRLS 2016". https://timssandpirls.bc.edu/pirls2016/

  20. "PIRLS 2016 International results in reading". http://timssandpirls.bc.edu/pirls2016/international-results/pirls/student-achievement/

  21. "Trends in reading results by country". http://timssandpirls.bc.edu/pirls2016/international-results/pirls/student-achievement/trends-in-reading-achievement/

  22. "2016 PIRLS Encyclopedia". http://timssandpirls.bc.edu/pirls2016/encyclopedia/

  23. "PIRLS 2016 Student Achievement Overview". http://timssandpirls.bc.edu/pirls2016/international-results/pirls/student-achievement/

  24. 5th grade students

  25. 4th grade students

  26. 3rd grade students

  27. 5th grade students

  28. "PIRLS 2011 International Report, Chapter 1" (PDF). timssandpirls.bc.edu. Archived (PDF) from the original on 26 May 2021. Retrieved 13 December 2021. https://timssandpirls.bc.edu/pirls2011/downloads/P11_IR_Chapter1.pdf

  29. 6th grade students

  30. 6th grade students

  31. 6th grade students

  32. 6th grade students

  33. 5th grade students

  34. "PIRLS 2006 International Report, Chapter 1" (PDF). timssandpirls.bc.edu. Archived (PDF) from the original on 27 March 2016. Retrieved 13 December 2021. https://timssandpirls.bc.edu/PDF/P06_IR_Ch1.pdf

  35. "PIRLS 2001 International Report, Chapter 1" (PDF). timssandpirls.bc.edu. Archived (PDF) from the original on 20 May 2019. Retrieved 13 December 2021. https://timssandpirls.bc.edu/pirls2001i/pdf/P1_IR_Ch01.pdf

  36. Represented by Ontario (548) and Quebec (487)

  37. "Reading Achievement of U.S. Fourth-Grade Students in an International Context" (PDF). https://nces.ed.gov/pubs2018/2018017.pdf

  38. "PROGRESS IN INTERNATIONAL READING LITERACY STUDY (PIRLS)". https://nces.ed.gov/surveys/pirls/figure11_4.asp

  39. "The Reading Literacy of U.S. Fourth-Grade Students in an International Context" (PDF). https://nces.ed.gov/pubs2008/2008017.pdf

  40. "International Comparisons in Fourth-Grade Reading Literacy" (PDF). https://nces.ed.gov/pubs2003/2003073.pdf

  41. "PIRLS 2001 International Report, Appendix A" (PDF). Retrieved 2018-08-10. http://timss.bc.edu/pirls2001i/pdf/P1_IR_A.pdf

  42. "PIRLS 2001 International Report, Appendix A" (PDF). Retrieved 2018-08-10. http://timss.bc.edu/pirls2001i/pdf/P1_IR_A.pdf

  43. as Buenos Aires (benchmarking only)

  44. French and Flemish separated

  45. French only

  46. French and Flemish separated

  47. French and Flemish separated

  48. 6th grade only

  49. represented by Ontario and Quebec

  50. as Alberta, British Columbia, Nova Scotia, Ontario, and Quebec

  51. as Alberta, British Columbia, Newfoundland and Labrador and Quebec (benchmarking only)

  52. 6th grade only

  53. 6th grade only

  54. English/Afrikaans (benchmarking only)