NPS 21st Century Pilot Study


About the NPS21 Project

Home Page

Data Collection Techniques

Original 2001 Surveys:
USEIT Teacher survey
USEIT Student survey

Reports/Reporting Schedule


Damian Bebell, Ph.D.
Assistant Research Professor
at bebell@bc.edu

Methodology/Data Collection Procedures

Teaching and learning practices, particularly those using technology, will be thoroughly documented in the pilot classroom over the course of the implementation while comparisons to both historic teaching and learning practices through the comparison of the district's 2001 USEiT survey results as well as to current traditional practices through the participation of a traditional classroom to serve a control setting. In other words, the current study will compare student and teacher practices in the pilot classroom to historic practices in Newton (documented by the USEiT surveys) as well as to current practices in traditional middle school classrooms via a "control group" setting without increased technology resources. Thus, most data collection procedures will be conducted in both the pilot and comparison/traditional classrooms. Given that measures of teaching practices and technology use will be collected before student laptops are provided, the study can be best classified as a pre/post comparative study.

A number of different methodological approaches and tools will be used in concert to best capture and address the wide variety of potential outcomes inherent from the investment of state-of-the-art digital learning tools in a pilot middle school classroom. The research design proposed herein includes both quantitative and qualitative methods measuring practices and attitudes from students and teachers over the course of the pilot implementation period. Each proposed data collection procedure is detailed individually below:


Student Survey

All students participating in the 21st Century Classroom and the traditional/control classroom will be surveyed via a state of the art computer-based student survey on two occasions during the 2009/2010 school year. Specifically, the web based survey will be given to both pilot and control students within the first month of the school year to record baseline conditions and again in the last month of the school year. The survey will be pre-populated with student names and information to decrease the time needed by students to complete the survey (approximately 20 min.) and ensure a high response rate through the real-time monitoring of survey data via email and the web. A major advantage of pre-populated teacher rosters is that the evaluation team and project administrators can monitor the survey responses and completion rates in real time through a password-protected web site or via daily email updates. It should be noted that student names and school information are only used for internal record keeping purposes and in the sign-in process, and all student responses and data will be kept strictly anonymous and confidential. Once data is collected, student names will be deleted from all files.

Beginning in summer of 2009, the Boston College team will work collaboratively with Newton's administration to refine the content and scope of the student survey instrument through the use of past surveys and item banks to best meet the needs of the current investigation. Given that Newton was a participant in the 2001 USEiT study, which measured a wide variety of student practices and attitudes across the district, there is a unique opportunity to examine the degree of change in students' practices and attitudes since that time by including a selection of the same survey questions used in the previous study.

Given the targeted outcomes established by the district, the student survey will include measures of students' access to technology in school, use of technology in school across subject areas, personal comfort level with technology, attitudes and perspectives towards technology and digital content, access to technology at home, and a variety of potential uses of technology at home. In addition, the student survey will be designed to measure shifts in students' media literacy as well as gather data regarding students' use of classroom time and personal organization both before and after pilot initiative.


Teacher Survey

All teachers participating in the 21st Century Classroom pilot as well as the traditional control classroom will complete a facilitated teacher survey on at least three occasions during the 2009/2010 school year. Specifically, teachers will complete the facilitated survey very early in the school year, ideally within the first weeks of the school year to provide an approximation of baseline conditions in each setting. Teachers will again be queried midway through the school year in December 2009 and again in the last month of the school year in May 2010 to demonstrate how changes in digital resources and training have impacted teaching practices.

Like the student survey, beginning in summer of 2009 the Boston College team will work collaboratively with Newton leadership to refine the content and scope of the teacher survey through the use of past surveys and item banks to best suit the needs of the current investigation. Again, given that Newton was a participant in the 2001 USEiT study (which measured a wide variety of teacher practices and attitudes across the district) there is a unique opportunity to examine the degree of change in teacher practices and attitudes since that time by including a selection of the USEiT teacher survey items.

Given the targeted outcomes established by the district, the teacher survey will include item sets dedicated to capturing the variety and extent of teachers' technology use, teachers' attitude toward technology, teaching, and learning, as well as teachers' beliefs on student motivation and engagement. Teachers will also complete brief item sets that measure more general pedagogical practices and classroom practices. Collectively, these items will help demonstrate evidence for the types of fundamental changes in the approach and delivery of the curriculum (as well as various aspects of teacher/student interactions). Looking across the teacher surveys over time from both pilot and control classrooms, the survey will provide documentation on the impacts of the pilot initiative on teacher practices, student practices, beliefs and attitudes. It should also be noted that the teacher survey will include a section that focuses on the professional development and training they have received related to the use of digital learning resources, the extent to which they have applied what they learned, and additional support that they require. Similarly, a section of the survey will focus on technical aspects of the program (e.g., slow network, difficulty accessing on-line materials, timeliness of technical support, etc.) that may be impeding the full effects of the initiative. In addition, the teacher survey will be designed to capture teachers' perception of the impacts of 1:1 computing on their students. Survey items will serve to document shifts in students' media literacy, use of classroom time, personal organization, as well as several indicators of student engagement, student achievement and discipline.


Student Focus Groups

Near the completion of the 2009/2010 school year, a random sample of 5–8 students from the pilot classroom and the control setting will be asked to participate in two separate student focus groups led by a senior member of the BC research team. In each student focus group meeting, which will be approximately 30 minutes in length, students will be asked to consider their experiences in the classroom during the past year. Given the targeted outcomes of the pilot program, there will be a focus towards the impact of digital age resources on teaching and learning, however students' general attitudes towards technology and schooling will also be addressed. The student focus group will serve to compliment and triangulate survey results as well as provide more in-depth information on how students' 21st century expectations and skills were met during the 2009/2010 school year across both settings.


Teacher Interview

Although all participating teachers will be surveyed on multiple occasions over the course of the evaluation, teacher interviews will also be performed to gain a richer and more in-depth understanding of how digital learning resources have impacted students' achievement, behavior, as well as general teaching and learning practices. Particularly, near the end of the pilot implementation, the pilot teaching team will be given an opportunity to reflect on the program through the interview process and provide more vivid and anecdotal evidence to compliment the quantitative evaluation data. In addition, pilot and control classroom teachers will be formally interviewed about the variety of ways in which they employed technology in and out of schools. The interviews allow the evaluation team an opportunity to query and clarify initial evaluation findings and assumptions on an as needed basis.


Teacher Weblogs/e-journaling

At various intervals during the 2009/2010 school year, the research team will use a secure and private weblog for pilot and control classroom teachers to briefly note various aspects of their experiences that may be of interest. Given the small number of teacher participants, the expectation is that the project administration, teacher participants, and the BC team will engage in fairly open and frequent communication. Some form of teacher weblog or e-journaling site will be designed in collaboration with project administration to provide a convenient, flexible, and easily adaptable tool to gather impressions and experiences of the pilot classroom teacher (and to a lesser extent the control teacher). For example, given the literature documenting the substantial time invested by teachers in transforming their teaching from traditional methods to those afforded by technology-rich classroom, the administrative and research team may ask teachers to periodically record and reflect on their personal time investments in transitioning to new resources.


Student Drawing (Pilot and Control/Traditional, Sept. 2009 and Late May 2010)

As demonstrated in previous educational research, student drawings provide a unique and surprisingly valuable perspective on the impacts of technology as perceived by the students themselves (Bessett, 2008; Russell, Bebell & Higgins, 2004; Russell, Bebell, Cowan & Corbelli, 2003). Although student drawings are an unusual tool for collecting information about students and their classrooms, student drawings provide a rich descriptive examination into students' perspective that is often neglected in more traditional data sources. To triangulate information provided by the teacher surveys, student surveys, and teacher interviews about how technology is applied, all students in pilot and control classrooms will participate in a pre and post student laptop drawing exercise at the beginning and completion of the 2009/2010 school year.


Classroom Observations (Pilot and Control/Traditional)

A notable absence in the literature and discussions of many emerging digital learning tools are examples of the ways teachers and students actually use the technology-enriched environment to approach the curriculum. In the current study, formal classroom observations will be used to develop a deeper basic understanding of the ways in which the new technology resources are being used throughout the school day. In addition, by conducting observations in both the pilot setting and control/traditional setting over the duration of the school year, the observation results will also serve to help document the impacts of the digital learning tools and potential changes in teaching and learning practices.

The classroom observations would be focused chiefly between December 2009 and May 2010 across the pilot and control/traditional classrooms. As we have done in past work, the research team of trained observers will employ a computer-based FileMaker database note taking tool that allows researchers to capture and categorize observation notes while in the field. During the observations, students' engagement level, the number of students working with technology, the number of students working independently, in pairs, in small groups, or in large groups and the role of the teacher are quantitatively recorded every ten minutes via an automated observational checklist. In addition, observers record narrative accounts of the activities occurring throughout the single class observation, with a specific emphasis on teacher-student interactions, student–student interactions, uses of technology, and student engagement.

Emergent analytic coding and content analyses of the observation notes in addition to traditional qualitative note taking software that features a prompt to the observer to record a variety of technology and curricular information at fixed ten-minute intervals including:


Analysis of School Records

At the end of the one-year pilot implementation, the evaluation team will work in conjunction with school and district personnel to collect and analyze various data sources from the pilot and control classroom. Data sources including attendance, tardiness and various disciplinary referrals will be collected to assess the impacts of the wireless laptop initiative through comparisons to past historic levels and to the control classroom and school averages.


Student Test Score Analyses

Given that one of the targeted program outcomes of the 21st Century Classroom is to improve student achievement, a central component of the evaluation is the investigation of students' pilot participation and subsequent technology use on their academic performance. As one component of the overall 21st Century Classroom pilot investigation, the evaluation team will undertake an examination of the impacts of student and teacher practices on student achievement (as measured by the 2010 MCAS administration) for all pilot and comparison students.