Applying the Framework in Education
It should be clear the most important questions to address when considering database solutions are: What are the goals and how is achievement quantified and measured (Schmoker, 2003)? For an educational institution, a broad goal is to promote learning. A more specific goal currently pursued as a result of reform legislation in K-12 and increased competitive pressures in higher education and online learning environments is to improve the quality of learning. McTighe and Thomas (2003) categorized educational improvement initiatives as either classroom or systemic, which may be useful in some situations.
No matter what goal is identified, however, it is imperative to identify appropriate ways to measure goal achievement (Carr, 2003). Monson (2002) suggested learning can only be effectively measured on an individual basis over the long-term and that educators need to engage all stakeholders in order to identify expectations. According to Marzano (2003, pp. 53-60), one of the most common mistakes educators make when defining measures of learning is to use indirect measures, such as standardized test scores that do not take into account the actual content used among other variables. Others have pointed out that learners are often tested on knowledge they have yet to learn; suggesting tests are not aligned with curricula (Doyle, 2003). Porter (2002) recognized the importance of measuring the content of instruction, the content of instructional material and how well they are in alignment using content analysis tools. Popham (2003) suggested beneficial data comes from instructionally useful tests that are characterized by the significance, teachability and describability of the skill being tested. It is imperative that educators carefully identify the measures that directly affect goal achievement and understand that using irrelevant measures and inaccurate data sets the stage for faulty decision-making leading to potentially disastrous results.
Identify the Data
With the goals and measures defined, the next questions are: What people and products directly and indirectly influence goal achievement and what characteristics are important? In the initial analysis, it is important to first identify goal critical people and products and the characteristics that potentially affect how well and effectively the goal is achieved. Many of the characteristics deemed important may further depend on the institution’s philosophy and beliefs in how learning occurs.
For example, to promote learning, the institution naturally needs learners and teachers. It also needs administrative and learning support staff. Critical products may include textbooks, course materials and media resources. While it is a relatively simple task to identify general characteristics of people and products, such as names, addresses and credentials, it is not so easy to identify affecting characteristics requiring a deeper look at institutional philosophy and beliefs. Does the institution believe that personality, teaching and learning styles, attendance, time on task, environmental conditions, physical and emotional well-being or degree of social interaction affect a learner’s ability to learn? Bernhardt (2003) identified four data classifications that help identify applicable characteristics including demographics, student learning data, perceptions data and school processes data.
Providing a means of classifying data enables analysts to disaggregate the data and intersect multiple categories in meaningful ways for a rich, complex picture of the institution and which factors influence goal achievement. Regardless of how the data is classified, it is important, during this stage, to refrain from focusing on cost, specific technologies, the current state of existing processes and data, or what may or may not be possible. It is also important to identify the people and products and the characteristics deemed critical as completely as possible, in order to get it right the first time and allow for a phased-in implementation if budget constraints ultimately require it.
Identify the Data Sources
With the data defined, it is possible to identify data sources. In a learning environment, general characteristics about learners are captured during enrollment and registration. Learner characteristics deemed critical to goal achievement may come from a variety of sources including the learners themselves, teachers, administrators, counselors, parents, possibly even bus drivers and crossing guards. Product characteristics may come from instructional designers, publishers, teachers or vendors. Teacher characteristics may come from classroom observations, learner surveys, professional development records and the teachers themselves. While few institutions are in a position to equip everyone with data entry devices, it is reasonable to consider Web-based access for these external sources if significant amounts of data are expected to come from them.
Identify Existing Data and Processes
With the data sources identified, it is time to evaluate existing processes, inventory what data is and is not currently being collected, evaluate the quality, accessibility and usability of existing data, identify the people who are in the best position to oversee each process and evaluate possible solutions. Although Brittain (2003) described the current state of educational data stores as “data rich and information poor” due to the incompatibility of the systems in use, it is important not to focus on the quantity, but the quality of existing data. It is equally important not to focus on system incompatibilities, but rather on ways quality data can be accessed and appropriately shared. For many districts and institutions that use a variety of disparate database systems, data warehouses may offer the best solution since they can access data from a variety of systems and allow for both longitudinal and latitudinal analysis (Cohen, 2003, pp. 53-56). Before a decision can be made, however, it is necessary to evaluate the quality of the data and if the existing processes align with those identified in the analysis.
In many early business implementations, process identification and evaluation was completely omitted, which at worst, resulted in project derailment or at best, resulted in similar incompatibility woes described by Brittain (2003). Since analyzing and evaluating existing processes requires an intimate knowledge of institutional goals and individual roles, it cannot be adequately done without full cooperation of all staff on all levels. By nature, people are protective of their respective areas of expertise and responsibility. It is easy to alienate people who become threatened by having their actions scrutinized under the proverbial microscope, their routines disrupted and overhauled, and when the data can be used against them (Dawson, 2002; Doyle, 2003). For this reason, it is imperative that cultural members feel respected and appreciated for their knowledge and contributions and do not feel threatened in any way. They must also be trained in the new processes and systems. When they are confident and believe in their own significance to the overall success of the institution, loyalty ensues, which leads to frankness and a belief in the worthiness of the change.
During this phase, it is important to have a good idea what database technology alternatives exist. Since the whole solution depends on the capabilities of the technology chosen and the desired capabilities of the system depend largely on identified processes, it makes sense to analyze these areas in tandem to ensure a good fit and revise the plan as needed. In corporate settings, had decision-makers taken the time to align technology choices with their cultures, this would have been a relatively easy part of the analysis when converting from manual systems since interoperability was not an issue. For executives who had become prisoners of their early decisions, identifying their technology needs was not a simple task. Having legacy systems and multiple copies of unsynchronized data in a variety of formats that contained incomplete and inaccurate data proved problematic.
McIntire (2002) articulated this problem in his description of his efforts to make data more accessible to decision-makers for strategic planning. He was quickly discouraged upon learning that data was stored in multiple unsynchronized spreadsheets, with no one knowing which one was the most accurate. Educators are commonly faced with this dilemma as they struggle to figure out what data they need, where it exists and how they can access it. Data warehouses may fill the void making it possible to connect disparate systems in ways previously impossible (Rudner & Boston, 2003). Levine’s (2002) article describes the School Interoperability Framework (SIF), which is another alternative that uses XML as the data exchange format as a way to share information between systems. Regardless which technology is chosen, however, the old adage “garbage in garbage out” applies, with data integrity being the driving factor.
With little to no reliable data to base decisions on, it is difficult to know when to cut losses and scrap ineffective systems and often results in a compounding of the problem while attempting to circumvent weaknesses. For Franklin Howell Schools (Lafee, 2002) relying on a system that contained years of inaccurate data proved disastrous before they realized they needed to scrap the existing system and start over. After learning from the auditors projections were overstated by $7.5 million, the district had to repay the funds and fire the people they anticipated needing to fill the projected demand. A good indicator of the need to scrap a system is when the vendor no longer exists and when the system no longer does the job effectively (Levine, 2002). Whether an institution needs to add a new layer to their existing infrastructure or start anew, however, the key point to remember is that institutional goals should underpin the data requirements, which underpins technology choices.