In my experience, the concept of 'closing the loop' has been one of the most problematic issues when doing any sort of course or program assessment/ evaluation. While I agree with Dr. Enger that ensuring the accurate measurement of the achievement of objectives is of primary importance to the effectiveness of a program evaluation plan, the fact that these measures are accurate does not necessarily mean that the ‘loop’ has been ‘closed.’
If, for example, an (accurate and well designed) assessment indicates that students are falling short in the achievement of a particular objective, the fact that we have accurately diagnosed that shortfall does not in itself ‘close the loop.’ The ‘loop’ (as it were) is not ‘closed’ until we have (in Allen’s words) “used the results to improve the program.”
Although undoubtedly useful, Allen’s six step process represents for me a perfect example of a ‘black box’ model, where a key (and arguably, the most important) element is left relatively unexplored while related steps are meticulously defined. Allen frankly admits that “step 5 is probably the hardest”, but fails to even remotely describe exactly how “good assessment leads to collective reflection and action.”
Whenever I am developing an assessment/evaluation instrument, I find it helpful to think in terms of “actionable items”, that is, where the measure itself indicates possible steps toward remediation, if necessary. Similarly, the inverse of the actionable items concept is also true: assessments that incorporate measurements that do not (or cannot) lead directly to an action item are time wasted, and opportunity lost.
One example of this that comes to mind is the common practice of measuring online students’ opinion of the effective of the Learning Management System. Since, in most cases, there is absolutely no possibility that the results of the query will drive a change (most universities are bound by contract and have no intention of changing their LMS), the effort is futile, and more importantly, the opportunity to ask a question that actually could lead to effective change is wasted.
Closing the loop is more than a black box, and goes beyond simple validity. Absent this crucial process, however, we are left only with the knowledge of exactly how skinny our pig is.
Just a thought…
I found Dan’s post interesting and I agree that by simply identifying the gaps between the program objectives and the assessment findings does not “close the loop”. However, closing the loop can be and most often is the most difficult and time consuming step in the assessment process.
ReplyDeleteBecause my involvement with assessment primarily involves campus assessment versus program assessment my perspective is slightly different. The findings collected and disseminated from my office often lead to the formation of committees to help address specific issues identified in the results. The recommendations brought forth by such committees can and most often involve changes to policy and procedure for the entire campus, this can create a whole host of other issues.
Although I am unfamiliar with the program assessment specifically, I only assume that the notion of closing the loop is cumbersome and at times difficult as well.
My thoughts………
Closing the loop is the reason for assessment: to support and promote student learning. We go through all of the work to assess, but if we don't use the results, what good is it? As we all know (all too well, it seems) higher education is slow to change, and because of the structure (which is based on academic freedom), it is very difficult to change -- assessment requires time and collaboration. Collaboration is a skill that we need to learn and practice, but is one of the most difficult to implement on a college campus, particularly at larger state institutions. Collaboration can occur more effectively at collegial institutions of higher education. Anyway, I'm rambling, here (what's new, huh), all to say that closing the loop is essential for effective change, but nearly impossible, due to our workloads and the need to collaborate in an environment that requires independent thought and initiative at the same time.
ReplyDeleteThe thing I have noticed with assessments indicating they have "closed the loop" is in fact, they have "opened the loop". I am not sure the reasoning but have my suspisions it it due to reluctance to "do another job". From my perspective, increasing the validity and reliability of the data will provide an easy transition for change to be implemented. Allan gives great examples in chapter 4. Dan nailed it with the "how skinny the pig" statement. It just cannot be simple validity to provide the feedback for change. From my experience, growth occurs when assessment done with a purpose. When the loop is clearly closed, the needed information is present to allow the teacher to effectively improve the course or program.
ReplyDeleteKory>
ReplyDeleteI think you've exposed another important part of the process: faculty fatigue. If program assessments are simply viewed as a long list of (excuse the mixed metaphor) "open loops to jump through", buy-in becomes more and more difficult. The open dialogue the Prof. E refers to is a critical component to ensure that efforts come to fruition (in the form of change, improvement, or even validation). Absent dialogue and buy-in, assessment becomes "just another job" that faculty avoid, and frankly, often view with suspicion.