Is the "Brain Drain" a Myth?

Haga, C. (2010, January 30). Expert: Brain drain is a myth. The Forum of Fargo-Moorhead, pp. A1, A10.

This article on the front page of today’s Fargo Forum caught my eye since it relates to the book we will soon be discussing, “Hollow Out The Middle”. Entitled “Expert: Brain Drain Is A Myth” the article claims that we should not worry that small, rural towns across the country are being emptied of their best and brightest young adults who seek better jobs, educational opportunities and more exciting experiences (“brain drain”). Ben Winchester, a research fellow in Extension Center for Community Vitality at the University of Minnesota, studied the changing demographics in a nine county in west-central Minnesota. He found that the alarming phenomenon of brain drain is counter balanced by the movement of a somewhat older group (35-44) from urban areas back to rural areas. Those migrating back have college degrees, good incomes, school -aged children and are often assuming active roles in community service. A similar demographic trend is occurring in Nebraska. The appeal of a rural versus a metropolitan, community for this age group seems to be lower cost of living, a simpler paced lifestyle, and the safety and security provided by a smaller community. From my experience, many of the adults who return to rural America consider it a better place to raise their families. Winchester asserts that we need to refocus our thoughts and consider the current changing demographics as a “brain gain” rather than a “brain drain”, suggesting that it presents rural communities opportunities for growth.

The article included a response to Winchester’s research by Richard Rathge, director of the North Dakota Data Center at NDSU. I strongly agree with his warning that it is a mistake to compare the demographic trend data of Minnesota with what is happening in North Dakota. Because it has many lakes, recreation, and scenic attractions, Minnesota has much more appeal to this population group than North Dakota. After all, it seems that most any “rural” Minnesota community is only minutes away from a Target, shopping center, and popular restaurant, which some of the people I associate with would argue are feasible criteria for the measurement of “rural”. Who wouldn’t want to live on a lake yet have all the conveniences of the “urban” city nearby. The same amenities seem to be true in states like Colorado, Montana, and Wyoming. As a person who grew up in “rural” North Dakota, I know that the rural scenarios of the more popular scenic states are very different than those of the plains area. Of course, the next question would be why Nebraska is finding similar data. I think Winchester’s assessment findings are significant and call for further research to determine if indeed the study results are consistent with rural demographic trends across the nation.

Carla Gross

The BEAR Assessment System

Wilson, M., & Scalise, K. (2006). Assessment to improve learning in higher education: The BEAR Assessment System. [Article]. Higher Education, 52(4), 635-663.

Abstract: This paper discusses how assessment practices in higher education can improve or hinder learning. An example is given to illustrate some common educational practices that may be contributing to underpreparation and underperformance of students. Elements of effective learning environments that may better address underlying metacognitive issues are discussed. The principles of the Berkeley Evaluation & Assessment Research Assessment (BEAR) System are introduced, and their use to improve learning is described in the context of the UC Berkeley ChemQuery project.

This article describes a threefold approach to successfully ensuring student learning. First, students need to understand the measures on which they will be evaluated. Second, students need to know where they stand on these measures. Lastly, students need to know how they can improve. The article illustrates the importance of feedback to students by following a student through an eighteen week computer science course sans instructor feedback. The student is frustrated, unclear of the instructor expectations, and unable to master the coursework. The authors offer embedded assessment as a tool for generating diagnostic information and feedback to students. The system described is the Berkeley Evaluation and Assessment Research (BEAR) used to assess, interpret, monitor, and respond to student performance. The system focuses on three principles. The first principle is developmental perspective used to assess development of understanding of concepts over time. The second principles explains the importance of matching instruction and assessment. It is suggested that instructors develop instructional materials and assessments at the same time to increase relevancy. The third principle focuses on quality evidence using item response modeling to map student performance. The last principle underscores the need for instructors to continually review and modify assessment practices.

This was an excellent overview a an assessment methods using multiple theories of learning and focused on improving formative assessment.

Assessment Conference for AACSB

AACSB Assessment Conference

I just made my reservations today to attend this conference in March. Dr. Angelo (of Angelo & Cross) will be the plenary speaker, and I am excited to get to meet folks from other AACSB schools to learn from them what is working best and what are the trends. It's rare I get to attend a conference that is so timely to what I am studying and working with.

Here is a description of what he is speaking to at the conference:

At the 2010 Assessment Conference in New York City, Dr. Tom Angelo, internationally renowned expert on assessment with best known publication Classroom Assessment Techniques: A Handbook for College Teachers (more than 75,000 copies in print), focuses on using research-based, practical options in assessment to promote deep learning—and how “less can often yield more” in this context. Here are some of his thoughts on the concept of his session, issues many schools face regarding assessment, and his advice for schools seeking accreditation.

With consulting experience at more than 250 institutions worldwide, have you found any common issues causing schools around the world to struggle with building assessment programs?

One major common issue most of us struggle with is the “add on” problem. Very few curricula are designed and implemented with assessment and evaluation built in. Consequently, we end up trying to add on assessment to programs already in place. Like adding plumbing and wiring to buildings built long ago, it’s possible, but much more difficult and expensive than simply building them into a new structure. A second common issue is finding ways to engage academic staff in assessment efforts over the long term. A third challenge is maintaining assessment programs in the years between AACSB visits. I see all three of these issues as interrelated.

What advice would you give to b-schools seeking AACSB accreditation that may or may not have an assessment model in place?

To paraphrase John F. Kennedy: “Ask not how you can do the least possible required to satisfy AACSB. Rather, ask how doing assessment can help your institution improve student learning outcomes, optimize teaching time, manage or lower costs, and raise staff morale and institutional reputation.”

You are known around the world as a renowned expert on assessment—how did you first get involved with the “movement?” And, what about assessment sparked your interest and research?

However well known I might or might not be, I got involved in the assessment “movement” mainly by chance. I happened to be in just the right place at just the right moment in the mid-1980s. My interest and work in assessment started much earlier, however, sparked by my teaching experiences. I’ve always been a bit obsessed with finding out how well students are learning what I’m trying to teach—and with exploring efficient ways to improve those outcomes.

What can conference participants anticipate to learn from your session?

Based on evaluations of past sessions, I can predict that those who participate actively will take away a few strategies or techniques, along with a new insight or two, to help them improve returns on their investment in assessment. They’ll also get suggestions for engaging their colleagues in assessment. And, since my sessions are highly interactive, they are likely to learn from other participants, as well.

The Chronicle: Tranparency in Accreditiation

I like the way this article takes universities to task for focusing too much on easily obtainable data that does not necessarily reflect the student achievement (what the article calls "inputs").

The text is below, the link is here:
http://chronicle.com/article/Education-Department-Official/63730/

January 26, 2010

Education Department Official Calls for More Transparency in Accreditation
By Eric Kelderman

Washington

Martha J. Kanter, the No. 2 official in the U.S. Education Department, took higher-education accrediting organizations to task on Tuesday for being too secretive about how they assess colleges and for using outmoded standards that don't give enough weight to measuring student learning.

"Accreditation isn't transparent enough, it just isn't," Ms. Kanter said here at the annual meeting of the Council for Higher Education Accreditation. "And it takes too long."

In her remarks, Ms. Kanter, the under secretary of education, said accreditation was a crucial part of maintaining quality in higher education as the United States strives to attain President Obama's goal of being atop the world by 2020 in terms of the share of its residents with college degrees. The council, an association of about 3,000 accredited colleges and universities, recognizes 59 accrediting organizations; in that sense, it accredits the accreditors.

The organizations that are responsible for assuring quality in higher education must consider whether their processes are really helping institutions improve and whether they are focusing too much on "inputs," such as the amount of time that students spend in class, and too little on what they have learned.

Accreditors and institutions also should be more willing to open up the accrediting process, by making self-studies easily accessible to the public and to other colleges that want to learn best practices, by announcing the teams of peer reviewers that make campus visits for accreditation purposes, and by opening accrediting commission meetings to the public.

"I just think everything we do ought to be open to scrutiny," Ms. Kanter said in an interview after her speech.

Belle S. Wheelan, president of the Southern Association of Colleges and Schools' Commission on Colleges, said she felt "caught between a rock and a hard place" on the issue of transparency.

Her organization is one of the six major regional accreditors that serve as gatekeepers for federal student aid, since colleges must be accredited for their students to be eligible for federal grants and loans. But her association is also a private membership group that has determined that a certain amount of information should remain out of the public eye, she said.

Making the entire process open could have the unintended consequence of giving an institution a bad reputation even as they are working diligently to correct problems, said Ms. Wheelan.

In reading an article published in Social Work by Mattaini and Kirk (1991), I was happy to see the struggle that every social worker should be struggling with written in an article. The article focuses on assessment in social work practice. In this case, the authors focused on the initial assessment of client functioning rather than the assessment of client change. The authors point out that the primary forms of assessment in human services do not couple well with the values of social work. Many assessment techniques are rooted in a deficit model and seek to provide the practitioner and the client with a mental health diagnosis that will serve to drive the intervention forward. This mental health focus leaves little room to entertain the mezzo and macro disruptions that clients face. With the field of social work adopting the medical model, it is a case of working with what’s available based on general consensus.

Unfortunately, this model of preliminary assessment is so outside of the scope of social work that the tendency to overlook the guiding professional ethics and values seems more likely than not. As Mattaini and Kirk (1991) point out “classification systems can unintentionally or intentionally be used to distort phenomena” (p. 261). This concern can be traced back to practitioners having to take into account method of payment, what diagnosable condition will be allotted the most sessions, and other irrelevant concerns regarding social work intervention.

The article did a good job of presenting the view that if practitioners move forward with faulty initial assessment criteria, how then will they assess client change?

Mattaini, M. A. and Kirk, S. A. (1991). Assessing assessment in social work. Social Work, 36(3).

260-266.

Learning Reconsidered

National Association of Student Personnel Administrators & American College Personnel Association. Learning reconsidered a campus-wide focus on the student experience. Washington, DC: Keeling, R.P.

On this report published by ACPA and NASPA a shift in the way that learning should take place is proposed. That shift is to transformative education. According to the report transformative education really builds on the idea of a liberal education, meaning that we need to look at developing the whole student versus teaching only a skill set and required courses. This idea recognized that in classroom and out of classroom learning are interconnected and that both vital to the development of students.

According to the report "Student affair, in this conceptualization, is integral to the learning process because of the opportunities it provides students to learn through action, contemplation, reflection and emotional engagement as well as information acquisition" (Keeling, p. 12). The text stresses the importance of having faculty involved in the process, adapting there teaching styles to incorporate these opportunities to meet the learning needs of students today. To summarize, student affairs and faculty need to work together to provide opportunities for students to take what they learn in the class room and have real world applications. Students need to be able to develop there leadership skills, learn about there strengths and weaknesses, and seek to understand others.

Once an institution make a commitment to change they need to set well defined outcomes. This will help students understand what they are suppose to learn both in and out of the classroom and will help guide the institutional goals and programs. According to the reading the main outcomes are similar for most institutions but the ones that are emphasised will be determined by the mission of the institution. a land grant university will have different priorities than a faith based private school or a for profit college. The key is that every campus need to clearly define the outcomes and ensure that they are evaluated. Page 18 and 19 of this report contains a matrix that has some good examples of specific outcomes including what is learned within each outcome and what body of knowledge they come from.

When it comes down to the assessment of the outcomes the text emphasises the importance of collaboration between student affairs and faculty. It also states that student learning should really become the focus of the assessment tools, recognizing that student satisfaction is important and can valuable it does not tell us how students are learning. The text recognizes that some learning outcomes are hard to measure, particularly those related to personal and social growth. These can often be the most important outcomes to asses and innovative assessment tools need to be created to measure them.

The text points out the importance of closing the loop, "In order to facilitate continuous improvements, the assessment data must be used in a timely fashion" (p. 24). I was a little disappointed that there was not some good suggestions on how to make that happen in the text.

The two main ideas that a got from this reading is that transformation education needs to take place to prepare students, and that the university needs to work in collaboration to make sure that it happens. It was reassuring to read some of the same ideas that we have been talking about in class such as assessing student learning and closing the loop. I know that there is a learning reconsidered 2 and I am going to try and get a hold of that for a future blog

Pre-Assessment for Business & Industry: The Magic 8 Ball

Lisa Walker's article "Assessing Competency During the Hiring Process," takes pre-assessment to another level. In the world of higher education, pre-assessment is traditionally used to measure the level of prior knowledge the student brings to an individual course or program. Upon completion of the course or program, they are given a post-assessment and the scores are compared not only to their own scores but to the scores of their classmates. Business & Industry has their own form of pre-assessment and the stakes are infinitely higher than those in education.

Human Resource professionals are using tools backed by sound research and data analysis to determine (or maybe predict is a better term) whether a person will be a "good fit" to their company. That's right! A test can be used to figure out whether HR or the hiring manager should consider an applicant before an offer of employment is made. These are not to be confused with personality tests, since we are not trying to determine who you are but actually what you have done or would you could possibly do. Talent assessment is one of the terms that business and industry uses for pre-assessment and it can be used to give some insight on a few things.

Tendancies, such as leadership style, values, thinking style, and emotional competencies can be identified even before someone is hired. Usually, the test is given after the candidate has been interviewed and submitted a resume'. The test is sent to an email address and the candidate completes it and sends it back. Most of the tests are scored automatically through a web-based tool and results are obtained quickly. The end product is a report that gives various details on the tendancies of the candidate.

Of course, these tests cannot be used autonomously, since there are many factors that predict success of an individual. If these are used with interviews and resume' materials, hiring managers get a bigger picture of who these people are.

My experience with these predictive assessment tools has been good. When I received my results I asked if they called my mom to get the information that they did.

Higher Education Assessment

Ruben, B. D. (2007). Higher education assessment: Linking accreditation standards and the Malcolm Baldrige criteria. [Article]. New Directions for Higher Education(137), 59-83.

This article provided an interesting overview of higher education assessment. It was an excellent supplement to our class discussion with Dr. Harrold. In particular, it described the work of The Spellings Commission, of which I was unfamiliar. The Spellings Commission produced six recommendations to address education reform within the United States. Of the six recommendations, I was most interested in the third.

“To meet the challenges of the 21st century, higher education must hange from a system primarily based on reputation to one based on performance. We urge the creation of a robust culture of accountability and transparency throughout higher education. Every one of our goals, from improving access and affordability to enhancing quality and innovation, will be more easily achieved if higher education mbraces, and implements serious accountability measures.”

Assessment is the key to this recommendation. However, during our class discussion, it was noted that assessment is often perceived as an endpoint rather than a means to a continual process of improvement.

The author also commented on accreditation and the review process. It was noted that the accreditation process “stimulated evaluation and improvement, while providing a means of continuing accountability to constituents and the public.” Of interest, the Baldrige Framework was discussed as a rigorous approach to assessment and improvement through accrediation. The goals of the Malcolm Baldrige National Quality Award program (MBNQA) are to:

Identify the essential components of organizational excellence
Recognize organization that demonstrate these characteristics
Promote information sharing by exemplary organizations
Encourage the adoption of effective organizational principles and practices

The author suggests that these criteria provide a standard of excellence to which higher education entities can and should aspire.

The Excellence in Higher Education framework (EHE) model was also discussed. This model was designed to adapt to the needs of a broader range of higher education institutions. The EHE uses the Baldrige and regional accreditation organization criteria to offer the best available standards of excellence for higher education. The EHE framework consist of seven themes: leadership, strategic planning, beneficiaries and constituencies, program and services, faculty/staff and workplace, assessment and information use, outcomes and achievements. The seven themes are scored anonymously and areas of strength and improvement are reviewed and discussed. It can be used to evaluate programs, departments, or institutional excellence.

This article gave me great insight into the challenges and opportunities offered by accreditation, the importance of embedding assessment into the culture of an institution, and frameworks that can be used to strengthen and improve higher education.

Diploma Mills

I thought it was very interesting when Dr. Harrold spoke about diploma mills during his talk on Tuesday. I found an article in The Chronicle of Higher Education that addressed some of that issue. The article, Inspector General Warns Accreditor Over Online College, Raising Fears Among For-Profit Institutions (2009) indicated the Higher Learning Commission of the North Central Association was in trouble for accrediting a college that was questionable. I took in interest in this as the HLC of the NCA is our accrediting agency. It appeared the American InterContinental University was being scrutinized for the way they count their credit hours and program length. The IG of the Dept. of Education recommended the "limiting, suspending, or terminating" of the organization. This could be a huge issue, as this all pertains how online credit hours are measured. Since most colleges now have extensive distance-learning programs, it will be interesting to see how it all turns out. The for-profit higher education sectors would be most impacted, as nearly all of them seek their accreditation through HLC. The OIG is cracking down on those accreditating agencies that appear to be lax in their accreditation process. As Dr. Harrold mentioned the for-profit organizations will need to reconsider their standards and how they assess their programs.

Inside Higher Ed

Inside Higher Ed. I have this delivered to my inbox each morning, it allows me to get a quick glimpse of the latest info and news, perhaps even trends that I need to be aware of in regards to all things related to higher ed. It also has some insightful blogs from many different perspectives.

I am always looking for new assessment methods--the College of Business uses a pre-test/post-test for the majority of assessment practices; although we have three faculty currently who are moving well beyond that and using some really interesting methodology in utilizing qualitative data this semester. Here is one resource I found through Inside Higher Ed in regards to a qualitative instrument for assessment. LINK

Internal Communication: Key to Assessment Success?

As I begin this course I am interested learning about my cohort’s experiences regarding assessment. In particular, I have an interest in the observations of those who have experienced assessment from the faculty perspective.

Much is bantered about in the local institution regarding the assessment process come accreditation time. The inherent problem is the “bantering” is not always positive or productive. A theme inherent in my past assessment experience is internal communication issues. Limited information for faculty regarding the assessment process invariably led to questioning and, in extreme cases, outright mistrust of the administration’s intentions. While there were those among us who realized the process was centered on assessing student learning, there were a number questioning every step in the process.

My previous institution was a two-year community college infused with trade and technical programs. Faculty in these programs typically had little or no previous professional teacher training. They were hired because of their content knowledge, not their methodological abilities. Hence, I assume assessment strategies were a foreign concept. This is by no means an indictment of their abilities to teach content, but from personal experience, the learning curve regarding teaching methodology (in effect becoming a better instructor) can be steep. This observation is a result of my serving as a mentor to new faculty in trade and technical programs.

When administration began giving out “nuggets” on upcoming assessment requirements, there were faculty members who immediately began questioning the process. In reflecting on one of these assessments, the following issues arose:

• Perception the administration was being “heavy-handed” in dealing with upcoming assessment processes.
• The Assessment “Sky is Falling” Syndrome.

“Oh no, here we go again.”

“The VP was at another assessment conference, what is it going to be this time?"

• Questioning the administration’s commitment to assessment. Just prior to one assessment stage, the position of institutional researcher was cut due to a reduction in force process. Faculty questioned the commitment of administration to assessment as institutional research was considered a key component of a successful process.
• Concerns the process was a front for faculty evaluation, not true program assessment.

Coming full-circle was the issue. The reason the process was problematic was the lack of communicating the intricacies and benefits of a well-designed assessment process. For assessment to be effective, a formative process must be implemented to ensure perceived program weaknesses are addressed through curricular and institutional changes. Hence the process can come full circle.

Allen sums the above concept with the following:

“Assessment programs are likely to fail if faculty and administrators forget the purpose of their efforts. Assessment studies examine student learning in the entire program; they should not focus on individual students or faculty. The purpose is not to assign blame, but rather to identify program strengths, areas of concern, and solutions that faculty are willing to implement” (p. 157).

Your experiences and comments in this regard are appreciated.

Assessing Intercultural Learning Programs

Hi Everybody,

I found an article on using rubrics to assess intercultural learning programs that I found very helpful. Cultural diversity is emphasized more and more in higher education, yet it is one of those areas that are difficult to assess. Students, and sometimes faculty, are not always clear about what is expected of them. The authors emphasize that rubrics can be used to provide guidelines that clearly articulate performance expectations and proficiency levels. They provide benchmarks for success and objective means to assess what can be considered subjective tasks or characteristics e.g. attitudes, values, critical thinking, etc. Rubrics can help students realize how their effort relates to evaluation.

The goal of cultural education, according to the authors, is for students to develop cultural fluency, which transcends just knowing about a cultural to a state of being. Clearly, this abstract competency is difficult to measure and assess. The goal was to facilitate and assess the developmental process of achieving competency in cultural fluency. Six objectives were developed which represent indicators of essential intercultural competence. Rubrics were then developed for each objective with clear performance indicators at three levels (insufficient, adequate, excellent). Three of the rubrics were incorporated into the article, providing great examples of how assess an abstract competency as objectively as possible. The learning objectives are listed below.
The student:
* Understands the dynamics of cultural differences and can apply that understanding through engagement with another culture.
* Sees global connections
* Has an expanded sense of self as a global citizen; understands self in relationship to the larger society
* Reflects and understands how personal identity is transformed through interaction with others
* Understands and can appreciate the context and perspective of others
* Can apply intercultural sensitivity and awareness to other situations, societies, and cultures

For example, in objective #1, the students are expected to move from seeing issues and complexities of people from other cultures as unrelated to their own life (insufficient), to appreciating how their lives intersect with people from other cultures (adequate) to the point in which students begin to participate in the multicultural environment, examining their own cultural empathy (excellent). Examples were given for the three rubrics presented, applying them to real student cases. This helped me to operationalize what I consider an abstract process.

The authors affirm that rubrics should be shared with students so that they know the criteria that will be used to measure their learning and performance. They have a right to know what the expectations will be. It is important that students and teachers dialogue throughout the assessment process and that the assessment techniques are used as tools for learning. When trying to measure a behavior as abstract as cultural fluency, rubrics provide a common language for faculty and students to measure skill development and competency.

I found the article interesting. To be honest, I do not use rubrics in assessing student performance to the extent that I should. I'm realizing how effective they are in communicating clear expectations and guiding assessment.

McGury, S., Shallenberger, D. & Tolliver, D. E. (2008). It's new, but is it learning? Assessment rubrics for intercultural learning programs. Assessment Update, 20 (4), 6-9.

Carla Gross

Online Survey Science?

Source: Crawford, S. (2006). Hardening the science of web–based surveys in assessment research. Concepts and Connections, 15(1), 14-15.

http://www.leadershipstudy.net/downloads/C_C15_1_MSL_assessment.pdf

It seems as if online surveys have become a way of life. If the past month is any indication, I receive about one request to fill out such a survey each week. The surveys come from a variety of sources: fellow graduate students, our campus dining service, the local business who has my name on their e-mail list, etc. By utilizing easily accessible and inexpensive online tools such as Survey Monkey, it seems like anyone can create and administer a survey. As a frequent survey respondent, I know some are more pleasant to complete than others. As a master's student considering an online survey as potential thesis methodology, I wonder if there is more to creating a well written survey than my simple gut reaction and anecdotal experience?

In the brief article, "Hardening the science of web-based surveys in assessment research" from Concepts and Connections, a newsletter published by the National Clearinghouse for Leadership programs (http://www.nclp.umd.edu/), the author attempts to explore such a question. He mentions that online surveys are a tool where every detail seems to count (from question order to the colors, fonts, and design that is used) and then mentions that entire books have now been written on the science of such online methodology. Despite these resources, the everyday general researcher often will "take short cuts in the implementation of this science" leading to a "significant amount of survey research" which "includes inferior techniques, or methodological flaws that put the data at risk." Of course, he is referring to those who are looking for more credible research data. I'm guessing that my local consignment shop isn't as concerned about the validity and reliability of their survey as much as the graduate student or official government researcher. However, could there be a few simple tips that all of us could follow, no matter what our scientific concern?

Perhaps, but the author of this article did not expand on such tips. He did include ten considerations when planning a web-based survey of student populations. This list is interesting and may be helpful if you are planning such a survey (feel free to click on the link at the beginning of this post for the full PDF article). Most of the considerations seem to lend themselves to publicizing and gaining student buy-in than the scientific side of the web-based survey.

So, now I know that more information about creating effective and valid online surveys exists, but I'll have to keep searching to find more answers.

In the news...

From the Chronicle

Goal and objectives

Kory and I tried our hand at developing an goal and objective.

Students will be able to generalize the relationship between the STEM courses:
Course Objective:

• Students will identify, generalize, and give examples of relationships in STEM courses
  

Activity(s):

• Students will work together in groups to physically separate a mixture of pure substances
• Students will investigate possible methods to separate substances
• Students will identify substances removed from mixture

Assessment:

• Students will present their findings in an essay format (10 minute free writing)
  

Student Knowledge

Although I have gone through formal education to be a teacher, the one assessment class that I earned my "A" in was a course in alternative assessment. It becomes quite ironic that the grade I earned in this course has nothing to do with what I learned. This brief time that I have been involved with this program and course has shed light on much of the reasons why I continually strive for perfection in my work. Understanding that much of what I went through in my undergraduate days allows me the time to start new with education in my doctoral program and start learning how to learn.

I teach mathematics at the community college level and have done so for about 14 years. It becomes quite easy to teach students that "get it" from the get go, but quite difficult when multiple angles of approach seem to fail the student and thus, the teacher fails? I have had students that will ask me what their grade is and my response is usually a question right back at them...I don't know, how much have you learned? Do you want a grade or knowledge? It's difficult for any student, not excluding myself, to want good grades. However, at the doctoral level of education I am finding that what I am going through now is what I want to learn...to retain for future reference.

So, I am grabbing a hold of what I can with my antiquated mind and am sharing it with my students. I find that when I allow them in on the assessment strategy I am offering, they become engaged. Angelo & Cross (1993) share that letting students know about the results you learned from a classroom assessment technique allows the student to become part of the assessment. I am finding that when my students know I am working my tail off to try and figure out the best way to help them learn to learn mathematics, they work harder at learning it.

I wanted to "blog",...(I'm a blogger, wait til I share this with my students!), this because I have exprienced that other faculty members don't always look at assessment as something that is going to be a good thing. "It might look bad on my evaluation if the results aren't up to par." Well then guess what? Time to find another occupation. If you can't teach, find a truck driving job somewhere or perhaps McDonalds is looking for a Big Mac tester. I don't mean to be heartless, but if we're going to be something, then we'd better try and be the best and if one thing isn't working for the student, then it is high time you reach out and find the right thing that is going to work. Does every minor league baseball player make the majors? Time to go back to your day time job buddy!

So anyway, perhaps education is the best thing to get those "I don't need anther thing to do" teachers on the right track or they better start worrying about their evaluations, because strike three buddy and you're outta there!

Program Assessment

My goal for the beginning of this week was to focus on the program perspective of assessment. My initial thoughts on program assessment was that is is used strictly for accreditation purposes, what I have learned is that is can be so much more for institutions if implemented based on student learning and not just for accreditation purpose. Ironically I find it very sad that I have been in higher education for a few years (as an adjunct-which may be a reason for my lack of knowledge) and other than NCATE (which) I assumed was another 'hoop' to jump through to obtain accreditation and is only based on what courses were taught in a program. I decided that I would use this first assignment for the blog to learn more about program assessment. What I found was that the possible reason for my lack of knowledge about program assessment is that in many colleges/universities the resistance from faculty on assessing programs for student learning outcomes can be difficult because of the possible issues arising in evaluation of the faculty members instead of the program and that faculty are so busy teaching to the course learning outcomes that often looking at the 'big picture' outcomes can be placed on the back burner.
I found this blog entry from the Chronicle on Assessment in Higher Education by Stan Katz: http://chronicle.com/blogPost/Assessment-in-Higher-Ed/7489/ In this blog entry article, the Katz spoke of assessment and the use of assessment in student learning based on a study conducted by Ikenberry which found was that most assessments at the program level are being used for accreditation requirements, but not necessarily at the institutional level partly because of faculty resistance. From my understanding, this type of assessment is of a summative nature. Shouldn't institutions be looking more at formative assessments to gauge student learning? The author suggested that part of the reason for this is that the learning outcomes tend to be very generalized. From our class discussion, I would label these as 'goals' if they are generalized to the point of being difficult to measure. Another issues in an article titled: Many Colleges Assess Learning but may not use data to improve survey finds: http://chronicle.com/article/Many-Colleges-Assess-Learni/47892/ discussed that much of the data collected through assessment measures is not being used to improve teaching and learning, instead it is being used strictly for accreditation. These two articles led me to a website titled: Collegiate Learning Assessment (CLA) http://www.collegiatelearningassessment.org/ which is a higher education assessment site that colleges can use to assess the learning of students through working with faculty, program chairs, and administration. Much of the assessment touted by CLA is of an authentic nature and in helping colleges with programmatic change that is then compared with other universities, however they do state that they do not rank universities based on scores, instead they highlight differences that can lead to improvements. I find this somewhat curious. It seems to be standardized testing gone commercial. The other factor with CLA is that the testing is not discipline-specific, rather it is on general education skills and does not test all students, rather just a random sampling of students in a pre-post test ability. I'm not a fan of this type of assessment because I think it lacks in relevance to the specific goals for each university. There is an initiative by the Higher Learning Commission (HLC) http://www.ncahlc.org/download/PathwaysOverview1009092.pdf for a new model titled 'Open Pathway' for accreditation in which institutions are put together with 'peer institutions' to work together in creating a "systemic comprehensive assessment of student learning"(p.3). What I like about this is the pairing of two-year colleges with four year colleges to create a plan for "internationalizing the entire curriculum" (p.3). The goal of this new model is to allow institutions to, "take risks and aim high" in the commitment of student learning.
Overall, what I found is that program assessment is a highly bureaucratic and in-depth process of assessment of student learning outcomes at the program and institutional level.
Sheri

Closing the loop

In my experience, the concept of 'closing the loop' has been one of the most problematic issues when doing any sort of course or program assessment/ evaluation. While I agree with Dr. Enger that ensuring the accurate measurement of the achievement of objectives is of primary importance to the effectiveness of a program evaluation plan, the fact that these measures are accurate does not necessarily mean that the ‘loop’ has been ‘closed.’

If, for example, an (accurate and well designed) assessment indicates that students are falling short in the achievement of a particular objective, the fact that we have accurately diagnosed that shortfall does not in itself ‘close the loop.’ The ‘loop’ (as it were) is not ‘closed’ until we have (in Allen’s words) “used the results to improve the program.”

Although undoubtedly useful, Allen’s six step process represents for me a perfect example of a ‘black box’ model, where a key (and arguably, the most important) element is left relatively unexplored while related steps are meticulously defined. Allen frankly admits that “step 5 is probably the hardest”, but fails to even remotely describe exactly how “good assessment leads to collective reflection and action.”

Whenever I am developing an assessment/evaluation instrument, I find it helpful to think in terms of “actionable items”, that is, where the measure itself indicates possible steps toward remediation, if necessary. Similarly, the inverse of the actionable items concept is also true: assessments that incorporate measurements that do not (or cannot) lead directly to an action item are time wasted, and opportunity lost.

One example of this that comes to mind is the common practice of measuring online students’ opinion of the effective of the Learning Management System. Since, in most cases, there is absolutely no possibility that the results of the query will drive a change (most universities are bound by contract and have no intention of changing their LMS), the effort is futile, and more importantly, the opportunity to ask a question that actually could lead to effective change is wasted.

Closing the loop is more than a black box, and goes beyond simple validity. Absent this crucial process, however, we are left only with the knowledge of exactly how skinny our pig is.

Just a thought…

Goals and Objectives

Pete and I took a shot at developing a program objective:

Promotional materials for the VCSU Education Department state a dedication to: preparing knowledge based decision
makers and highly qualified teachers who possess the ability to:
• use technology
• communicate
• solve problems
• collaborate
• demonstrate competency as citizens in a global society

From the above statement, we inferred the following program goal:
Creating knowledge-based technology decision makers.

Which led, in turn, to this program objective:
Graduates demonstrate the ability to make informed and reflective decisions regarding the effective integration of appropriate educational technologies in instructional environments.

Its certainly a bit wordy, but I think it covers most of the Smart bases. Waddya think?

Comments or feedback would be welcome.