Practice Exchange

Ilene Gast, Associate Editor

This month we have two special articles. In the first article, Paul Kaiser describes an innovative assessment program implemented by the New York State Department of Civil Service. This program received IPMAAC's Innovations in Assessment Award for 2000. The second article by Laura Huffman and Carolyn Linér of San Marcos, Texas, describes the program for which they were awarded the Certificate of Merit in IPMAAC's Innovations in Assessment Award for 2000.

If you are conducting a project that would interest the ACN readers, or if you know someone who is, please let me hear from you. I can be reached by phone at (202) 305-0590, by fax at (202) 305-3664, or electronic mail at ilene.f.gast@justice.usdoj.gov.

The New York State Promotion Test Battery Program

by Paul D. Kaiser, Chief Examiner
New York State Department of Civil Service

The Promotion Test Battery (PTB) Program

The New York State Department of Civil Service has initiated a fundamental change in its selection approach for promotion to supervisory, managerial and administrative positions throughout state government. Instead of title-by-title testing for promotion, all eligible employees have the opportunity to participate in an annual battery of tests designed to evaluate job dimensions common to most supervisory and managerial positions. This change allows a more comprehensive assessment of the core managerial competencies associated with these positions; enables the Department to more effectively and efficiently use its limited resources; and expedites the establishment of eligible lists.

Why did the Department develop a new promotion system? Like many governmental testing organizations around the nation, New York State had a number of long-standing merit system testing problems that had to be addressed. First, there were a large number of individuals serving provisionally in management-level titles, many in one-position classes. We lacked sufficient resources to hold individual promotions examinations to meet the needs of the merit system. Second, it was taking too long to establish eligible lists. Third, our customers, agency program managers indicated a need for a more comprehensive test of managerial skills. And fourth, candidates complained that every time they took a promotion examination they were being re-tested in the same skill areas.

Development of the PTB

An extensive job analysis study identified three the dimensions that the PTB measures: Supervision and Management of Staff, Written Communication, and Problem Analysis. Because over 30,000 employees are eligible to take the PTB, it was essential that the instrumentation be not only psychometrically sound but also efficient and economical. Written simulations were selected as the best approach to test for the Supervision and Management of Staff components of the PTB. This assessment technique presents realistic and complex managerial situations and, in a way that replicates the actual work situation, provides feedback to candidates as they proceed through the exercise. Due to their efficiency and reliability, multiple-choice questions were used to test for Written Communication and Problem Analysis skills. The PTB uses a large block of these questions to test in these areas; hence these dimensions are covered in greater depth than was possible under title-by-title testing.

Because the PTB is used for a wide range of titles (800+) over a large number of State agencies (50+), all of the test material is developed through the extensive use of job experts. Two teams of managers and executives assist in the development process. The first team helps draft the test exercises and questions; the second team then 'cross-reviews' the work of the first. The test material then undergoes sensitivity reviews. Panels of African-American and Hispanic reviewers examine the material to identify any elements that may be perceived as disadvantaging or in any way offensive to candidates from these groups. For example, the Hispanic reviewers have occasionally identified wording that may be confusing to non-native English speakers. Such material is revised or removed from the test.

After each administration of the PTB, the score data is subjected to rigorous psychometric and statistical analysis and scrutiny. In addition to standard item analyses, other, more complicated analyses are done as well, including analysis of patterns of responses, factor analysis, and analysis of part-test scores. Also, the job expert panels are reconvened to review the test results and to look particularly at those items where the response patterns were not what was expected. The Mantel-Haenszel differential item functioning (DIF) procedure is applied to the test item responses of the candidates. Sensitivity review panels are reconvened to study items flagged as having significant DIF. The reviewers look for anything in the items that might account for the DIF. Internal consistency reliability estimates average 0.85+ for the total PTB score.

How the PTB Works

Implementation of the PTB Program resulted in a number of innovations in the way the Department administers promotional examinations. A number of these could be readily incorporated into the processes of other assessment systems. The PTB is given annually and candidates can take the PTB each time it is offered. Battery scores are active (banked) for up to 10 years; the candidate's highest score is used for eligible list purposes. Candidates are provided with a comprehensive study guide, which explains how the program works, how PTB scores will be used, and how candidates should approach the test to achieve their best score.

Another unique facet of the program is that scores from the PTB, by themselves, do not result in an eligible list. When a list is needed for a particular title, an examination announcement is issued stating the minimum qualifications and any additional job-specific tests (e.g., orals, training and experience assessments, technical written examinations) the candidates must pass. It is this feature which allows for lists to be established quickly. If the job analysis shows that testing candidates on specific technical competencies is unnecessary, then the candidates' PTB scores are used alone to produce the list--virtually on demand. If specific technical tests are necessary, the second phase of the examination can focus solely on the job's technical content (since the battery already assessed the position's core managerial competencies). This reduces test redundancy and allows a much more rapid test development and list establishment process.

The Bottom Line

Because the PTB program is so highly publicized and impacts so many candidates, extensive program evaluation analysis is conducted annually on the results. From the very beginning, it was clear that enormous improvements over the previous promotion test process had been realized. For comparative purposes, the Division produced 207 promotion examinations between January 1994 and June 1996. Between January 1997 and June 1999, 423 promotion eligible lists were established. In January 1994 provisional appointees comprised almost 5% of New York State's competitive workforce. In January 1999, the provisional rate was less than one percent, the lowest level in the Department's history. The PTB program offers a very comprehensive test of required competencies which agency managers affirm is providing better supervisors and managers. Annual testing and score banking provide candidates with more opportunity to demonstrate and improve their test performance. Candidate acceptance of New York State's approach to selection for these important positions is at an all time high. All these improvements translate into more efficient, effective, and economic use of testing resources.

If you would like more information about New York State's Promotion Test Battery Program, contact Paul D. Kaiser at (518) 457-7007 or PDK@cs.state.ny.us.

Improving Team Performance through Evaluations

by Laura Huffman and Carolyn Linér
San Marcos, Texas

In a complex world of change, getting results through high performing teams can help ensure that communities grow and prosper far into the future.

San Marcos is facing phenomenal growth and change. As part of the Austin-San Antonio corridor, we are one of the fastest-growing metropolitan areas in the nation. Over the past five years, we have experienced a 121% increase in building permits. We have experienced a 19% increase in our population during that same time. Ten years ago the organization employed 240 people. Today we have over 500 employees. In the midst of this change, citizens are demanding higher levels of service and more accountability for what is happening with their local governments.

Several years ago we were managing this growth and change through a traditional structure. The City is organized in 18 departments ranging from Parks and Recreation to the Electric Utility. Each director was primarily responsible for managing his or her individual department. While this worked well for departmental issues, the difficulty was in effectively managing cross cutting issues. In addition, we faced the challenges associated with a tenured management team. We knew each others' strengths, weaknesses and styles. However, tenured teams sometimes forget to account for individual change and growth. We have worked together for a long time and needed to find ways effectively manage the growing number of cross cutting issues. In an effort to confront these issues and resolve them, the City Manager posed a single questions at the 1997 directors' retreat. How can we better manage San Marcos?

We concluded that our focus needed to be on creating the highest performing management team possible. Through improved teamwork we knew we could more effectively respond to the complex issues facing our community. The organization and community issues no longer confined themselves to a single department or a single division. For example, how do you manage the physical growth of a community in a way that respects the environment and ensures a high quality of life for all? This kind of problem solving requires holistic thinking and managing.

Background

We turned to an often ignored tool to stimulate this change: the performance evaluation system. It didn't take long to conclude that the existing system was a good way to give merit increases, but did very little to enhance the team's performance. The existing process included:

Formal appraisal instruments
Mandatory supervisor / employee evaluation meetings twice annually
Merit increases in 3%, 4%, or 5% increments (tied to evaluation instrument)

The system was far less successful at emphasizing the behaviors associated with high performing teams. In fact, most directors agreed that they found the system to have minimal value in terms of enhancing professional performance or team values.

Other weaknesses of the existing system included:
Too much executive time to administer
No mechanism to give critical feedback in a positive manner
Stale and predictable evaluation instrument
Too many evaluations that happened too often
No requirement for active director involvement

Building Teams through Appraisals

The objective was to design a process and an instrument that created a high performing team individual by individual. To be successful the new system must meet the following criteria:

Promotes teamwork
Provides self-explanatory feedback
Helps people develop professionally
Ensures fairness
Creates a positive way to discuss sub-par performance
Results in support from senior management and participants
Produces meaningful "take away" after evaluation meeting

With these criteria in place, the deputy city manager and human resources director began researching evaluation systems. We were particularly interested in 360-degree evaluations. In contrast with traditional employee evaluation systems, the traditional 360-degree evaluation process involves multiple levels of evaluation for each employee. Typically, the process includes a self-evaluation as well as evaluations from supervisors and subordinates, going full circle or 360-degrees. The goal of our research was to identify key elements of success and failure.

To promote teamwork and buy-in for the new system, we asked six directors to participate on the project team. The group was charged with creating the new system. The critical objectives that needed to be addressed included:

Create a feedback instrument that emphasizes teamwork
Design a process that ensures full information and buy-in
Determine who participates
Create a safe environment where directors can be honest with one another
Avoid a popularity contest

The team developed a basic instrument and presented it to the full directors' group for evaluation and feedback. In addition, this began creating buy-in with the full team. The key to the instrument was that it was focused on the behaviors we believed to be critical to the outcomes we had defined at the beginning of the project.

The other objectives were also addressed. We decided to start with a modified 360-degree evaluation process. We did not include the directors' subordinates, but agreed to discuss that possibility at a later date. We agreed to begin the process by having each director complete a self-evaluation and evaluate all other directors, the city manager and the deputy city manager (the managers). In turn, the managers were to self evaluate, evaluate all directors and each other. Directors' scores were aggregated as Staff Feedback and the managers' scores were aggregated as Managers' Feedback. This helped preserve anonymity which we knew to be critical to the process

The team also decided to hire a consultant to help fine-tune the instrument and provide training to the group. This went a long way in sending the message that this was not a "gotcha" exercise but was meant to serve as a team development tool. Rollie Waters of Waters Consulting was instrumental in devising the existing evaluation system some ten years ago and continued to be involved in updating the evaluation system and assisting the organization with compensation issues.

At the recommendation of our consultant, we adjusted the instrument to measure gaps between actual behavior and expected behavior. For each item on the instrument, the evaluator assessed how often the person did exhibit the behavior and how often the person should exhibit the behavior. For example, each assessor had to indicate how often the person did coordinate priorities and how often they should do so. The difference between these two scores is called a "gap" and it becomes the focus of the evaluation. If you "Almost Never" coordinate priorities but the expectation is that you should "Almost Never" coordinate priorities there would be no gap--your performance is right on target. However, if you "Almost Never" coordinate priorities but the expectation is that you "Always" should, then you have a significant gap. This is an area that needs work.

The gap analysis allowed us to customize each evaluation item to the particular needs of the person being evaluated. In a City with 15 departments ranging from Parks to Public Works, the leadership needs vary. This instrument allowed us to account for that variation.

A critical element in our success was introducing the new system in a way that ensured buy-in while providing honest and, sometimes, tough feedback. In September 1998, the team completed the first round of evaluations. The consultant compiled the scores and prepared feedback binders that included a team profile showing our group strengths and weaknesses.

That year the directors' retreat centered on the new evaluation system. Each participant received his/her feedback binder and we discussed, in detail, how to interpret the data. This debriefing took an entire morning and was an intense session, both because the analysis was complicated and because people were seeing peer feedback for the first time.

The debriefing was followed by a low-ropes course, facilitated by Wade Bibbee, Director of The Challenge!, a consulting firm. This part of the program involved games that emphasized the value and importance of teamwork. But perhaps more meaningfully, the games provided a much-needed outlet for the group to reconnect and a time for some healing of wounded egos. The simple but salient messages of teamwork were very well received.

Following Through with the Feedback

Individual debriefing meetings allowed each director to create a Performance Improvement Plan that outlined a specific action plan for the next six months. The next month the managers met with each director to discuss the results individually. Directors prepared a Performance Improvement Plan based on the information contained in their evaluation and brought the plan to the session. Each director selected several areas for improvement and provided a written explanation of how he/she would accomplish each goal. Individual plans became the basis for discussion at the evaluation meeting and the foundation for a merit increase of 0-4 percent.

In March 1999, we replicated the evaluation process so that everyone received interim performance information. We also asked each director to provide a mid-year report on how well the plan was being implemented. The managers provided an additional 1-2 percent bonus for those that had done an outstanding job of implementing their plans.

We evaluated the combined team score and established small groups to develop strategies for eliminating our team weaknesses. From the group scores, we identified the four most significant areas needing improvement. We established teams for each area of improvement and directors were asked to work on one team. These improvement teams met at least monthly and found ways to help the entire group improve in the following areas:

Handling differences of opinion effectively and resolving differences or conflicts fairly
Coordinating priorities and building effective relationships with other departments
Demonstrating a commitment to the team mission and goals
Developing 2^nd tier managers

Results through Teams

Over the past 18 months we have seen phenomenal individual and team improvement. Everyone had to adjust to seeing such honest (and anonymous) feedback. Your initial instinct is to want to identify the person that "gapped" you and retaliate. We knew this would happen because our research had predicted it. However, because we had placed such a strong emphasis on teamwork and combined the tough feedback with practical follow-up, we found the process was effective.

We have developed a much higher and more meaningful level of teamwork among the directors. Staff meetings are becoming opportunities to share and exchange information on what is happening in the organization as well as an opportunity for problem solving. People are making a concerted effort to learn about each other's operations. Even if there doesn't seem to be a need to coordinate--we are finding that in virtually all cases there are tremendous benefits from improved teamwork.

We have discovered that we are better able to respond to the City Council's priorities and vision because we are thinking more holistically about the programs and services we provide. We have concluded that through high quality teamwork we are better managing San Marcos.

Co-Authors Laura Huffman, Deputy City Manager, and Carolyn Linér, Director of Human Resources, are part of the management team for the City of San Marcos, Texas. For more information, contact the authors. Carolyn Liner can be reached by phone at (512) 393-8072 or e-mail liner_carolyn@ci.san-marcos.tx.us. Laura Huffman can be reached by phone at (512) 393-8014 or e-mail huffman_laura@ci.san-marcos.tx.us.