Practice Exchange

Ilene Gast, Associate Editor

This column highlights innovative assessment programs of interest to assessment professionals. If you are conducting a project that would interest the ACN readers, or if you know someone who is, please let me hear from you. I can be reached by phone at (202) 305-0590, by fax at (202) 305-3664, or e-mail at IFGAST@AOL.COM.

Monitoring Organizational Performance

by William G. Saylor

Contemporary management philosophies over the last several decades have progressively moved toward statistical process control methods. These methods, also referred to as continuous process improvement (CPI) or continuous quality improvement, are frequently considered to be one of the governing principles of the management strategies referred to as total quality management (TQM). CPI involves the use of statistical tools to convert data into meaningful information. The methods are based on the presumption that measuring (observing and quantifying) phenomena allows for systematic analyses that can lead to insights and a better understanding of a process. A better understanding provides opportunities for creating more control over a process, resulting in a reduction in unwanted process variance and consequently an improvement in the process and the product it produces. Decades of experience with SPC and TQM have demonstrated that sufficient knowledge of a process and the use of statistical models can produce an effective method for monitoring organizational performance.

Although CPI was initially more prevalent in engineering and industrial environments it is now common to find these methods employed in many types of organizations, including service oriented organizations such as the Federal Government. The Clinton administration has implemented two related initiatives, the Government Performance and Results Act (GPRA) and the National Performance Review (NPR), that have mandated Federal agencies to establish measurable performance standards and procedures for implementing continuous process improvement. Since 1988, the Bureau of Prisons' Office of Research and Evaluation has provided a tool to assist management with monitoring the organization's performance. The tool, the Key Indicators/Strategic Support System (KI/SSS, Saylor, 1988, 1994), is a PC based, menu driven executive information system designed to provide management with on-demand access to a synthesis of the operational data they require to draw conclusions and make sound decisions. The KI/SSS is a voluminous reservoir of easily accessible measures, providing managers with a continuous flow of characterizations and measures of organizational behavior and performance about the Bureau of Prisons.

The KI/SSS design is based on the concept of organizational climates. It contains two varieties of climate measures, objective and subjective (Saylor, 1983). The objective measures are a by-product of the organization's operational data needs. These measures are meaningful summaries of individual inmate, staff, and financial units that are culled from the Bureau's operational mainframe MIS with a monthly periodicity. By meaningful summaries I mean summaries that relate to the mission of the Bureau and to the goals and objectives of management. The subjective measures are obtained from administrations of the Prison Social Climate Survey (Saylor, 1983); a survey questionnaire administered to a stratified proportional probability sample of Bureau staff on an annual basis and to inmates on an ad hoc basis.

Strategic planning and quality control are two processes that empower managers by providing vehicle for iteratively monitoring and modifying performance in order to achieve a desired level of performance. In my view strategic planning and quality control are interrelated processes. Strategic planning is an effort to produce fundamental decisions and actions that shape and guide what an organization is, and what it does. This is accomplished by setting or changing organizational objectives, identifying and obtaining the resources required to meet these objectives, and determining the strategies, programs, and policies needed to accomplish the objectives. Quality control can be defined as a set of activities or devices that help to attain the excellence of something. It is a process of establishing standards of comparison against which to check the results of a process. The control of quality is accomplished by the establishment of measurable standards for comparison against measures of quality characteristics. Quality characteristics are any properties that define or describe the nature of a product.

Strategic planning and quality control are complementary activities. While strategic planning establishes the requisite policies and resources, quality control monitors the progress made in pursuit of these goals and provides the information needed to make any necessary adjustments. In this regard, strategic planning is a prospective process, one that requires us to look forward and compare where we are now relative to where we want to be at some future point. Conversely, quality control is a retrospective process, one that requires us to look back and compare where we have been relative to where we are now, to determine if we are satisfied with the current product. As such, strategic planning and quality control have a cyclical relationship whereby strategic planning sets the course and quality control tells us how well we stuck to our course and whether we arrived at our intended destination. Consequently, the utility of any strategic planning effort is contingent upon the quality (accuracy and comprehensiveness), and quantity (volume) of information available to the process. The greater the reliance on quality control processes in formulating a strategic plan, the more efficient and effective our planning effort becomes. KI/SSS promotes the use of strategic planning and quality control cycles by providing managers with quick access to organizational measures over spans of time.

In order to plan for the future, measures of institutional operations must be readily accessible to prison system managers. The development of a strategic system is the essence of pro-active data management. Planning for an organization's strategic information needs can produce measurable resource savings. Conversely, the costs of management decisions made without the benefit of adequate information may not be obvious, and are often difficult to measure.

The Bureau's strategic support system can contribute to management's ability to ensure quality planning because it will place a multitude of relevant information at their disposal. This strategic system provide managers with a tool with which to monitor their organization's performance. The ability to monitor the organization's operations can provide the Bureau's managers with a better understanding of whether there is compliance with existing strategic goals (and the plans and policies designed to achieve them), whether policies are having their intended impact, whether policies are in need of modification, or whether new policies or plans are needed. The strategic system allows the Bureau's managers to make decisions based on information that is selected for its relevance rather than by the constraints of what is available and can be accumulated and assembled within the time allotted before some action is required.

For more information, contact William G. Saylor, Deputy Director, Office of Research and Evaluation, Federal Bureau of Prisons at (202) 307-3871, ext. 105.

References

Saylor, W. G. 1983. "Surveying Prison Environments." Unpublished manuscript, Federal Bureau of Prisons, Office of Research and Evaluation, Washington D.C.

Saylor, W. G. 1988. "Developing a Strategic Support System: Putting Social Science Research into Practice to Improve Prison Management." Paper presented at the American Society of Criminology, Chicago, 1988.

Saylor, W. G. 1994. "The Design, Development and Maintenance of Strategic Support Systems for Correctional Agencies." Presented at the 1994 International Symposium on Criminal Justice Information Systems and Technology, Washington D.C., 1994

Understanding and Evaluating Integrity in Law Enforcement

by Kerrie Q. Baker

The Federal Bureau of Investigation (FBI) is focusing a great deal of attention and resources to the importance of integrity and its inclusion in the selection, hiring, and training process. To demonstrate the value that the FBI places on integrity, its motto is "Fidelity, Bravery, and Integrity." Accordingly, individuals applying for the Special Agent position are carefully assessed for integrity at several stages of the FBI's selection and hiring process. At the first stage, an applicant's integrity is measured via two paper-and-pencil tests, a Biodata Inventory and a Situational Judgment Test. At the second stage, questions on a structured interview also measure an applicant's level of integrity. Once receiving a conditional offer of employment, an applicant's integrity is carefully examined during a comprehensive polygraph examination and extensive background investigation. To further demonstrate the importance that has been placed on honesty and integrity in law enforcement occupations, new Agents receive formal ethics training and testing at the Academy.

While the fact that integrity is validated time and time again as highly job relevant, critical, and necessary for law enforcement officials, it remains one of the more difficult characteristics to define, measure, and score because of its multi-dimensionality. As described above, the FBI employs several traditional methods to evaluate integrity among its applicants. Interestingly, we are not finding a clear cut linear relationship between integrity and performance. Given the importance of integrity, we hypothesize that a curvilinear (i.e., inverted-u) relationship exists between integrity and performance. A moderator in this relationship may be one's judgment in determining whether a person's high ethical behavior is appropriate for the situation.

Since August 1994, the FBI has tested over 30,000 applicants in Phase I and 9,000 in Phase II. Results from these tests indicate that, consistent with the current literature on integrity testing (e.g., Ones, Viswesvaran, & Schmidt, 1993), integrity as measured by the paper-and-pencil tests did not correlate highly with the cognitive ability score. Nor did the integrity score from the interview correlate highly with the cognitive ability score. Interestingly however, the integrity items from the Biodata and the Situational Judgment Tests were not predictive of the interview scores of integrity. Finally, data suggest that sensitivity to a situation and excellent judgment are critical factors when determining a course of action in an ethical dilemma. Further research will be conducted by: re-examining the definition of integrity and incorporating the situation, judgment, and consequences of actions; collecting personality data, suitability data, and training performance data to examine the nonlinear relationship; and improving the measurement of integrity in the selection system.

The FBI does not rely solely on the selection instruments to gauge an applicant's integrity. After the selection instruments are passed, a full field background investigation is initiated to examine an applicant's past behavior, to include demonstrated integrity. Such a background investigative process further screens out applicants whose past, or ability to deal honestly with the FBI during the application process, is indicative of behavior in the future that does not coincide with the Director's lie, cheat, or steal policy. The background investigation typically takes 3 to 6 months to complete per applicant. Some of the items that are verified for each applicant include: birth/citizenship, residences for the last five years, all college attendance and degrees, all employments and military service, court records, credit history, and arrest checks. Interviews are also conducted with neighbors, professors, supervisors and coworkers, social acquaintances, and references to examine such things as an applicant's character, reputation, loyalty, financial responsibility, attitude toward others, and alcohol or drug abuse.

The FBI also uses the polygraph to closely examine applicants. It is designed to focus on areas that cannot be addressed through conventional investigative methods, including national security issues, sale and use of illegal drugs, and the accuracy and completeness of the application for employment. Experience demonstrates that a pre-employment polygraph program serves as a deterrent for some would-be applicants and can identify individuals withholding important information. The polygraph can only address past behavior which may be an indication of suitability.

Not only are applicants thoroughly screened on their honesty and integrity, they receive training on ethics and integrity once hired as New Agents. To further demonstrate the importance that is placed on integrity within the FBI, the FBI recently has established an Office of Law Enforcement Ethics (OLEE) at its Academy. The OLEE is responsible for developing, implementing, coordinating, and managing the ethics initiatives for the FBI. These initiatives include, but are not limited to, providing instruction to Special Agent trainees, field Agents, and other domestic and foreign police agencies.

The OLEE training curriculum consists of seven two-hour classes, covering a wide variety of ethics topics, with a comprehensive examination at the conclusion. The curriculum incorporates a balance of instruction of the basic ethical principles that are presupposed by the Constitution and that serve as the foundation of our moral intuitions. Such instruction is coupled with discussions concerning how these principles impact law enforcement in general and the duties of an FBI Agent in particular. Situational exercises and philosophical arguments force individuals to examine possible justification for and outcomes of ethical and unethical behavior. The FBI's proactive and preventative approach to ethics training hopes to play a role in identifying and reducing incidents of ethical misconduct in law enforcement. In furtherance of that goal, and among several ongoing research initiatives, a joint project between OLEE and the FBI's Behavioral Science Unit is underway, attempting to identify and assess causes for ethical misconduct.

For more information, contact Dr. Elizabeth Kolmstetter at (202) 324-8254 or Dr. Kerrie Q. Baker at (202) 324-1512.