420 FW 2, Evaluation Procedures

A. Organizational Evaluations are evaluations of the effectiveness of an organization in the Washington or Regional Office or field station in administering the program(s) assigned to it, including such management processes as integration of internal and external program and policy issues; organizational representation and liaison; direction and guidance of programs, projects, or policy development; acquisition and administration of financial and material resources; utilization of human resources; and review of implementation and results. See 421 FW 1 for evaluation criteria.

B. Program Evaluations are evaluations of program operations, usually at more than one organizational level, to measure actual outcomes and results against program goals and objectives, including Service priority objectives; assess application of Service policy; and examine program execution against established standards. See 422 FW 1 for evaluation criteria.

C. Process Evaluations.

(1) Any evaluation may include an examination of the processes of an office or program. The core elements for this type of evaluation are:

(a) Problem identification, including assessment of program delivery systems;

(b) Solution development;

(c) Solution implementation; and

(d) Assessment of feedback.

(2) An emphasis item is the Service's ecosystem approach to fish and wildlife conservation, a process that transcends traditional organizational lines and programs. See 423 FW 1 for evaluation criteria.

D. Special Evaluations are evaluations of organizational performance, programs, or processes as determined by the Director and required with little or no advance notice.

2.2 Evaluation Procedures. Normally, all evaluations include the following steps.

A. Planning and Design. A critical part of any evaluation, adequate planning and design will maximize the use of resources needed for the evaluation. Exhibit 1 contains questions that the evaluation team may consider in planning and designing the evaluation.

(1) Team Composition. An evaluation team should:

(a) Generally consist of two to five persons but no more than seven, including a team leader;

(b) Be interdisciplinary so that different perspectives can be obtained; for example, a member trained in evaluation techniques, a member with a working knowledge of the organization(s) or program(s) being studied, and a member (Service or non-Service) who is a customer of the organization being studied.

(c) Have at least one manager, supervisor, or staff specialist from the organization(s) or program(s) being studied.

(2) Evaluation Workplan. The evaluation team leader will coordinate the proposed evaluation schedule with the manager of the organization or program to be studied, including advance receipt of documents, persons to be interviewed, and sites to be visited. The team will develop an evaluation workplan, for concurrence by the convening authority, that includes the areas and sites to be covered by the evaluation and schedule of activities. The workplan should be developed with full consideration of the ongoing workload of the organization or program being evaluated.

(3) Data Collection.

(a) Data Collection Instrument. A data collection instrument such as focus groups or a survey, questionnaire, or interview checklist, developed by the evaluation team, is used to obtain information about evaluation subjects and focus efforts of the team for further analysis. It is recommended that the instrument be tested, more than once if necessary, with a representative sample of respondents, prior to issuance to the organization or employees of the program being studied, to ensure that the questions are understood, provide data needed for analysis, and can be answered within the response time.

(b) Data Collection from Non-Federal Organizations. The Paperwork Reduction Act generally provides that no agency may collect information from ten or more persons annually unless such collection is authorized by the Office of Management and Budget. See 281 FW 4, Information Collection Program and Budget, for requirements concerning collections of information, including surveys and telephone interviews, from non-Federal entities.

B. Analysis. This part involves a review of documents, interviews, and analysis of information obtained.

(1) On-site Review. An on-site review is the visit by the evaluation team to the organization or program office being evaluated, typically for one workweek and may involve travel expenses.

(a) On the first day, the evaluation team will have an initiating or opening conference with the Assistant or Regional Director and selected program managers. The conference will include a discussion of evaluation objectives, schedule of activities and interviews, any other areas of concern the managers may have, and points of contact for the evaluation team. The opening conference may be waived if the evaluation schedule begins with reviews of programs/offices outside the Regional Office.

(b) During the week, the team will conduct interviews and examine documents and records. The team may need to request clarification about documents being reviewed and visit other program offices and agencies. Employees may also volunteer to speak with the evaluation team during the week.

(c) On the last day of the review, a close-out meeting with the Assistant or Regional Director and selected program managers will occur during which a verbal summary of findings and recommendations will be presented. Program managers will be given the opportunity to clarify issues and respond to the findings and recommendations during this meeting.

(2) Alternative to On-site Review. An alternative to the on-site review is the conduct of an evaluation by a team without visiting the organization(s) or employees of the program(s) being examined. Documents are obtained through the mail or electronic means; meetings and interviews are conducted by telephone or electronic mail. An alternative method involves the same evaluation activities as those described for an on-site review.

(3) Self-Assessment. A self-assessment is an evaluation conducted by team members who are employed by the program/office being examined. A self-assessment involves the same evaluation activities as those described for an on-site review.

C. Reporting. In addition to the verbal report provided on the last day of the on-site visit (close-out), the team will prepare a draft report for comment and a final report signed by the convening authority. Reporting requirements are contained in 420 FW 3.

D. Followup. The completion of corrective actions stated in the recommendations will be monitored by the implementing office(s) and the Policy and Directives Management Staff.

2.3 Definitions.

A. Annual Evaluation Plan. The written document, approved by the Director, that outlines the organizations and programs subject to evaluation during the fiscal year.

B. Convening Authority. The member of the Directorate responsible for convening a team to conduct an evaluation.

C. Days. Calendar days.

D. Evaluation. The examination of organizational performance, programs, and/or processes against a standard of acceptable performance.

E. Evaluation Workplan. The document prepared by the evaluation team that contains evaluation coverage, sites, and the schedule of activities.

F. Evaluation Report. The written account of evaluation activities and results.

G. Interview. A conversation, either in person, by telephone, or by electronic mail, designed to elicit information that serves the purpose of the evaluation.

H. Management Control Program. The program described in 290 FW that covers the evaluation of management controls.

I. Program Evaluation. An assessment, through objective measurement and systematic analysis, of the manner and extent to which Federal programs achieve intended objectives (Government Performance and Results Act of 1993).

J. Standard. A level of achievement or established measure of comparison for quantitative or qualitative value, e.g., laws and regulations, management controls, Service priority objectives, performance measures developed as a result of the Government Performance and Results Act of 1993, and program policies, procedures, goals, and objectives.