PERFORMANCE MEASURES FOR FEDERAL AGENCY WEBSITES:

FINAL REPORT

To Sponsoring Agencies:

Defense Technical Information Center

Energy Information Administration

Government Printing Office

Charles R. McClure, Ph. D. <cmcclure@lis.fsu.edu>

Francis Eppes Professor, and Director

Information Use Management and Policy Institute

School of Information Studies, Florida State University

Tallahassee, Florida 32306

Voice: 850-644-8109; Fax 850-644-9763

J. Timothy Sprehe, President <jtsprehe@jtsprehe.com>

Sprehe Information Management Associates, Inc.

4201 Connecticut Ave., NW, Suite 610

Washington, DC 20008

Voice: 202-364-5300; Fax: 202-364-5302

and

Kristin Eschenfelder, Ph.D.

Assistant Professor

University of Wisconsin

Madison, WI 53706

October 1, 2000

Available at URL:

http://fedbbs.access.gpo.gov/measures.htm

Performance Measures for Federal Agency Websites

EXECUTIVE SUMMARY

1.0 INTRODUCTION

As federal agencies move to the Web environment as a primary means to disseminate information to the public, they require performance measures concerning the extent to which their websites are successfully presenting and conveying the government information the public needs to access and use. Performance measures for rating the success of federal agency websites in meeting their goals are consistent with the provisions of the Paperwork Reduction Act Amendments of 1995, the Clinger-Cohen Act, the Government Performance and Results Act (GPRA), and OMB Circular No. A-130. Foremost among these is GPRA, which mandates the adoption of a strategic and annual planning process tied to budget and authorization cycles and based on established and measurable performance indicators for every program.

The purpose of this study was to develop performance measures that will assist agencies to assess the quality and usefulness of their websites and to improve public information access services of those websites. The study was sponsored jointly by three federal agencies: the Defense Technical Information Center, the Energy Information Administration, and the Government Printing Office. The investigators conducted a literature review, reported in Chapter 1. They undertook an analysis of laws and policies affecting website performance measures, reported in Chapter 2. They met with agency representatives individually and collectively and analyzed documents provided by the agencies, as Chapter 3 explains. Chapter 4 describes the performance measures that emerged from the study. Chapter 5 sets forth key issues and recommends next steps.

The study employed a multi-method approach to data collection and analysis. Given the relatively short time line for the study and the limited resources available the following approaches were employed:

Literature Review. The study reviewed selected literature and reports from both inside and outside the government.

Group Interviews with Participating Agency Representatives. The investigators (1) met individually with representatives of each participating agency, (2) met with the participants as a group, and (3) conducted a number of individual follow-up discussions with representatives of the three agencies about activities and procedures for assessment in their agency.

Best Practices. Each of the three participating agencies provided the investigators with (1) descriptions of their assessment techniques, and (2) documentation and reports related to these assessment techniques; these materials were reviewed and analyzed by the investigators.

Policy Analysis. The investigators reviewed existing federal policies, laws, and guidelines to produce an overview of policies affecting the development and management of federal websites.

Survey. The investigators conducted a survey at the FedWeb 2000 conference held in March, 2000 at Bethesda, MD, that provided a snapshot of current practices and key issues of concern to federal officials regarding the assessment of federal websites.

Participant Review. Officials from the three participating agencies had a number of opportunities to review and discuss preliminary findings and reports developed by the investigators.

Website evaluation can be defined as the use of research or investigative procedures to systematically investigate the effectiveness of a web based information system on an ongoing basis. Evaluation is used in conjunction with a number of activities:

· Planning, Goal Setting and Determining the Degree to Which an Information Systems Organization is Meeting Set Goals

· Decision-making and Resource Allocation

· Determine the Effectiveness of a Previous Decision

· Determine the Degree to which the System/Service Adds Value to the Organization

· Provide Trend Data to Assess Change Over Time

· Contribute to Continuous Improvement Efforts and Benchmarking

· Identify Problems and Possible Solutions

· Empower Organizational Actors to Seek and Enact Solutions

· Develop Accountability

· Organizational Learning

Performance measures are the tools that enable evaluation. Performance measures typically provide measurements of the following:

Inputs and Efficiency. The use of resources in providing or accessing web services.

Effectiveness. How well the networked information service met the objectives of the provider, user or customer.

Outputs. Indicators of the products or services resulting from the use of those resources.

Extensiveness. How much of a service the network provides.

Service Quality. How well a service or activity is done.

Impact. How a service made a difference in some other activity or situation.

Usefulness. The degree to which the services are useful or appropriate for individual.

· Adoption. The extent to which institutions or users integrate and adopt web resources in organizational or individual activities.

One can see federal agency web system evaluation as a special subset of web information systems (IS) evaluation that in turn is a subset of general IS evaluation. IS evaluation has become an increasingly important topic within the competitive U.S. business environment. The last five years has seen a surge of interest in website evaluation. Federal website evaluation has been ongoing since the inception of federal websites. One early landmark was the World Wide Web Federal Consortium publication of suggested guidelines for federal website development.

In summary, website evaluation is an important part of the ongoing management and enhancement of federal agency websites. Evaluation, whether as part of a formal planning process or as a stand-alone activity, can provide managers with key information that can aid decision making. Although web sites are a relatively new medium, IS evaluation has a long history, providing tools and lessons learned that federal web managers can apply to their web evaluation efforts. Current academic research on federal agency web evaluation also provides strategies and tools that agencies can adapt for their efforts.

2.0 Federal POLICIES Affecting WEBSITE Development and Management

Federal information policies establish the legal and procedural framework in which government information and services are made available to the public. The study provides an introductory review of selected U.S. Federal information policy instruments that affect the development and management of Federal websites. This listing is not comprehensive, but rather offers a general sense of the range of existing Federal policies that should be considered in the development, management, and evaluation of websites. Issues that Federal agencies need to consider in developing websites include information security, information privacy, information access, electronic records management, and intellectual property.

Table 2.1 is a summary presentation of the study’s elaboration of federal policies affecting agency websites. For each topic the table shows the relevant statute, executive order, or other key document, plus implementing policy guidance, if any. The right hand column summarizes the implications for websites. The text of Chapter 2 summarizes the policy instruments and includes pertinent quotations from the instruments themselves.

3.0 REVIEW OF AGENCY PRACTICES

The report provides an overview of evaluation and performance measures practices at the Defense Technical Information Center (DTIC), the Energy Information Administration (EIA), and the Government Printing Office (GPO) current as of July 2000. On March 15 and 16, 2000, the investigators interviewed agency personnel from each of the participating federal agencies in site visits to the agencies. Each of these agencies supplied documents in answer to the investigators' queries.

From each agency, the study team requested the following:

Table 2.1 Summary of Federal Policies Pertaining to Agency Websites

	Topic	Statute, Presidential Directive, or Other Document	Implementing Guidance	Website Implications
I. General Government Policy	A. Performance and Results	Government Performance and Results Act	OMB Circular A-11, Part II	Performance plans, goals, and measures for agency programs
	B. Customer Service	E.O. 12862, Setting Customer Service Standards	-	Identify customers, their needs, and set standards and benchmarks
	C. Accessible Information Technology	Rehabilitation Act, section 508	-	Information technology accessible to persons with disabilities
	D. Electronic Government	Pres. Memo on Electronic Government	-	Standardized access to and ease of finding government information, plus privacy and security
II. Federal Information Policy	A. National Information Infrastructure	NII Agenda for Action	-	Make govt. information more easily and equitably accessible
	B. Privacy and the NII	Principles for Providing and Using Personal Information	-	Guidelines to personal information users and providers
	C. Copyright	Digital Millennium Copyright Act	-	Protecting copyright in electronic media
	D. Rights of Access to Information	Freedom of Information Act	-	State FOIA procedures on websites
		Electronic Freedom of Information Act	-	Establish electronic reading room on websites
		Privacy Act	OMB Circular A-130, Appendix I	Handling of personal information
			Pres. Memo on Privacy and Personal Information in Federal Records	Review privacy policies and practices; update notices of systems of records
			M-99-18 on Privacy Policies on Federal Websites	Display privacy policies on websites
			M-00-13 on Privacy Policies and Data Collection on Federal Websites	Discouragement of and restrictions on use of “cookies” on websites; comply with COPPA
		Public Printing and Documents: Depository Library Program	-	Publications provided to depository library program.
	E. Paperwork Reduction Act	Paperwork Reduction Act	OMB Circular A-130	Framework for agency information management plan, including information dissemination
	F. Clinger-Cohen Act	Information Technology Management Reform Act	E.O. 13011, Federal Information Technology	Websites to be interoperable and standardized across government
	G. Security	Computer Security Act	Proposed Revision of OMB Circular A-130, April 13, 2000	Mission based performance measures for information systems
	G. Security	Computer Security Act	OMB Circular A-130, Appendix III	Security controls for federal information systems
			PDD 63, Protecting America’ Critical Infrastructures	Performance measures for website security
	H. Electronic Collection and Digital Signatures	Government Paperwork Elimination Act	OMB Notice: Implementation of the Government Paperwork Elimination Act	Increase and encourage electronic data collection and implement digital signatures
	I. Federal Records	Federal Records Act	General Records Schedule 20, Electronic Records, and various NARA Bulletins	Provide for management of records created on websites

I. Set of documents by which the agency evaluates its website material.

II. What types of statistics and performance measures is the agency currently using for its website?

III. What overall evaluation methodologies are currently in place and how are they conducted/operated?

IV. Can we determine the types of reports generated by the agency evaluation efforts and the success or quality of these reports?

V. To what degree are information policy instruments/guidelines (as per the compilation in I) being considered in the documents and reports?

To the above questions, the investigators added:

VI. Preliminary review of website

VII. Summary comments

The objectives of the review and assessment were to:

· Identify the basic evaluation methodologies, statistics, and performance measures currently in use at each of the agencies

· Provide a composite view of the three agencies that would allow each to compare its practices and techniques with others

· Consider which activities might constitute “best practices” for federal website performance measures

· Establish a first step toward developing statistics and measures for federal website performance measures.

DTIC. The DTIC website acts as a portal for the public to other sites designed for fuller public use. Further, it provides fuller service to users who are technologically sophisticated and familiar with the agency’s structure, context, and background. In the past, DTIC has relied on technical self-evaluation sources such as log file analysis, bandwidth usage and tracked web site response time. But recent efforts have also included user based evaluation techniques such as the Secure STINET customer satisfaction survey. DTIC is also involved in a larger CENDI Agency working group effort to incorporate greater use of web evaluation tools.

EIA. The task faced by EIA in putting its information on the web is an organizational conundrum. The EIA web committee has the unenviable assignment of posting highly diverse technical information from many sources on one website suitable for use by the layperson. To this end, and to their credit, the EIA web committee has recognized the need for continual evaluation and redesign of the site in reaction to a constantly changing environment. They have conducted studies to aid in the creation of the next version of their home page to be unveiled in the near future. The new design takes a channel approach, strives for less jargon, and is designed with the layperson in mind. To prepare for this new design, EIA chose to perform cognitive usability testing. The results from this usability testing suggest that EIA’s current web site design meets key usability requirements as most users could find the required information.

GPO. GPO established the GPO Access website to fulfill the mandate required by PL 103-40. Under the provisions of this law it has subsequently evolved to provide public access to many key government documents. As of November 1999, the website provided access to more than 104,000 titles on Government Printing Office servers and over 62,000 additional titles through links to other federal agency websites. GPO considers Permanent Public Access to be an important performance measure for federal agency websites. GPO Access maintains historical archives of information previously available on its servers, ensuring permanent public access to its collection of electronic government information products, and facilitating the same historical research that is possible through printed media.

As a content oriented web site, GPO Access primarily measures the number of documents retrieved by users as a selection of the units of content being distributed by the site. In addition, staff constantly evaluate other aspects of the website through focus groups, online user surveys, user comments, log analysis, trade shows and conferences. Further, GPO conducts user-training sessions. GPO has ensured that the information on its website is listed in commercial search engines. GPO also has conducted bandwidth usage studies to determine peak user times and peak usage pages. GPO has a well developed management approach for its website and has committed significant effort and thought to evaluating it.

The evaluation and measurement efforts at DTIC, EIA, and GPO demonstrate the wide range of strategies for measurement. The efforts also suggest that different strategies can be useful in different agency settings. For example, some agencies may be better able to employ regular and automatic log analysis techniques than to implement full-scale usability studies. Others may find that the establishment of formal usability labs to “test” the effectiveness of selected pages and content may be more effective. Ultimately, the “best” strategies may be those that best accommodate agency mission and available resources.

The review of agency practices described here may also be useful for other federal agencies to compare themselves against. It is not that one agency is better or worse than another; rather, that these agencies have taken different approaches to developing evaluation and measurement techniques. A range of issues and situational factors affect the success with which any agency can engage in an ongoing program of evaluation and measurement.

Each of these three agencies is in the process of developing and evolving their assessment techniques. Within this process each has struggled with a comprehensive planning approach to formally establish responsibilities, tasking, schedules, statistics, and measures to be employed to assess the quality of agency websites and their usefulness to users of that website.

4.0 PROPOSED PERFORMANCE MEASURES

The report describes and elaborates on the findings from the assessment of the materials collected from the three sponsoring agencies concerning the types of statistics and performance measures the agencies were using. Appendix A, Criteria for the Evaluation of Federal Websites, and Appendix B, Website Performance Measurement and Evaluation for GPO Access, supplement the information about performance measures.

As the investigators sifted through the data collected from the agencies and from their own analyses, they realized that the term “performance measure” can be understood in several ways. Many things lumped under the general heading of performance measures might be considered conditions prerequisite to performance measures. In other cases, the materials pertained to management or operational considerations rather than measurement of the website’s performance. Performance measures necessarily imply performance goals and the measures yield indicators of progress toward achieving the goals.

The study divided performance measures into three types:

1. Legal and policy conditions affecting agency websites

2. Management and infrastructure factors

3. Performance measures in the stricter sense of the term

These three types of performance measures are summarized below in Tables 4.1, 4.2, and 4.3.

Table 4.1. Checklist of Legal and Policy Conditions

Pertaining to Federal Agency Websites

Statute/Policy	Checklist Question
1. Privacy	A. Does the website contain a privacy notice that complies with the OMB guidance and model language for federal websites?
	B. Does the website avoid the use of “cookies” or observe OMB-stipulated restrictions?
	C. Does the website comply with the Children’s Online Privacy Protection Act, particularly with regard to collecting personal information from children? (Same as 9-B below)
2. Freedom of Information Act	A. Conventional FOIA: Does the website contain clear procedures for requesting agency records under FOIA?
2. Freedom of Information Act	B. Electronic FOIA: Does the website contain an Electronic FOIA Reading Room?
3. Copyright	A. Does the website management include provisions to ensure that copyrighted materials are not posted without permission from copyright holders?
3. Copyright	B. Reuse Restrictions: Can the site content be freely reused without restriction?
4. Accessibility	Does the website make provision for accessibility for persons with disabilities in accordance with §508 of Rehabilitation Act?
5. Security	Does the website management include adequate provisions for protecting the security of the website and other agency information systems?
6. Paperwork Reduction Act	A. Do information collections undertaken via the website have appropriate OMB clearances?
6. Paperwork Reduction Act	B. Does the website comply with provisions for the Government Information Locator Service?
7. Depository Library Program	Have publications posted to the website been made available to the Federal Depository Library Program?
8. Government Paperwork Elimination Act	A. Does the website permit and encourage electronic information collection?
8. Government Paperwork Elimination Act	B. Does the website permit use of digital signatures?
9. Federal Records Act	Does the website management include adequate provision for identifying website records and transferring records to agency record keeping systems?
10. Access for Children	A. Does the website comply with the President’s April 1997 guidance on expanding Internet access for children, parents, and teachers?
10. Access for Children	B. Does the website comply with the Children’s Online Privacy Protection Act, particularly with regard to collecting personal information from children? (Same as 1-C above)

Table 4.2. Management and Infrastructure Factors

Pertaining to Federal Agency Websites

Type of Factor	Factor
1. Infrastructure	A. Does the website have sufficient technology infrastructure (bandwidth, server controller array, etc.) to ensure adequate response time?

To Sponsoring Agencies:

4.0 PROPOSED PERFORMANCE MEASURES

Table 4.1. Checklist of Legal and Policy Conditions

Statute/Policy

Checklist Question

Table 4.2. Management and Infrastructure Factors

Pertaining to Federal Agency Websites

Type of Factor

Factor