Shows & Panels
- AFCEA Answers
- Ask the CIO
- The Big Data Dilemma
- Carrying On with Continuity of Operations
- Connected Government
- Constituent Servicing
- Continuous Monitoring: Tools and Techniques for Trustworthy Government IT
- The Cyber Imperative
- Cyber Solutions for 2013 and Beyond
- The Data Privacy Imperative: Safeguarding Sensitive Data
- Expert Voices
- Federal Executive Forum
- Federal IT Challenge
- Federal Tech Talk
- Mission-critical Apps in the Cloud
- The Modern Federal Threat Landscape
- The Path from Legacy Systems
- The Real Deal on Digital Government
- The Reality of Continuous Monitoring... Is Your Agency Secure?
- Veterans in Private Sector: Making the Transition
Shows & Panels
Monday - Friday, 6-9 a.m.
Hosts Tom Temin and Emily Kopp bring you the latest news affecting the federal community each weekday morning, featuring interviews with top government executives and contractors. Listen live from 6 to 9 a.m. or download archived interviews on our daily show blogs.
Reevaluating program evaluations
Wednesday - 8/11/2010, 10:41am EDT
In fact, convincing budget overseers that you're performing serious, metrics-based evaluations will be key to getting 2012 dollars.
Now OMB says it will set aside $100 million to help agencies establish evaluation programs and teams.
Federal News Radio told you earlier this week during an In Depth Conversation about the memo from former OMB Director Peter Orszag regarding agency performance.
So, what do you need to do at your agency to comply?
Dan Chenok is former branch chief for information policy and technology at OMB, was a participant in the Obama transition, and is now with the IBM Center for the Business of Government.
He explains what the changes will entail.
"[It] really talks to the integration of program evaluation with budget planning. For many years, OMB has overseen program evaluations of particular programs in somewhat of a series out of fashion, meaning that a large program would be evaluated, that evaluation would occur, and then time would march on and another one would occur. What this memorandum does is to say that that approach, which is to take one program at a time and subject it to rigorous program evaluation is being stepped up by several orders of magnitude by tying it to budget planning and to the delivery of dollars in the fiscal '12 budget process."
Agencies will now have to put forward comprehensive, scientifically sound estimates of what they will need in order to complete these evaluations when they ask for money, "So, OMB isn't saying that you can just come forward and put anything through, you really have to do your homework and determine which programs should be evaluated and how best to do so."
Chenok says that this is something relatively new for OMB and federal agencies. There have been program evaluation initiatives in the past, but those were generally conducted in terms of management. In the past, some budgets also contained stipulations about particular programs or sets of programs, but what OMB wants to do now is different.
"In my memory, there's no precedent for a program-wide evaluation initiative that's tied to a pot of money that comes forward in the budget process like this. It really shows the seriousness with which OMB and the administration is taking significant review of the effectiveness of programs as it makes budget decisions."
In addition, Chenok says that program managers should be prepared to ask for help. He says the wording of the memo specifically says, "rigorous and independent," which means someone other than the program manager should be conducting the evaluations.
"Setting in motion the process for doing a statistically rigorous evaluation methodology, which the OMB memo talks to, and which there is a long history of and precedent for in the government, and identifying an independent source, whether that source comes from within the agency . . . or going through a grant process or a contract process, having an independent entity come in is very important for the validity and the salience of the results of the evaluation."
Third parties often mean more time and resources, though, and Chenok says this new memo means program officials really have their work cut out for them now.
"[They] really have a full time job and they have a lot to do to get their operations moving and to serve their constituents and to operate in the federal environment where there are many different stakeholders. Having that independent voice that can really come in and take the time to do a program evaluation properly is critically important."
One challenge has to do with the fact that there is no set standard for evaluating the success or failure of all programs. Though the units of measurement can vary, though, Chenok says OMB is striving for the development of a valid methodology which will assess the impact across programs, regardless of how they are measured.
"This type of evaluation really has a significant return on investment, because a small amount of evaluation dollars can yield a large result in terms of indicating which programs work or, more importantly, which programs aren't working and from which dollars should be redirected. In the past, that type of activity has resulted in both the operations of particular programs and in moving resources from resources from programs that are poorly performing, to programs that are performing well."