Quality Assurance for Residential Retrofit Programs (Text Version)

Good afternoon, everyone, and welcome this afternoon's webinar from DOE's technical systems program. My name is Jim Grevatt and I'm going to be the presenter today. I work at a company called Vermont Energy Investment Corporation where I am a managing consultant. And just for background, I love residential retrofit programs. I've been doing this work for about 20 years, and just until last month I was the director of residential energy services for Efficiency Vermont where we had a fairly aggressive residential retrofit program. For about a dozen years before that, I worked with a local natural gas utility managing their efficiency programs including the residential retrofit program. And I started out in this business doing audits for the local low-income weatherization program. So it's a field that I know and really enjoy, and I'm really delighted to have the opportunity to talk with you about it today.

So as I said, today's webinar is about quality assurance for residential retrofit programs. And what we're going to do is briefly I'll introduce the technical assistance project, talk about why quality assurance is important and think about with you some realistic goals for quality assurance, and also talk about what I view to be the key elements of a QA program. And then we'll share some resources and do questions at the end. So if you do have questions, I'd invite you to type them in on the screen in the dialogue box on the right, and I won't be answering them till the end but I'll do my best to answer everybody's questions at the end of the session.

So the technical assistance program is sponsored by the Department of Energy, and it's designed to support the many different municipalities and other entities who have received funding through the ARRA program and including EECBG and TAP and Better Buildings grantees and recognizing that there are a lot of information to know about how you run clean energy programs. DOE is committed to providing the support to make sure that everybody can succeed with their program.

And I'm sure some of you have seen this before. TAP offers one-on-one assistance and there's a lot of information online as well. Webinars are happening. There is one every day this week after today, and I'll share that list at the end. And there's a calendar, there's a blog, there's information coming online every day about best practices and the new resources on all kinds of topics related to program delivery. And a little more detail about these and, of course, this webinar. The presentation will be available online so you can check back and look at this in more detail afterwards if that's a benefit to you.

And this particular series of webinars comes to you from TAP Team 4 which is about program design and implementation support, and there are a bunch of groups. As you can see on the screen, across the country we're providing support and as I said, I'm from VEIC. Today sitting in sunny Burlington, Vermont where it's 70 degrees here, almost Halloween.

So the topic for today: quality assurance, what is it and why do we need it. And I think that at the most basic level, quality assurance is really about assuring that the investment that programs make in energy efficiency, whether it's federal dollars or utility money or money from some other source, to make sure it's getting what it's designed to do and that's generally saving energy. But when we're talking about retrofit programs or any programs, we also want to make sure that customers get what they pay for. And generally what they are trying to pay for is durable, safe, and effective energy efficiency improvements. And we want to make sure that those improvements don't come with unintended negative consequences.

So there are a lot of pieces that go into making the dream of quality assurance come true, and I think one of the first things we need to think about is all the other elements of program design that go into making a quality assurance program possible. And one of the things you want to do is make sure that your customers are steered in the right direction, that they're actually pursuing retrofit measures that will actually save them energy. And so protecting them from false claims and false advertising is a good thing to do right at the outset, and this is just something that I found on the web. It was a ten-second Google project, cut your energy use in half by putting in windows.

And we also want to make sure that the work that contractors do once they do get into a house is the kind of work that you want to see. So for example, the picture on the left, using duct tape to seal leaky ducts isn't a good solution. Dries out and it falls off and it doesn't do anything. If you have a contractor who's going into an attic and they're going to insulate the attic, but if they don't do the air sealing to stop warm air leaking into that attic in the cold climate here, it could have consequences. Warm air leaks into the attic and it condenses on the roof sheeting and you get frost and when it freezes, you can get mold growth. And I've been in attics where the roof sheeting right above the attic hatch, you could put your finger through because it's rotted out. These are the kind of unintended consequences we'd really like to avoid.

Similarly, if you're sponsoring a program that's encouraging people to put in attic insulation, we want to make sure that attic insulation is going to work. The idea that more is better doesn't necessarily apply if you don't install it right. You have a contractor who is going to insulate an attic to save a lot of money but insists that you need to have ventilation to deal with any moisture issues. Well, ventilation can be a good idea. I don't think this contractor probably expected that the water was going to run out of the vents and then freeze into icicles. It's not usually what it's designed to do, but, again, these are the kinds of things that you want to avoid having your customers call you about as a result of your program.

And most importantly, when we're talking about retrofitting houses, contractors can do things that will cause the occupants harm, and you need to be really careful about this. This is a house where clearly the fireplace drafts into the house. The house works as a better chimney than the chimney does, at least at some points in time. So for a contractor to go in and say I'm going to save you a lot of money by tightening up your house, if that contractor didn't address the fact that the chimney doesn't vent up -- excuse me, the fireplace doesn't vent up the chimney, that could send people to the hospital or worse. So very important to address these things through quality assurance.

When we're thinking about quality assurance, though, we know there are a lot of things that can happen that are bad, but we have to set reasonable expectations. So we'd love to say every job is going to be perfect, that that's what we're going to do, that's going to be our role. But that's really hard to do given for any program that I've been a part of, there are limited resources. Even where the resources are not so limited, for instance, low-income weatherization where, in Vermont, typically there's a hundred percent inspection, we still can't make every job be perfect. And so I propose that a reasonable expectation for a quality assurance program is to provide reasonable assurance that contractors are installing and reliably reporting appropriate measures that will save energy as represented and keep any harmful consequences from occurring for either the buildings or their occupants.

And if that's the goal of your quality assurance program, there are a lot of things that a program can do before you get to what we think of as actually doing the inspections to increase the chances that you're going to be successful. So certainly having clear goals for the program can lead to clear objectives for your quality assurance program. If you have a program that's all about getting people to put in new high-efficiency boilers, what you're going to do on quality assurance is going to be very different than if you have a program that's about getting people to install comprehensive retrofit projects including air sealing and insulation and any other appropriate measures. You need to really define your program goals very clearly before you set the quality assurance objectives.

Contractor qualifications being well defined are very, very helpful in doing quality assurance for your program. If you were trying to do a comprehensive retrofit program and it's open to any contractor with no qualifications, the amount of work that you will do in trying to make sure that the work is done to your expectations will be vastly, vastly greater than if you have clear qualifications required that narrow down the pool of contractors doing the work to those who have the best possibility of understanding what it is you want them to do.

And along with those qualifications, we strongly recommend having a written agreement with your participating contractors. Give them a bar that they have to clear in order to participate in the program to improve the chances they'll do good work, and then write down the expectations and ask them to sign it so that you have clear understanding. And, of course, training both on the terms of the agreement and the expectations of the program but also technical training. I know this is being discussed as a topic in other webinars. It's critically important for folks who aren't coming from this industry, who aren't coming from this trade, and even many who are, there's -- it's a big leap to get to the point where you're a skilled installer in comprehensive retrofits. So a lot of training is required.

I'd like to think of this as doing everything you can to make the quality assurance easier and have a better outcome because, again, for folks who have any experience with programs, when you have calls from customers because they're not happy, when you go out and do inspections and you find that the contractor completely missed the boat on what you were hoping they were going to do, that becomes very hard to deal with. Better to try to have them get everything right, put all the pieces in place so that your quality assurance program is easier to run.

Also in thinking about quality assurance, it's really important I think to recognize that different regions, different markets are at different stages of readiness for retrofit programs, and you have to reflect that in your quality assurance program. You can't do the same program in Indianapolis, for example, as you might do in Buffalo. In Buffalo there's been -- ______ has sponsored a Home Performances with Energy Star program for ten years. There's a certain level of maturity there. In Indianapolis it probably is at a younger stage of development.

And having a clear remediation process for complaints and deficiencies is really critical. Again, if you start doing inspections and you find that the jobs don't meet your expectations but you haven't thought about what you're going to do about that, you find yourself in kind of a tight spot. You know you want to work with the contractor to make it better. You know you want to help the homeowner, make sure that they're satisfied with the job. You don't want to really be trying to think that through on a case-by-case basis as it comes up. Great to have a clear process where the contractors understand it before you get there.

So there are a lot of different things that you can do in thinking about what you want to accomplish through quality assurance. And having defined the program goals, I think it makes sense to define what it is you're going to try to do. If your program is promoting comprehensiveness, then you probably want to check to see are the contractors delivering comprehensive work or are they only insulating attics. I think it's generally true that you want to confirm the customers receive what they got -- what they paid for. If you have specifications in your program, certain work specifications, well, on quality assurance, you want to make sure that they met those specifications. We say to the extent possible ensure that customers are satisfied with the work because, again, anybody who's done this for a while knows that some customers aren't going to be happy no matter what you do. But you want to make every effort to make them happy and help them understand what the work is that occurred, how it was done, and so that they're happy with what they got.

Some programs, and I think this is a good approach, will make a statement, a program objective that we're going to really try to work cooperatively with our participating contractors because you want to keep the contractors involved in the program, you want to keep them continuously improving. If you go in as the inspection police, especially with a new program and especially with contractors who may not have as much experience, you can scare them off pretty quickly. And generally better to try to keep them improving and keep them coming along and growing with the program.

Programs who do quality assurance often find, whether it's a role they want or not, that there's some expectation they're going to resolve disputes. So you may want to state that as an objective: we're going to try to proactively resolve disputes before they escalate. Most of the time I think programs can do that but not always.

And, of course, when -- with any program that has funding attached to it, there are reporting expectations. And with retrofit programs you rely on contractors often to -- they're your source for the data about what happened in the house. So you want to make sure that there's some way that you're monitoring what they report, that it's accurate, that they're doing it according to plan, and that it's all being tracked properly.

Sorry, computer froze there for a second. Always a frightening thing on a webinar. Back online.

Other things that you can think about in terms of putting pieces in place that will make your quality assurance in place more successful. Certification and accreditation of contractors can be an enormous benefit to your program. In effect, certification standards, national standards, can save you from having to work all of those requirements out on your own. Building Performance Institute, BPI, RESNET, there are other organizations but those are the two primary organizations that have some certification standards for how work is done that can just help you enormously.

And for clarification for folks who are new to this, typically certification is about an individual's credentials. The certification assures that the individual who is doing the work understands the requirements. Accreditation applies typically to companies and it puts some prescribed practice requirements on the company as a whole. So, for instance, with Building Performance Institute, I could be a certified individual. It just means that I've passed some tests and I have some understanding of the practices described by BPI. If I'm an accredited company, I'm expected to do my work according to those standards.

Different markets are probably open to different requirements along these lines. It's a bigger leap to require accreditation than to require certification. But if your market's ready for that, there are benefits to requiring accreditation and basically it shifts some of the quality assurance liability off of the program and puts it on the accrediting organization which, again, can be a very nice thing.

Participation agreement with the contractors. There's so much that you can cover in this agreement and, again, having the clear communication up front with the people who are doing the work will save you many, many headaches down the road. The things that you can do in the participation agreement that I would recommend that you do, define your standards. If you're -- you have an attic insulation program, do you just say insulate the attic? Do you say air seal the attic before you insulate it? Do you say insulate the attic and air seal it to BPI specifications or BPI standards?

There's a new work product that's coming online soon. It should be available early in 2011, and it's actually going to be released, my understanding, for public comment in mid-November. It's being developed by DOE. The National Renewable Energy Lab is doing this work on behalf of DOE, and standard work specifications, which actually outline -- they define the proposed requirements for high-quality work and the minimum conditions necessary to achieve the desired outcomes of a given energy efficiency retrofit measure. So, again, written description of what's expected when you insulate an attic. Having an agreement with contractors that says in order to participate in this program, you need to insulate attics to this standard is a great benefit to have that understanding well defined.

Also in the agreement if you're sponsoring any direct install lighting or water conservation measures that you want the contractors to do on site, lay that out in the participation agreement. You can put in information about how the incentive structure works, what you have to do in order to be eligible for incentives because it's not a nice call when a customer calls you and says, "Well, my contractor promised me I would get incentives for this work and this is what I did," and you have to tell them, "Oh, I'm really sorry. The contractor didn't understand and they didn't do the right thing and I can't give you the incentive." So lay it out in the participation agreement.

Something that -- an area where you hate to even have to put anything into an agreement but I recommend that you do, professional conduct. There are things that you don't want contractors doing on site. Lay out what's not acceptable. And you also want to put in -- you may want to put in some language about who's responsible for meeting code and about health and safety. And however you shape the participation agreement, it's critically important that you do training and question-and-answer sessions with the contractors about it. It can be a little complex. Depending on the sophistication and the experience of the contractors, some of this stuff is going to be very new. And it's one thing to write it down; it's another thing to have a dialogue and make sure that they understand what the expectations are.

So just some sample language from a -- actually from an Efficiency Vermont participation agreement with contractors where we just want to say, you know, we expect you to do accurate assessments. This isn't about a drive-by audit. We expect you to recommend comprehensive work scopes and we expect you to behave professionally. Having it in writing and having the contractor agree to it with a signature gives us something to point to if there are problems in the future. But more importantly, it just lays the expectations out so hopefully we avoid those problems in the first place.

And here's some additional language we in Vermont included in our participation agreement. Contractors are not going to get any incentives for their customers if they disturb vermiculite insulation because it might contain asbestos, and we don't want anybody to get sick and we don't want to own any liability for that. About that, everything should meet BPI standards. There's -- lead-safe practices are required for contractors. I'm sure you all know for doing many kinds of activities, it'll potentially disturb lead-based paint. We want the contractors to know, even though it's the law, we still want them to know that we expect them to comply with that law. And there are other issues as well that come up. Failure to adhere to these health and safety issues, you're going to be out of the program if you don't adhere to the requirements here.

So a good resource -- and there's a link to this later in the presentation, Home Performance with Energy Star has a sponsor guide that lays out their expectations of anybody who's sponsoring a Home Performance program, and there's a section in that sponsor guide about quality assurance. And it's right at the beginning of the quality assurance section they say you need to have a quality assurance plan and your plan needs to talk about how you're going to address these six points. And by implication, it says you need to have these in your quality assurance plan. And even if you're not doing Home Performance with Energy Star, it's a great place to -- it's a great resource for designing a quality assurance program, and I think these points are spot on for what you should have.

So you need to have a reporting process. You need to have a review process for looking at the reports that the contractors send you. You need to have some way of hearing from your customers. Customers are -- you rely on them in this program. Sometimes we focus on the contractors because if you don't have contractors, you don't get any work done. But the same thing, if you don't have any customers, you still don't get any work done even if you have the contractors. Home Performance with Energy Star has an on-site inspection protocol including sampling rate they say here of five percent. For some contractors and some programs, that's probably perfectly adequate. For many contractors at different stages of their experience or the program maturity, that's probably nowhere near enough. But they lay out what their expectation is, and similarly in your QA program, you can lay out what your goal or your expectation is.

Home Performance with Energy Star says you need to have a conflict resolution mechanism. I agree. You should have this laid out in advance, and you need to have a way of keeping track of the results from all the inspections for the surveys and the corrective actions.

So let's look at these in a little more detail. For a reporting process, it's kind of basic. You can't really monitor the work if you don't know what's happening. Sometimes contractors will do a bunch of jobs, and especially smaller contractors who don't have a staff and an office to handle all the business transactions, they're focused on doing the work. And maybe six months down the road they'll remember that they need to talk to you about it or let you know what they did. So it's nice to have a process in place, actually an expectation that they're going to report the work on an ongoing basis so it doesn't stack up and they're going to give you details about what's going on. You know, where did the work take place, who's the owner, what were the conditions that they found, what were the improvements that they installed, what did it cost, how much energy do they think the customer's going to save. And hopefully they have some tools or perhaps the program has provided them with some tools for estimating the energy savings. But, again, if the contractors are doing that calculation, you want that information, and you want to know what health and safety conditions they corrected. And part of the reason for asking that in a report is to make sure that they're actually looking for it and that they're addressing those health and safety considerations.

And you need to review those reports for a couple reasons. Again, it kind of depends on what the key objectives of your program are. If you're all about comprehensiveness, well, look at the report. Was it comprehensive or was it just, you know, we're going to put in a new furnace? Did they address the health and safety? I can't say that enough. It's critically important. Especially if you're doing comprehensive retrofit, you have to address the health and safety issues in the building. Did they estimate the energy savings properly? Did they use the tool correctly? Or are they telling customers that they're going to save 50 percent of their energy by replacing their windows? Any other requirements you have. If you require specific materials and so forth, review the reports and make sure. It's a good tool for making sure that the contractor is doing what you want them to do, and you can learn a lot from reviewing the reports more quickly and less expensively perhaps than going out on site.

I'm only going to touch on customer feedback. It's really important. It's great to have surveys of your customers. If your program is going to go on for a period of time, and I hope they all are, it's great to design a survey and use that survey over the duration of the program really without changing it very much so you can see the trends in how customers feel about what their experience has been, what suggestions they may have for program improvement. It's a great tool for really both gauging the satisfaction level and, again, if you don't have satisfied customers, it's going to be very hard to continue to grow the program. So get that information, gauge their satisfaction, continuously improve, and continue to get new -- more new customers engaged. And another aspect of customer feedback, when you do site inspections or maybe even want to do some follow-up phone calls, sometimes customers will tell you things in a casual conversation that they might not put in writing. So it's a great way to learn from people first hand. And the anecdotal information is just a great tool for program improvement.

And then I think getting into more of the on-site inspection protocols, and I'm not going to into this in great detail about what you should do on site at least at a technical level but I do want to talk about it for a few minutes about the kinds of things you need to think about. So I think it's important to me to recognize that the protocols for your on-site inspections will vary a lot depending on where your program is. What you're going to do if you have a new program that's starting out is going to be different than what you're going to do if you have a very mature program that's been in the field for ten years. And basically I would say with a newer program, especially where you have contractors who are kind of being trained as they sign up, your focus is going to be on training and technical assistance to build capacity and you're going to work on addressing any customer complaints. And you're probably going to focus on health and safety, making sure that there aren't -- that nothing seriously bad is happening out there. And it's a process that there's a lot of back and forth communication with the contractor. It's less about going out and seeing pass/fail and sending them something in the mail. It's engaged and cooperative generally. It seems to be the way that that works best.

Where if you have a mature program where you have a lot of contractors who have a lot of experience, you still want to focus on that training and the engagement but you can really have clear expectations for results, and I think you can be much more severe about the consequences of contractors not meeting those results. But at any point in time with on-site inspection, the ongoing feedback and communication is always, always important. Always think that we're trying to do better in a cooperative way. Contractors are our friends. We really rely on them. Without them, there's no program.

I think it's also important to note that the protocols vary with the contractor experience and the size, not just the newness of the program. Contractors who are new to programs, many programs will require a hundred percent inspection for the first five or ten jobs. Instead of the kind of sampling that Home Performance with Energy Star recommended, we're going to go out to every job you do for the first five or ten. And I think it's a great idea to think about actually going out to some of those jobs while the work is in progress instead of waiting till it's all done. You can see what the crew is doing. You can get a sense of how focused they are, if they're looking for the right opportunities. You can do some on-site training with them and it's a great opportunity. It's a great way to bolster the effectiveness of the work that the contractor does which in the end is really what you want to do.

Again, on the other hand, if you have a contractor who's very experienced, sampling protocol might be just fine. And often those sampling protocols work that as long as jobs are passing, maybe you do one in ten or even one in 20. If you get to a job that doesn't pass, well, maybe you're going to inspect the next two jobs or the next three jobs and make sure that they're on a good track again before you drop down your sampling rate.

I'd like to just also mention that depending on where you are in the country, whether you're in an urban setting or a very rural setting, probably has some effect or some impact on the kind of contractor base you're working with, and that may have an impact on how you do quality assurance. If you're doing with larger contractors with sophisticated business systems, the things that you can ask them to do in terms of reporting and QA, they're more able to handle those kinds of requests, whereas if you're doing with a two-person company, the owner does all the paperwork at night, you may have to change your expectations a little bit.

So just a couple of words about the qualifications of the field verifiers or the inspectors. There are a lot of different words that people use to describe what happens when they go out on site to do -- to look at the work. Inspection sometimes for some people implies that maybe it carries the weight of, say, a code inspection, and you may not want to suggest that your program is carrying out that rigorous an inspection because it may shift some liability to you if you're doing inspections. If you're just looking to verify that the materials were installed as specified, that's another term that a lot of people use.

So let's call them field verifiers for the moment. They have to know at least as much as the installers, preferably more. But they have to have the same training, they have to know how to use the same diagnostic tools. Field verifiers really need to be good communicators. It's possible you could just send somebody with technical skills, they go out, they do their quick assessment, and they leave. But homeowners like to know. They say, "Well, hey, did the contractor do a good job? Did I get what I paid for?" And if the field verifier says, "Well, geez, you know, they shorted you the cellulose in the attic and I don't know what we're going to do," the homeowner will get very upset. It's not that they don't have a right to that information, but how the information is presented will make a big difference in the outcome. If the person who's doing the field verification is sort of an agitator, can't communicate it well, gets the homeowner very worked up, it gets much tougher to get a resolution that everybody can live with, whereas if the field verifier can be tactful, not dishonest certainly but still effectively communicate want the situation is and how we'll work to resolve it, that can give everybody a better chance of successful resolution.

And it's not just about working with the homeowner and keeping the homeowner satisfied but it's also about the confidence in the brand. If you're out promoting an approach and your program has a label -- not necessarily -- I don't mean a label for a home but a brand, and your field verifier goes out and says, "Oh, this contractor didn't do a good job," and that homeowner starts talking to the neighbors about how the contractor didn't do a good job, the program's no good, it can cause a lot of damage. And field verifiers also actually need to be pretty good at documenting what they find. You can have great technical people who are not so good at the paperwork, and that doesn't serve you very well in a quality assurance program. And, of course, the reason for that documentation, both because it helps you learn how to improve the program, but also if you do run into problems with a contractor with the performance, if you don't have it well documented, it's very hard to act on.

So this is Pat. Pat's been doing field verification inspection for a long time, and he is well versed in blower door. For anybody who knows blower door, this picture's kind of old because it's got those analog gauges instead of the digital gauges that we use now. But your inspectors, your field verifiers need to know how to use blower doors, they need to know how to check for combustion safety, and they need to know how to get up in an attic and measure the depth of the cellulose.

Another thing to think about for the on-site inspection is that I think typically we think about we're going to inspect the work or we're going to verify the work when it's all done, but there are benefits to doing -- to looking at different stages of the work. If you go out before the work has actually begun, you can verify that the existing conditions that the contractor is reporting, that you see the same conditions, that they're not making things up or gaming the system in a way or not doing a good job of doing the pre-inspection because they don't want to spend the time on it. So make sure that they're meeting the audit expectation.

If you go out while the job is in progress, it's a great opportunity for training with the crew, for talking about -- again, here's where the field verifier needs to have the same training as the installers or more. It's a great opportunity to say, "Hey, you know, when you run the blower door, look, you can find this leakage over here and here's something you can do to seal it up."

But after the job, quality assurance is also really important to make sure that it matches the proposal. And here's just a few items kind of listed out for things that you might look for at different stages, different types of site visits. You know, see if the contractors know how to run the blower door, make sure that they do. Duct blaster, that's part of it. Make sure that they're doing the combustion safety testing properly. Are they looking for all the health and safety? If you're out -- if a contractor's doing an assessment and you note that there's all kinds of scorch marks on the front of the boiler, it must not be venting properly and the contractor doesn't pick that up, you want to know that information. How does the contractor communicate with the customer? Are they writing a comprehensive set of recommendations? And if it's a program where you're asking the contractors to do some direct install during the audit, are they doing it? Did they miss opportunities?

If you're going out when the job is in progress, you can talk to the homeowner. How do they feel about things? Is the contractor cleaning up at the end of the day? Things like that. Are they following safe work practices? Are they -- from your assessment on the job, is it likely that this job is going to have combustion safety issues at the end? And is the contractor aware of that, are they dealing with it? Is the crew respectful? If you don't have a respectful crew, it makes it really, really hard to sell the program to the neighbors. If you have a great crew, the neighbors are going to talk to each other and you get more jobs out of it. You know, making sure they're using the right tools, that they're actually doing the work that was listed in the proposal and so forth.

And at the end, again, you can talk to the homeowner about the job. It's a great way to get information. Make sure that the blower door number is within some tolerance of what it was reported to be and so forth. Combustion safety. If they said they installed a thousand square feet of attic insulation, is that a thousand square foot attic? Does that match up? And so forth.

And with the information that you get from these site verifications, field verifications, inevitably at some point you're going to run into conflicts, and it's important to, as I said earlier, understand how those are going to be resolved. And I would say that there's kind of two categories of conflicts or deficiencies or failure to perform issues with contractors. There's installation issues and there's professionalism issues. You may need to handle those differently. If you have a contractor who is doing something unethical on site, they may not get another chance to make it right. If you have a contractor who didn't do the job as well as you wanted to, maybe you work with them to improve. Things that it's just worth thinking about in advance.

And defining acceptable levels of accuracy I think is good to know up front. Do you want to see them within ten percent, with 20 percent, within two percent of the reported numbers? Having defined those things up front will help you decide what to do if they don't meet the expectations.

And I always hope that programs can work cooperatively with contractors because one of the issues that certainly we face over time but continues to be an issue in many areas is you need a good group of qualified contractors if you're going to get any volume of work done. And as the goals nationally get bigger, the group of contractors that we need to do this work, it has to get bigger as well. So trying to continue to bring the contractors along through a quality assurance process will serve us all better in the end as long as the evidence is that they are coming along. At some point it may not work with some contractor and you need to know what your steps are for either putting that contractor on probation or suspension or finally all out removing them from the program.

Again, with these on-site inspections, have a tool, whether it's a paper tool or a -- not automated but an electronic tool for documenting the information that you find, keep track of it. Lovely if you can database it and actually use the results to track trends, but in the minimum, track it, look at it, and use it to inform your program.

As I said, quality assurance provides the necessary information for contractor improvement, and if you run into problems, it also gives you a tool for addressing those problems. And it also is a great way to learn about how you can make your program better.

So that was really a higher-level summary. As I said, I didn't get into the technical details of what inspectors need to do on site. I think just to reiterate, it's important to have realistic goals for your program for what you're going to accomplish; to clear, communicate, and train the contractors in the program requirements; make sure the field verifiers have the skills to do the inspections that they need to do; and adjust your QA program for where you are in developing the overall program. Different needs at different times and different needs with different contractors. I think it's pretty important for programs, while you want to have very clear and high standards, you need to have some flexibility if you want to get people to move along with you. Otherwise, you run the risk of alienating your contractor base and not having anybody to do the work anymore. And as always track and document the results.

So here's a couple of resources for folks. The Home Performance with Energy Star sponsor guide is great. It's specific to Home Performance with Energy Star, of course, so the requirements for sponsors -- but it has a lot of good information in it for anybody who's doing a retrofit program, whether you're being part of Home Performance with Energy Star or not. The standard work specifications that I referenced, this is a web link to a site that has information about the development of that -- those specifications and other parts of related processes, and there's a webinar that you can get to from that link to get more information about the work that's being done there. And as I said, it's expected that there will be a draft released for public comment in mid-November so you can follow what's happening on that website on that project because the standard work specifications I think will become a very valuable tool for program -- for laying out the expectations for contractors hopefully in the beginning of next year.

And Building Performance Institute, BPI, those certifications and accreditations for installing contractors, and the certifications are also a very useful credential for people who are doing inspections. RESNET has certifications relevant to field verifiers or inspections as well, the Residential Energy Services Network. And another thing that you could look at, another document is the Existing Homes Program Guide that the Consortium for Energy Efficiency has put out this past year which has just quite a lot of information on a variety of subjects about -- a little bit more focused on utility energy efficiency programs but a lot of it is really directly relevant.

And, of course, there's the TAP blog. It's a great source for information on the latest and greatest things that are coming along. And you can get to take advantage of additional resources through TAP either through the online solutions center with a link here -- and as I said, this will be posted online after the presentation is done -- or you can submit a specific request for technical assistance through the technical assistance center. And there's a range of one-on-one types of support that are available to you through that, and I strongly recommend folks if you've got questions or need help with setting up your programs, it's a great resource for you to go to.

Here's just a list of the upcoming webinars. As I said, one every day for the rest of the week, and the following week has -- following several weeks have more. My contact information here, and on the last slide there's also more contact information for other folks in Group 4, Team 4 for the technical assistance project here -- program. Excuse me.

So I'm going to open up to questions here, and let me see what we have. First question: should the program administrators or the field verifiers handle quality deficiencies with contractors? And it's a good question. You can certainly structure it either way, and I think probably I would say it depends a bit on the level of the deficiency. So my experience and my inclination would be to say it's reasonable for the field verifiers -- it's one approach and it's an approach that I like -- for them to start having some relationships with the installers. Now, you can say that you're running the risk of them losing their objective view, but I think because there's so much training that can be done through this QA process, for me that outweighs that argument about always maintaining objectivity and there are ways that you can monitor that. So I would say where there are training questions, where, "Boy, you didn't get all the air sealing in that attic. Here's how you might approach it on the next job," I think that's great to come from the field verifier.

But if you get to more serious questions like, say, some kind of unprofessional behavior -- a customer walks in and finds a contractor going through their clothes in the closet or something like that -- I wouldn't have the field verifier address that question. I would have a program administrator deal with that directly with the contractor.

So the next question is what's your opinion of the Home Performance Resource Center from Efficiency First. And I have to say I think highly of Efficiency First. I'm not familiar with that particular resource so I can't offer an opinion about it. There are a lot of great people involved in Efficiency First who I have a lot of respect for, but I'm not familiar with that resource specifically.

And if anyone else has any questions, now would be the time to put them in. I only have those two at the moment so I think we can wait another minute or so. And if we don't see any other questions, then we'll sign off for the afternoon.

Okay, let's see. Union labor always makes things tricky. Any advice? Agreed that it makes things very tricky. I have not -- in this field I have not worked with union labor so I don't have any advice that has any basis in anything other than gut. My gut has always tried to work cooperatively so if there are ways that -- to work with union labor, I think you probably have to work with the local and try to get some of the union staff on board with your program because if union -- if the union organizers support your work, I think you will have a much better prospect for success. That may not be too helpful but that's my only thought.

Here's a comment from somebody: there are a bunch of papers and studies at HPRCenter.org which is the research arm of Efficiency First and that's worth a look. Thanks for that.

Should contract be in standard legal form, hard to read for us non-lawyers, or in layman's terms? So contracts, assuming that -- I don't know if this question is about contracts between contractors and homeowners or sort of a participation agreement contract between a program and a contractor. I like lawyers who know how to work with English so I guess we certainly -- in my experience with Efficiency Vermont, any agreements that we had, we certainly had the lawyers review them, and I think that is worth doing. But we would try to make them accessible to the people who were going to be involved with them. We -- in some cases a participation agreement, we borrowed a participation agreement from another program that was probably 20 pages and we shortened it to about seven or eight just to make it be a meaningful document that people would actually read. But as a former program administrator, I'm a pretty big fan of making sure that you're covered legally so, unfortunately, I say have the lawyers take a look at it.

How do you think about allocating funds to support to your QA program? What does it cost? And, again, I think it varies with where you are. If it's a -- and I don't mean geographically but where you are in the program development. If it's a new program, the line between training and quality assurance can get pretty blurry. And if you don't have a strong contractor base, it's probably where you need to invest a fair amount of money. And in Efficiency Vermont five years ago the big gap was there were no contractors to do the work, and we now have in Vermont over 70 certified individuals, BPI certified, who can participate in the home performance program. But that's taken doing a couple classes a year to BPI certification standards for four or five years and a lot of one-on-one technical support for those folks. So the actual QA of doing the site inspections and all that isn't such a big part of the budget, but doing the training and the technical support to make sure that the QA is successful was a fairly substantial part of the labor budget. Percentage wise, I mean, we might have spent -- well, I bet we spent 25 percent of staff time over five years really on doing training and technical support.

If a MUNI policy is to not recommend certain contractors, how does it create a list for the program? List of resident certified contractors? I think what you do is you are very clear about what the requirements are in order to be on a list. So if you keep a list, you want to have some language with the list that says we're not recommending or endorsing any of these contractors. However, they all have complied with the requirements that we've put out there to be on the list. So that could be RESNET certification. It could be BPI certification. It could include minimum insurance requirements. There are a number of things that you might think about as requirements for putting on that list. It could include, as it does with Efficiency Vermont, the contractor has to have signed the participation agreement that says they're going to meet the terms of the program, they're not going to use the Home Performance with Energy Star brand inappropriately, they're going to behave professionally, they're going to guaranty their work for a year. So you can be fairly specific about the requirements I think and create a list based on that and then still decline to actually recommend individuals out of that list. And there are a number of programs who take that approach.

Can the energy auditors also be the program analyzers? I'm sorry, I'm not sure that I know what you mean by program analyzers. So I'm going to move past that one and if you want to put in a clarification, I'll look for that.

I'm trying to set up a warranty to protect homeowners against workmanship deficiencies over 12 months. Any thoughts or advice on this? I think, again, in the participation agreement if you require contractors to sign it and if it says, "I as a participating contractor will warranty any work that I install against defects and performance or workmanship for 12 months," that's a good step. Better if you have contractors who sign an agreement with the homeowner that's essentially a contract between the contractor and the homeowner that includes that they are going to have that 12-month warranty. Depending on where you are and sort of how business operates, you may or may not -- that may or may not be a common expectation. In rural areas like Vermont, not so common for contractors to have contracts with homeowners, much more done on a handshake kind of basis. But at least having it as a program requirement, it gives you at least some tool. If the contractor doesn't honor it, you can throw him out of the program.

Are the Home Performance with Energy Star resources available in languages other than English and are there resources for workers where English is not first language? That is a great question. I'm sorry, I don't know the answer, but if you go to EnergyStar.gov, I bet you can find it pretty easily if they have it. Or if it's about Home Performance in specific, I don't have an extension, I'm not going to be able to give that out over the phone, but Chandler Von Schrader at U.S. EPA in Washington manages the Home Performance with Energy Star program, and he would be a great person for you to follow up with.

Should QA inspection be conducted same time as install or schedule another appointment with owner and contractor? I think there are advantages to both, and if -- the main advantage to doing it at the same time as install, say if you try to -- well, disadvantage: it's pretty tricky to coordinate. But if you can manage to coordinate it so that you have a contractor who says, "I'm going to be finishing this job on Thursday. We'll be done by 4:00." So you get out there on Thursday at 2:00, the work is largely done. You can really -- you will be able to tell at that point how the job is going to finish out. And if you get out there and the crew is still on site and you find some deficiencies, you can take the contractor aside and say, "Hey, look, you didn't do this and this and you need to do it before you finish" and that contractor can get that work done, you don't run into the issue with the homeowner of having to say, "You know, geez, the contractor didn't do everything we wanted him to do so we're going to have to get him back." It saves a ton of money for the contractor because they don't have to come back out and re-stage all their equipment. So it can be a great way to do it. It's just hard to coordinate.

Some programs will schedule a time for the contractor to come out after the job is completed with the homeowner and with the inspector. That can be kind of burdensome on the contractor if you try to get them to commit that extra time. But it's a good thing to do if you can manage that. It just takes more time and sometimes harder to get them to agree to. And, again, the down side of that is if there's something that they have to come back and fix, it's more expensive, it's harder to schedule. It may be more of an inconvenience to the customer to say, "Well, the contractor's got to come back out. Can you arrange to take another day off of work so that they can come back out and finish this work?"

And I guess an advantage of doing it that way, though, let's say if you scheduled a week after the job is done, the dust is all cleaned up, the homeowner's not in the situation of just having lived through three days of construction project in the house so they might be a little more receptive to thinking about all of the good things that they've experienced as a result of the project.

Okay, my next question: do you think programs should be geared to align with Home Star or read proposed outlines or be flexible enough to adapt should these financing opportunities come along. Boy, that's a question that program administrators all over the country have been asking themselves. It's so unpredictable to try to figure out what's going to be happening at the national level. And I guess I would say that my experience in Vermont, the national legislation aside, there have been plenty of things happening on a state level. And I think this is true for probably many program administrators.

So we really have taken a strategy here of trying to be very clear about what our direction is but to also know where we can be flexible to adapt to other programs or other funding sources that may come along. We certainly went through a period of time where it seemed like every two or three months, something new was coming down. And if you're always flexing and changing and bending, you don't get any traction in the market. People really don't understand what it is you're trying to do. So we've really tried to kind of establish an anchor brand and a steady approach and think about how we can integrate and be flexible with other initiatives as they come because our conclusion at this point is they're going to come and go. We want to be in this for the long haul with a long-term approach, but these other things, they're going to come and go and it's going to be out of our control. We can't really ignore them. I mean, if Home Star were to pass, you know that customers and contractors would be ringing the phone like crazy trying to understand how it fits together. But very hard to plan for.

Next question is did you hire participating contractors to perform field verifier work. If not, who. And the Efficiency Vermont model is we actually have people on staff. There are plenty of other models where they do hire not participating contractors but some other third-party contractor. For instance, some programs will -- may have contractors who actually do the installation work and there may be other independent operators out there who are not doing installations, they're really only doing audits. I think it'd be a perfectly acceptable model to hire the folks who are only doing audits to do verification of the work. It gets a little tricky -- I mean, if I were to hire an installation contractor who also does the audits, hire that person to do field verifications, I wouldn't be comfortable having them do verification for work that contractors who are in competition are doing because it'd be -- I think put them in a -- put him or her in a position where it would be hard for them not to be in conflict. And they would both potentially get sort of inside information about what their competition was doing by reviewing the job, but they might also have some bias. So I would stay away from that.

And I will say again that all this has been recorded. It'll be posted on the website so folks can feel free to review it at their leisure or direct other staff or co-workers to it if you thought it was interesting, and I hope you did. And I think we'll wrap up with that. I thank all of you for your attention and hope you have a great afternoon. Thanks very much.