Text-Alternative Version: CALiPER Round 11 Test Results
Below is the text-alternative version of the "U.S. DOE CALiPER Program Summary of Most Recent Testing" webcast, held February 8, 2011.
Terry Shoemaker: Welcome, ladies and gentlemen. I'm Terry Shoemaker with the Pacific Northwest National Laboratory, and I'd like to welcome you to today's webcast, "U.S. DOE CALiPER Program Summary of the Most Recent Testing" (CALiPER Round 11). This is brought to you by the U.S. Department of Energy's Solid-State Lighting Program.
We are very happy to have our speaker today, Mia Paget. Mia Paget's primary areas of expertise are in mechanical engineering and energy policy with a specific focus on solid-state lighting. She has developed and managed the DOE Solid-State Lighting Commercially Available LED Product Evaluation and Reporting Program, CALiPER, since its inception in 2006, working closely with several stakeholder groups within the lighting industry from luminaire manufacturers to lighting testing laboratories to standards committees to develop and apply measurement — excuse me, to apply measurement methods and analyze results of extensive product testing.
Today's webcast will focus on a summary of CALiPER Round 11 testing, the result of each CALiPER round forms a snapshot of solid-state lighting technology status, identifying market trends, and issues related to solid-state lighting product performance. Round 11 looked at five applications: arm-mounted roadway luminaires, post-top roadway luminaires, linear replacement lamps, high-bay luminaires, and small replacement lamps.
Mia, please begin.
Mia Paget: Thank you, Terry, and thank you to everybody who's on the line today. We're happy to be able to give this presentation to you, and we hope it will be informative, and that we'll be able to answer as many questions as possible after the slides are given. Well, as Terry said, my name is Mia Paget. I work at Pacific Northwest National Laboratory, and I've been running the CALiPER testing program since we started it in 2006. The CALiPER testing program is one of many programs run by the United States Department of Energy to support solid-state lighting. Many of the folks on the call are probably very familiar with these programs, but a few might not be, so I'll just give a quick overview. The DOE provides support to research and development for solid-state lighting, but primarily also provides support that we call commercialization support, doing demonstrations, working on testing and quality control, doing lighting competitions and providing support to buyers and other market mechanisms. The CALiPER testing program is one that works very closely with our standards development work and also our SSL quality advocates work that work on the lighting facts program for labeling and quality with solid-state lighting products. Most of the CALiPER testing is done using standardized testing methodologies, IESNA LM-79 testing, which is a standard that we helped to bring to publication in 2008.
As I mentioned, we started the CALiPER Program in 2006. At that time, there were only about — we were only able to purchase about four products in the end of 2006, four commercially available solid-state lighting products, and they were difficult to find. Now there are many, many, many products on the market, and we've seen quite a bit of change over these last four years going from those initial products, pilot products that we found, to a market that's fully developed and covering a lot of different applications in solid-state lighting today. So we've been putting out summary reports every three or four months. You'll find links at the end of this presentation, and at any time, you can go on the website for the Department of Energy and download any of these summary reports that I mention during my talk. We also have detailed test reports on all of the products that are tested by CALiPER and those too are downloadable online. So if you have further questions about what I'm presenting, you can get further information online.
So in 2006, as I mentioned, we found four products to test. They were difficult to find, and they were just barely functioning as lighting products. And then in 2007 we ran three rounds of testing, so every few months we purchased more products and tested them at that time. The products that we were purchasing were really small products, small replacement lamps, desk lamps, undercabinet lights and a few small downlights. That's about all that solid-state lighting was able to achieve at the time because the LED devices were not yet powerful enough to fulfill bigger applications.
In 2008, in our Rounds 4 to 6, we added downlights to the suite of products that were tested and T8 linear replacement lamps, also started doing benchmarking because we discovered that a lot of folks had questions about how these LM-79 test reports, which is absolute photometry, how they compared to relative photometry that was used for traditional lighting photometric testing. So in 2008, we started doing benchmark testing and provided benchmark comparisons against downlight. Again, all these summary reports are available online, so you can go back and look at them if you're interested.
In 2009, we started seeing bigger applications come on the market, so we started testing streetlights, bollards, more downlights, 2x2 foot panels, 2x4 foot troffers, as we had done some in our Round 5 in 2008, but they started to take up a lot of interest in the marketplace, so we started to test more of them in 2009. And last year, we continued in that direction and did some rounds looking at parking garage luminaires, wallpacks, cove lights and always more replacement lamps. We almost always have a bunch of replacement lamps in every round of testing that we do.
So finally in the fall of 2010, we were running our Round 11 of testing. As Terry mentioned, it included roadway arm-mounted luminaires. You see the thumbnail pictures of them here, all the ones that we tested during this round of testing. Roadway post-top luminaires, two of these are solid-state lighting and two of them are benchmark products using metal halide lamps. We tested more linear replacement lamps along with some fluorescent high performance troffers. When we test these linear replacement lamps, we test them as individual lamps, and we test them in a troffer so that we can compare them to fluorescents tested in a troffer system. We tested a few high-bay luminaires. As they're just starting in this market, we're starting to find high-bay luminaires, but we just tested a couple of them to see how the initial products were looking, and then we tested a series of small replacement lamps. I'm going to go into more detail about most of these products that we tested and the results of most of these categories. I won't have time to touch on everything, so I do recommend that if you're interested in more details you go and download our summary report.
As an overview before I go into the details, we can look at the overall performance that we've seen over these last few years. This plot here looks at the average efficacy of solid-state light products. Now this is looking at the efficacy of the entire product each time, not just the LED device. We're looking at luminaire or replacement lamp efficacy, so we're looking at the product as a whole and how does it do as far as producing how many lumens per watt. And what we see is that when we started this program, even in 2007 when we started having full rounds of testing, our average efficacy was only 21 lm/W. In 2010, the average efficacy that we saw over the whole year was 50 lm/W, so more than doubled from what we saw in 2007. The average efficacy in Round 11, so at the end of 2010, was 57 lm/W.
Now efficacy isn't everything, and we'll go into a lot of other performance characteristics, but it is an important indicator and shows us that this market really is evolving quickly and significantly, in particular because this 57 lm/W average that we saw in Round 11 is achieved through larger products that are putting out a lot more light output than products that we were seeing in 2007 and 2008. And we're also seeing that most of these products are doing a better job of reporting their product performance. We're seeing Lighting Facts labels on products, and I will show one a little bit later in the presentation if you're not aware of them. And we're seeing many manufacturers that are publishing LM-79 reports showing detailed photometric performance of their products using standardized testing, which was not even available back in 2006 when we started the testing program. So we're seeing that solid-state lighting is covering more competitive applications and covering — doing quite a good job now actually meeting the needs of some of those applications, but at the same time, if you look at this large bar here, that's showing the — a difference in performance from the worst to the best. And the worst products that we tested in 2010, the worst solid-state lighting products, are not achieving even the average efficacy that we saw during the pilot phase in 2006. So there are still some products on the market that are really not using the technology to its best advantage, and the best products out there are doing very, very well achieving more than 90 lm/W for luminaire efficacy, which is quite phenomenal considering where we were just a few years ago.
Looking a little bit more detail at these averages again. What I've done here is provide you with averages looking at both replacement lamps and luminaires so that we can see the difference between these two different kind of categories, the larger products and the smaller products, and where we see basically the same efficacy across replacements lamps and luminaires. We see that the outdoor products, mostly luminaires, we tested this time were outdoor products. We see that they on average have a higher CCT, so these are colder temperature, color temperature products, and much bluer if you want, than the replacement lamps which tend to be much warmer light which is typical for the application. Now what's interesting about this difference is that while in theory SSL devices can achieve higher efficacy with colder color temperatures, we see that they have the same efficacy on average across these products. So the concept that colder color temperature products are going to be higher efficacy products doesn't bear fruit when you look at the final products on the market today which may have something to do with marketing strategies.
We also see that the CRI, the Color Rendering Index, of the outdoor products is a little bit lower than of the replacement lamps, which is pretty much what we would expect because the replacement lamps tend to be in applications that are going to be more color sensitive than the outdoor products.
And then we see that the power factor of the outdoor products tends to be — and the luminaires, tends to be close to 1.0, but what's interesting here is that the average power factor of the replacement lamps is over 0.8. Now when we started this program, we started the Solid-State Lighting Commercialization Support, we had plenty of folks telling us that it wasn't possible to achieve over 0.5 power factor in these small products, that it would be very, very difficult. And we're seeing now that actually it's very possible, and that the average is higher than 0.8 right now. There are a few tradeoffs for having this higher power factor, but is a very promising sign.
Now moving on into looking a little more detail at the first category of products we tested. We're looking at the outdoor roadway luminaires, the arm-mounted luminaires, right now, and on the — this chart here is plotting total efficacy, so we're just looking at the total efficacy of each of the products that we tested in this round here on the right. And in previous rounds, we had the average of solid-state lighting products here in the middle, and on the left we have three benchmark examples that we tested in Round 7. Now that was just a couple years ago, but these are representative of products you can find on the market today.
There's one high pressure sodium cobra head here on the very far left, and then two different types of induction lamps that can be typically seen in roadway applications today. What we see here is that the average efficacy of these benchmarks is about the same as the average efficacy that we observed in Round 11 of testing. Now in fact, these two products here on the left of the SSL group of products, 09-62 and 09-113, are slightly older products. They were purchased in 2009, so their lower efficacy may not be representative of what we're seeing on the market today. If you look more at these other four products, we see that their average efficacy is higher than the average efficacy of the benchmarks that we tested. We'll also note that this product here, 09-62, is a replacement lamp. It is not a luminaire. An entire luminaire is meant to be mounted in a cobra head, and it's sold as a replacement for HPS luminaire, which if it was actually used would only be able to replace a 35 watt high pressure sodium product. So actually the product that it says it could replace doesn't actually exist typically on the market.
So efficacy isn't everything. These 3-D visualizations help us to see that. They're all built using — imagining the product mounted at a 27 foot mounting height and looking down on a typical surface with a typical mounting, and we can see that all these different 3-D plots are entirely different. Some of these products are putting out very wide beams. We look at our high pressure sodium benchmark. It has a very wide beam with a certain amount of throw. It has a little bit of a conical shape in the middle, but it has a fairly wide distribution of light along this path. We see one of the solid-state lighting products over here on the top right corner which has very similar beam patterns, pretty broad, a little bit peaky in the middle, but very similar to what we see in the high pressure sodium here. Now our two induction lamp benchmarks give us very conical beam patterns. This one in particular, the one in the middle, has a very much higher luminous level in the middle of that cone than the other one, so it's much more conical and narrow beam than what we see in the high pressure sodium benchmark. And across the solid-state lighting products that we tested, we see a huge variety. We see the two products with lower light output here that have very low levels of illuminance if they're mounted at a 27 foot mounting height. We see our highest wattage product down here in the middle, which was 151 watts, with a conical beam, pretty wide, because it does cover — is 150 watts so it does provide quite a bit of overall light output, a little bit more overall lumens than our benchmark product. And then as I said these SSL products here on the right that have fairly uniform light distribution, so efficacy is not everything. We need to look at these other details.
Looking a little bit more detail at these products we see that they're not side-by-side comparisons for one another. They don't all have the same wattage level, and they don't have necessarily the same type. They're not designed for exactly the same application, although in some cases they may claim to be direct one-for-one replacements, you have to look at the details about what they're intended to replace and how they actually achieve their photometric distribution.
So what we see here is our high pressure sodium product is actually 100 watt high pressure sodium product, and it draws 117 watts total when you look at the system. The two induction lamps, along with three of the solid-state lighting products, are about 70 watts. And then we have one SSL product that has much higher wattage, and a couple that have much lower wattage. So we're looking at very different products here that may be intended for different purposes.
And we see that looking at the overall lumen output and the average illuminance that they are quite different from each other. We see our basic high pressure sodium benchmark at about 6,500 lumens, and we have that in our highest wattage solid-state lighting product down here. We have 7,000 lumens. The other products don't achieve quite as much light output, but again, these products might have good light distribution and might, in some applications, be good replacements for our HPS products.
Now looking at the average foot candle level of these products from an application that was calculated based on 24 foot wide, two-lane street with a 27 foot height, arm-mounted luminaires sat back six feet and spaced 170 feet on center, we get basically average foot candle values here that are comparable for one of the solid-state lighting products with the SSL product. We see that the two induction products do not achieve as much average illuminance in this application as the HPS product. And most of these solid-state lighting products do not achieve that level with this particular application. Now in other applications with other spacings or other mounting heights, the solid-state lighting products may do a better job.
Average illuminance isn't everything. We also want to look at the uniformity of that illuminance, whether or not these products are doing a good job of getting the light in a uniform manner across the surface that is intended. And we see that the HPS benchmark has a uniformity ratio of 6 to 1, and we see that achieved in a couple of the solid-state lighting products. Now that doesn't mean that they're going to be direct replacements for this product in every application, but they do have similar uniformity in this particular example application we're looking at. We see that a couple of these solid-state lighting products, along with the induction lamps, have a much more conical beam pattern, and so they don't achieve as much uniform light distribution as the benchmark product.
So just in summary, looking at these roadway luminaires, it's very important to remember to look at the overall luminaire performance. That means not just looking at the lamp performance. That high pressure sodium benchmark that we were looking at there had a lamp rating of 9,500 lumens per 100 watt lamp which would have been an efficacy of 95 lm/W. In fact, when it's installed in the arm-mounted cobra head and you test it as a system, it actually achieves 6,500 lumens for about 117 watts which is only about 55 lm/W. So you're going from a lamp-rated lumens of 95 lumens per watt to a system-rated luminaire performance of 55 lm/W. So it's very important to look at the overall luminaire performance, how these products perform when you test them as an overall luminaire, and you want to look at the initial performance and also consider the performance projected over the life of the product.
Now most importantly with roadway applications, it's important with a lot of outdoor applications, and in some cases with indoor as well, it's important to analyze these products in light of the type of lighting system that they may be used in. You need to look at the mounting height, spacing, position, street width, and other application requirements. You need to look at the overall input wattage, overall light output, average illuminance and uniformity, light distribution, looking at forward and lateral light, looking at BUG ratings, but being careful to look at them. We did discover, and there's more details about this in our summary report, that the BUG ratings are not necessarily very good indicators because of the differences in light levels, and the BUG ratings are not adjusted basically for the light level of a product. So comparing side by side products by looking at the BUG ratings may not be appropriate in a lot of cases.
Glare, which is in some way shown by the BUG rating, is something that's not simple to predict, and we highly recommend that a product be looked at in situ or in a similar environment similar to in situ application to determine whether the glare is appropriate for a given application. Uplight and backlight may be issues for these applications and need to be considered. Color is — may be less sensitive for roadway luminaires, but we did find that four out of six of the products that were tested had at least one color quantity that was not within specs or would not be what would be expected by what the manufacturer had claimed, so that's something that could be an issue in some applications.
We were happy to see that five out of six of these products met their manufacturer's performance claims, at least for the numerical values. Many of them, however, have equivalency claims stated in the product literature saying that the product is equivalent to high pressure sodium or equivalent to some other benchmark product. And it's very important to be aware that those equivalency claims can only be valid for very specific application scenarios, specific installation scenarios, so that even if they might be meeting the claims that the manufacturer says numerically, they can't meet necessarily every equivalency with a benchmark product. It's going to depend on the application, so these products need to be looked at very carefully in an application specific way before making any decisions about them.
Now moving on to look at the other outdoor application that we tested. We looked at post-top luminaires. This is the first time that we looked at post-tops to this extent, and we got two metal halide benchmark luminaires and two SSL products that we tested. They all look similar. You can see these images here. They look very similar. They are about the same size. They're going to look the same as far as what the lamp looks like when it's off on top of a luminaire, but they're not going to look the same when they're on. These benchmark products that we have here draw quite a bit more wattage. We have 178 luminaire watts for this top one, 192 for this bottom one here, and these two solid-state light products, which are also sold as post-top luminaires for similar applications, only draw 48 watts for this top one and 25 watts for the bottom, and they do not achieve such high levels of efficacy that they can compensate for that lower level with their light output. So this top one is putting out about a third of the light output of this top middle halide product over here. This lower product here in the lower right, is only putting out 854 lumens. That's about a ninth of the total light output of this product here on the lower left, so it is quite a different product, and although it might be very good for some applications, it's not necessarily going to be good for all applications, and it's definitely not a one-for-one replacement for this benchmark product here on the left.
Now another thing to note before we look at a 3-D view of these products is that this top benchmark product here does not have an opaque top. The bottom one does have an opaque top. Our two solid-state lighting products both have opaque tops, and one of the selling points for this type of application for solid-state lighting products is often that they achieve their lighting, the ground level lighting, with a lot less uplight than some of the benchmark products, especially the benchmark products that have clear tops. Now what was interesting here was that we could not obtain photometric data for the opaque top benchmark from the manufacturer. We actually couldn't find any. Our lighting experts went to try to find some photometric data from manufacturers covering opaque top post-top luminaires and had a very hard time finding any information whatsoever. So the manufacturers of the benchmark products publish photometric data for the clear top versions of their products but not necessarily for the opaque top versions. What's interesting there is that if you look at the non-opaque top version, the photometric data for it, you imagine that the product has a much higher efficacy, much higher amount of useful light than it actually does in the opaque top version. So again, when we're comparing SSL products to benchmark products, it's very important to try to compare apples to apples and look at the luminaire photometry for the actual products that you're interested in.
Now looking at the light output for these products again with our 3-D view, which allows us to just very intuitively understand where the light is going and how much light these products are producing, we see very clearly that this lower this SSL product here in the lower right achieves much less light output than either of the benchmarks, and actually a lot less than the other SSL products here as well. These have a fairly even distribution. They're doing a pretty good job of getting a — not a conical distribution that we were seeing in the roadway lights. This product here, this SSL product here in the upper right, does a pretty good job of getting uniform distribution, and with our graphic here in the upper right corner you can see what the uplight looks like. And we see again that this metal halide benchmark that has a clear top puts an awful lot of light in the upward direction where this solid-state lighting product puts almost no light, no percentage of its light, in the upper direction. That's one benchmark, the opaque top benchmark product puts a little bit of uplight but not very much compared to its overall light output. But this other solid-state lighting product here, which is the lower wattage product, actually ends up emitting quite a bit of light as a percentage of its total light in the uplight direction. So really, these products are very, very different. They're not similar, and although, for example, this product in the lower right does claim to be a direct replacement for benchmark - - for metal halide product, it does not achieve the light output level that you would expect, and it doesn't even have as high an efficacy as the opaque top metal halide benchmark.
So just to summarize on the post-top luminaires, similar to what we said on the roadway arm-mount luminaires, it's important to consider the overall luminaire performance, look at the initial performance and the projected performance over the life, and consider the lighting system as a whole for the application requirements that you're looking at. Look at the input — total overall input watts, overall light output, average illuminance and uniformity, light distribution, glare, and uplight.
Now one out of four of these products had complete accurate manufacturer claims. That means that even on the benchmark products, for that opaque top benchmark, we could not get complete accurate manufacturer claims for how it performs. One of the solid-state lighting products had some performance data, but they were overstated by about 25% when we tested it ourselves. And the other solid-state lighting product, the one that was the lower performer, provided no performance data, and it does perform poorly. And it's being sold as a replacement for typical, traditional post-top luminaires. That product provides no performance data to the buyers and actually performs very, very poorly. So it's very, very important that photometric data be available and that it represent the exact luminaire that you're considering to evaluate and considering purchasing.
Now moving on to look at linear replacement troffers. I'm saying linear replacement lamps in troffers actually. We tested a number of LED linear replacement lamps with this round of testing. We see here, if you look at the lamps here on the right, we have quite a variety of LED devices in these lamps and the ways that the arrays are formed and the way that the optics may be built, the tube may be diffuse or not over those LED devices. There's quite a lot of difference in the products out on the market today, and we test them, as I mentioned, as bare lamps and then we also put them in a parabolic louvered troffer, this one here in the upper right, and test them in that troffer. We have benchmark data for typical fluorescent lamps in this same benchmark troffer, so we can compare them side-by-side with a direct comparison against the LED lamps in that same troffer.
In this round of testing, we also added two high performance fluorescent troffers, one with two lamps. Actually we had tested this one in an earlier round, and we retested it with a different ballast so that we have results for this product using two different ballasts, two different ballast factors, because fluorescent products do perform quite differently depending on the ballast being used. And we also tested a single lamp high performance troffer, this one here on the bottom, with a fluorescent lamp. This one we tested with a high ballast factor ballast, and we are currently testing it with a lower ballast factor ballast as well to add that data to our information as well. But for Round 11, we tested it with a high ballast factor ballast so that we could get slightly higher light output to compare it more readily to some of the SSL products.
Now first level of looking at the information, again, we're looking at the luminaire information here, not the bare lamp information, so we're looking at how these perform in a troffer. The top chart here is looking at efficacy. The bottom chart's looking at light output. Our blue bars are the solid-state lighting products that we tested in this round, and the two green bars are the two high performance fluorescent troffers. When we look at this first chart here with efficacy and it looks very clear here that at least three of these products are definitely achieving the same levels of efficacy that we're seeing with fluorescent products with these high performance fluorescent products in their troffers. So the SSL products now in the troffer are capable of achieving a similar level of efficacy of fluorescent.
Now looking at the light output levels, we see that they're not quite doing quite as well. They are achieving — two of these products are achieving as much light output, so these are products where we have two SSL lamps in a troffer. They're achieving about the same level of light output as one fluorescent lamp in the high performance troffer. These other two SSL products here on the left are not achieving as much light output as that one single lamp fluorescent troffer, so two SSL lamps here do not make as much light as one fluorescent lamp in a high performance troffer. For these other two products, they are achieving that amount of light, but they are not achieving as much light output as the two fluorescent lamps in the two lamp troffer. Now again, these are being tested here. The SSL lamps in this case were being tested in a parabolic troffer. We haven't yet tested them in a high performance troffer to see how they do in those particular troffers, but in any case, in this next chart, we'll see we have compared them as well against the fluorescents in that same parabolic troffer.
So here we see them compared against several lamps. First of all, these kind of dimmer, smaller, diamond shape spots here are solid-state lighting linear lamps that were tested in earlier rounds of testing. These four big blue triangles are the luminaire output levels for the solid-state lighting four foot lamps that were tested in a parabolic troffer in Round 11, and we see very clearly from these dimmer points down here up to these blue points, that there's quite a bit of progress that's been made since Rounds 5 and Round 7 when we started testing these products. First tested some in Round 5. No, we tested more in Round 9 actually, and now we've tested these four in Round 11, and we see clear progress with increasing efficacy, increasing light output and using less power overall. However, when we compare them to the fluorescent benchmarks, we see these — we have various different benchmarks here. One benchmark here, which is way up here in the corner, is a fluorescent T12 troffer with two T12 lamps in it with a prismatic lens, and we see that all of our solid-state lighting products are achieving greater efficacy than that fluorescent T12 troffer but only about half of the light output of a T12 troffer equipped with two lamps.
Now but when we compare them to the T8 troffer with — in the same parabolic louvered troffer as we tested these solid-state lighting lamps, we see that they are achieving higher efficacy now than that typical parabolic louvered troffer equipped with two fluorescent lamps. So three of these products are achieving better efficacy than the fluorescent lamps in that same troffer, but they are not achieving as much light output. They are achieving anywhere from about ½ to ¾ of the light output of the fluorescent equipped troffer. And then when we compare them to the — that's just circling them there so you can see exactly which one I'm talking about. These ones were installed in the same troffer as this fluorescent one here. When we compare them to the high performance benchmark troffers, we see that they're achieving — two of them are achieving as much light output as the single lamp troffer but not as much as the two lamp troffer.
Now we just said that they're doing very, very well with respect to efficacy. How are they actually doing with respect to illuminance and getting that light where it needs to go? This chart shows very clearly that they are not providing the same illuminance patterns as the fluorescent baseline product, so the black line here is the — let me see if I can make this show up. I was just told that you're not seeing anything that I point to, so I'll grab a little red dot here so you can see this, the black line. Hopefully that's visible to you, what I'm showing you, is the fluorescent baseline, so the parabolic louvered troffer using two fluorescent T8 lamps and the red line in each case — oh, okay. Thank you. So just show you that with the little red dot there. That is our fluorescent baseline, and the red lines here in each case represent the illuminance of our SSL product. If you very clearly that all but one of these products in the center beam achieves a similar level of illuminance to the fluorescent baseline products. It does not give you a nice wide bat wing distribution that we see in the fluorescent baseline, so if you have — if you replace your fluorescent product with this solid-state lighting product, even this best achieving solid-state lighting product that we've tested here, you will have stripes on the floor of your room with brighter light in the middle and no light in between your fixtures. With these others products, this 10-16 here on the left, you see that you have much lower light levels, and you definitely don't have the coverage off to the side. So the SSL T8 lamps in the parabolic louvered troffer definitely do not provide light distribution that has the breadth or intensity of fluorescent T8 lamps tested in the same troffer. Also looking at the spacing criteria, we see that more fixtures will be needed for an equivalently installed system using the solid-state lighting products as compared to using fluorescent products.
Another point that we looked at here was color quality. These are all products that are used in indoor applications where color quality can be important. The color — correlated color temperature may be important because you don't want to have different colors of light on the ceiling of a room, and the CRI may be important. So these are applications where you probably want to be able to match color temperature from one installation to the next, from one lamp — from one luminaire to the next, and over time, if you're changing out lamps to replace them with new ones, you want to be able to match the color temperature of other lamps. Well what we found was that none of the solid-state lighting products that were tested had achieved both correlated color temperature and Duv, which is basically your distance from the blackbody locus on your chromaticity diagram, none of them achieved both of them within tolerance, so some of them only have your color temperature. Product 10-16 here has a CCT value which is out of tolerance compared to the target CCT value that it's being sold as. It's sold as a 5,000K product and it's at almost 5,400K. You have another one down here that is sold as a 4,000 to 5 — 4,500K product, product 10-19 here, which actually performed fairly well in output and efficacy, but it's actually at 5,000K, much colder white than what it's being sold as, and it has a Duv of 0.008, which means that the light is probably going to be a bit yellowish or bluish. This other product here, 10-36, that we tested, has even a worse Duv. It's CCT is within range, but the Duv is very high which would mean it's going to have some sort of tinge, colored tinge, to it and not quite look white.
Now, the two — interestingly, the two fluorescent products that we tested did not meet the fluorescent tolerances for white light. Now we have to note here though that those tolerances for fluorescent lights are much tighter. That's a 4-step MacAdam Ellipse, and that's much tighter color tolerances than what are currently permitted for solid-state lighting products. So if they were subjected to the same color tolerances as the SSL products, they would definitely achieve those color tolerances, but they are supposed to be held to tighter tolerances and neither of them met the chromaticity tolerances that they should have. One of them had the — did achieve the CCT value, but its chromaticity coordinates were slightly outside the MacAdam Ellipse that it should be in. So it's very interesting that even the fluorescent products here were not achieving the color quality that we expected to see, although they are doing better than the solid-state lighting products.
So looking again overall at the linear replacement lamps used in luminaires, we see definitely that the efficacy is increasing in the SSL products, but it's very important to look at the overall luminaire efficacy and not the lamp efficacy. We see that the light output for SSL does not achieve the same level as fluorescent for the same number of lamps in the same troffer. We see that the light distribution does not at all achieve the levels and the breadth of distribution of fluorescent lamps in a troffer. Again, we will be probably testing some of these in some other troffers to see if they do any better, but generally speaking, the louvered troffer gives us a very good general benchmark for these products in a troffer. The color quality is not within tolerances, and it is definitely poorer than the fluorescent lamps. Reliability for these products may be problematic. We have observed a number of test samples that we've received that have failed during the testing or have been received with damaged pins so that makes testing very difficult. We have had a lot of trouble getting follow-up samples when we need to get replacements for products that have failed during testing or have been damaged during shipping, and companies do not — not all of the companies are following through. They don't necessarily have product lines that are ongoing, so that can be a problem. We also have observed that each of the solid-state lighting products comes with different wiring instructions and different retrofitting instructions. Some of them have one type of wiring and some have another. All of the four that were tested in this round, I believe, required removal of the fluorescent ballast and rewiring of the troffer, so that is quite a challenge and may present some difficulties going into long-term for maintenance and safety. I know that some — we are working closely with some of the standards groups and with NEMA to try to get more information to manufacturers and to installers regarding how to do this retrofitting and what may be appropriate or not appropriate in designing retrofit products for this type of application where they need to basically bypass the ballast to use the products.
Most of these products had fairly good numerical ratings for the products, their but manufacturers have equivalency claims that are misleading. They basically claim that these products are one-for-one replacements for T8 fluorescent lamps and they are not direct replacements for T8 fluorescent lamps at this time. They do not provide the light distribution that T8 fluorescent lamps provide, and they raise challenges for retrofitting and for color matching of the products.
So going on to look at smaller replacement lamps now. We tested a couple of MR16 replacement lamps in this round of testing. The two here, 10-02 and 10-30, were both tested, also included 09-49 here, because it is our best MR16 tested to date with the highest efficacy and the best light output before these two products that were tested in this round of testing. Now we see that we also tested some 35 watt halogen benchmarks. This is the first time that we've tested 35 watt halogen benchmarks in the past. We've only been comparing the SSL MR16 products to 20 watt halogen performance. This square here gives us values, some that we have tested and some that are from ratings, for 20 watt halogen MR16 products, so that we can see that there's a range of performance in halogen products as it is. And these two products here, 10-30 and 09-49, both surpassed the average efficacy of 20 watt halogen MR16 products. 10-02 almost meets the lower end of the efficacy levels and light output — I mean I said efficacy. I should have said light output. They're exceeding the light output, and they're definitely exceeding the efficacy. 10-02 almost achieves the light output of a 20 watt halogen but not quite. It does definitely achieve better efficacy than the 20 watt halogen. Now none of these products, none of them are achieving the light output levels of a 35 watt halogen benchmark, but we have that there now as a stretch value for the next generation maybe of solid-state lighting products.
Now in MR16 products, they are directional, so the beam angle and the ability to generate the same center beam intensity is also important. Efficacy is not the most — the only factor to look at. These curves here show us center beam intensity for MR16 lamps. The red curve shows basically the center beam intensity for 20 watt halogen, and this dotted orange beam shows it for 35 watt halogen. In fact, the 35 watt halogen benchmarks that we tested did better — they achieved better intensity than this benchmark curve that we have from rated values. So in fact, the solid-state lighting products are going to have to do an even better performance job to get up to that performance level of the 35 watt MR16 benchmarks. But if we're comparing them to 20 watt, we see that they are doing a good job. One of these products has a very wide beam, the 10-02 product, and one is a much narrower beam, but they are achieving — basically achieving similar beam intensity similar to halogen. Now that product 10-02 claimed to be an equivalent of a 35 or a 50 watt halogen. It was not even barely an equivalent of a 20 watt halogen, so we have to be careful looking at equivalency claims. One other sample there — one of the samples that we tested definitely is a replacement for 20 watt halogen, although there are a few problems that we note here. Attention: Most of these SSL MR16 lamps exceed the maximum length for MR16 products, so they are a little bit out of tolerance with respect to their geometries, so they may not fit in every application. Also, the better performing lamp here, 10-30, which did a very good job performance wise, fails to mention that it's not for use in totally enclosed fixtures. It has a small sort of fan that makes a little bit of noise if you listen to it very closely, but not very much, but it has an air circulation that is needed in order to maintain the temperature, the thermal control, so that it doesn't overheat. So there are other caveats to using these products. They do have good color quality and that one product, sample 10-30, that we measured would actually meet Energy Star photometric criteria for integral LED lamps. Not saying that it's applied for that, but looking at the criteria, it would at least meet the photometric criteria for those lamps.
Now moving on to PAR lamps, we see that we've tested a number of PAR30 and PAR38 lamps here. They're the ones in the middle of this bar chart. Looking at light output in orange and total efficacy in green here, we have the average performance of PAR lamps from earlier rounds of testing here on the left, and we have a couple of benchmark values here on the right. One infrared halogen lamp here, 46 watt HIR PAR30 and one ceramic metal halide PAR38 here, which was a new thing for us to test. Now… Sorry, this went back. Went forward a slide. I meant to put these little arrows up here. We see that we're definitely increasing in both light output and efficacy from earlier rounds of PAR testing, but we're also increasing in wattage value. We see the earlier rounds of testing on average had 8 or 10 watt lamps, and now we're seeing lamps that are managing — doing thermal management for 11 to 18 watts in solid-state lighting products, so they are using higher level wattage levels, and they are managing to put out quite a bit more light output.
When we compare them to the halogen benchmark, we see that all four of the products exceed the 46 watt halogen benchmark in light output. That's this top line here, so all four of our PAR lamps exceed that level of light output, and they definitely exceed the efficacy by about three times or four times the efficacy. But when we compare them to the ceramic metal halide, we see that they're not achieving the light output level or the efficacy of this 25 watt metal halide benchmark. Now there are some caveats to using ceramic metal halides with respect to restart and dimming, but it still is a good benchmark for solid-state lighting products to aim toward. Again, these products, like the MR16s, are directional, and we want to make sure that they have center beam intensity which is on par for what we're expecting. We see that all of these products, all four of these blue squares, are above the 50 watt halogen PAR30 benchmark, and three of them are very close to the 75 watt halogen PAR38 benchmark line that we have here. So they're doing very good in achieving those levels.
They are almost all warm-white color temperature products. There were a couple of problems with color temperature on one of the products which appeared to be a manufacturing line problem or product labeling problem, but that is something to be aware of that's important to look at the ratings on the products and make sure that they are meeting your needs there. Most of these products did not meet some of their claims, in particular for equivalency, they tend to say that they are replacements for 75 watt halogens when they maybe are only replacements for 50 watt halogen, and a few of them are not standard format products.
Now moving on to our small omni-directional lamps, we tested a few lamps that are basically A-lamp replacements in this round of testing and one candelabra lamp. We can see that this one here in the middle claims to be a 40 watt incandescent equivalent. This line up here is the average value of light output for a 40 watt incandescent, so this product is getting close to being a 40 watt incandescent equivalent, but it's not quite there. It does achieve quite a bit better efficacy so maybe three times or four times the efficacy of the similar incandescent lamp. This other product here — this is going backwards. Sorry. This product here claims to be a 60 watt incandescent equivalent, and the 60 watt incandescent line is up here where this red dotted line is up here, so we see that even though it claims to be a 60 watt equivalent, it definitely is not achieving the light output level of a 60 watt lamp. And it is doing a very good job efficacy wise, but it's more likely to be a 40 watt replacement than a 60 watt replacement.
Now the candelabra lamp here is actually achieving the average light output level for a 15 watt candelabra, so it is a replacement for a 15 watt candelabra. They are definitely approving in output and efficacy. We do see more and more products now that are exceeding the light output levels for 40 watt incandescent and achieving very good efficacies. There are starting to be some products on the market that are getting close to 60 watt equivalency. Hopefully in our next few rounds of testing we'll be able to test some of those, and as we saw, that one candelabra product is now actually able to meet the levels of a 15 watt incandescent light bulb. And they have good warm-white color temperatures and CRI in a lot of these products. So overall, they're doing well. Again, we have this question of products that claim to be equivalents that might not have the same format. That one lamp was a much larger diameter bulb than an A19 light bulb, so that can be a problem in some applications. It's something to be weary of.
Just before we conclude, I just want to go into a couple of other points. One was that we do observe photometric flicker in a lot of the SSL products but not all. Here's the example of the SSL MR16s that were studied in this round here. We see — here on the left, our halogen benchmark. You see a small amount of photometric flicker up here. Just a little bit of waveform that would not necessarily be visible to the human eye, and our — one of our SSL products, 10-30, had very similar level of photometric flicker up here, very similar amount of modulation. But then our other MR16 product, 10-02, that was tested, has light modulation that drops all the way down to zero with every waveform, with every period of this waveform. Now that could be disturbing for some people and it could have human physiological effects for some users, so that is being tested and looked at by a number of standards committees at this time, and we are just concluding a large survey of CALiPER products to put this information in the public domain so that people can go further with understanding photometric flicker in solid-state lighting products and in other products.
Another key point to conclude with was this question of manufacturer claims. We saw that a majority of SSL products in Round 11 have accurate manufacturer specs at least for the numerical values. A lot of them have Lighting Facts labels. That's this label here, if you haven't seen it, and that's what you need to look for on products. It verifies that the product has been tested with LM-79 testing and provides standard format so that you can understand what the basic performance is. Products that have that Lighting Facts label tend to have much more accurate claims, although it's not necessarily a guarantee of them having accurate claims, what we found was that most of the products with — that carried the label have accurate claims on those labels. There are still many products that provide no ratings or no photometric information or have very false ratings, and the biggest problem that we're seeing is that equivalency statements are almost always false or misleading, so they should be disregarded. And you need to learn for yourself, each user needs to learn what equivalency is in lumens for a given lamp wattage and then look at the light output on the Lighting Facts label and see where that falls in the equivalency chart. For now, that's about the only way to be sure that you know how the equivalent performance is doing.
So bottom line is that we're seeing overall increase in light output, efficacy, better light distribution, very good power factor. We're seeing that the suitability of solid-state lighting products depends on the application. We're finding comparable products in many light applications now with traditional products, but there are also poor performing products in almost every light application. So careful comparisons are really needed based on accurate performance data, based on LM-79 test results. For most applications and products, it's also really good to do an in situ check to see that there is not that glare or other problems that you wouldn't be aware of from the LM-79 test report. Quality issues, as I said, glare, flicker, color tolerances, and physical formats of the products can be issues that need to be considered and looked at carefully, and reliability is always a factor. We have some additional results from CALiPER long-term testing that show similar results to what we saw in the past. Some of the products, some of the solid-state lighting products are doing very, very well after 6,000 hours of testing, but a lot of them are not doing very well and are not meeting the lumen (inaudible) that they expect. So it's very important to keep that in mind when looking at these products and ask for a maximum of information from the manufacturers.
So as I mentioned in the beginning, almost all of the information that we've provided is available online, and the summary report for this round of testing, or for previous rounds of testing, there are - - there's actually a new detailed listing format out there for this round of testing where we already had detailed reports for previous lamps, and we're putting them out there now in a more dynamic format. And you can also download the IES files for those products now, which will be interesting to see if folks can make use of those. We also have benchmark reports online, proceedings from meetings that we have, and we also do exploratory studies. In some cases, the — we have reports on those exploratory studies that can be requested as needed. We also work with a lot of other groups, in particular, mentioning at this point NEMA activities, because they're working on a number of standards or whitepapers that have to do with things that we're looking at here such as the retrofitting of T8 lamps, and you can look toward them for information about that in the future.
So that basically concludes my presentation portion for today. I'm going to put up one more slide here which is just our reminder about our no commercial use policy. We are not in the market of trying to bring advantage or disadvantage to any manufacturer, so this information is being provided for educational purposes, and we encourage you to use it for educational purposes. But it should not be reused or republished in any way for advertising purposes or to promote a particular company's product or to — actually to talk negatively about a company's product or service, not directly about the names of the products or the manufacturers.
So I've just put up the link now where you can download the presentation material if you would like, and apparently, it will also be available online in the future with responses to some of our questions.
And I'm going to turn it over to Ruth now, Ruth Taylor, who helps me to run the CALiPER Program, and she has been fielding your questions, and she will start us off with the initial question.
Ruth Taylor: Okay. Thanks, Mia. We have had several questions come in. I'm going to start with a couple of high level things, give Mia a chance to catch her breath, and then we'll go into a few others. There's no way we can cover all of these, so we'll try to see how we can answer up on some of these that we don't have time to get to. I did have a few people asking about where to find out more detailed information of some specific manufacturer names, perhaps of products that you saw Mia summarize. Out on the CALiPER site, you can go to these detailed reports. Do a search, a category search. You can either search by the name of the manufacturer, name of the products. You can look at all of Round 11 like Mia mentioned. You can get —all of those are dynamic. When you click on that specific link to that report, the manufacturer name will be there and all the details of the entire report, so we encourage you to do that when you're wanting to find more detailed information. Hopefully that will be helpful.
Ruth Taylor: I did have another person ask whether LM-79 is going to make CALiPER obsolete, which I thought was an interesting high-level question, where we're sort of going and the purpose of the program. And I guess one thing I would mention about that is that one thing that's unique about CALiPER is that we are going out and finding products as a typical consumer would find them, so if you're looking at LM-79 reports published by a manufacturer, you can analyze those, but that could be a special product sent in by the manufacturer or we might call it a hero product. So one of the advantages of us going online or through a distributor, finding a product just exactly as a normal consumer would find it, just gives an extra level of QA there to see where the market really is. So we still do have that vital role, but as standards are progressing and things are getting better, we're not going to be here forever. It will be easier and easier for you to find out better information about the products and look at Lighting Facts labels and all those kinds of things.
Mia, we did have a couple of clarification questions, if maybe if we could get to those first. If you could flip back to slides 9 through 11. We had a couple of questions about your red type on those and what those mean. So as people are looking back through these slides and perusing, let's — can you give some clarification on some of those slides?
Mia Paget: Yes, Ruth. That sounds great. I'm back at Round 9 here, slide 9. That should be showing right now. The red is mostly just to highlight the differences here in the sense that we are looking at products that range from 38 watts. So we have 38 watt here up to a highest wattage of 150 watts, and our high pressure sodium benchmark is kind of in the middle there at 117 watts luminaire value. The 44 watt one here is just to show another lower extreme. They're in red just to highlight that while the other products here are inductions and most of these SSL products are around 70 watts, we are seeing quite a wide range here. That's why those are in red there just to highlight that because it's very difficult to purchase two products that are exactly the same and compare them. Even if the manufacturer says that they're a direct replacement for something, we will find often that they are quite different, and in some cases, we do try to purchase very similar products based on — expecting to get the same performance. In other cases, we purchase products based on equivalency claims and find that they are, in fact, very different. And in other cases, sometimes we purchase products even if they don't have exactly the same performance characteristics of another one, but we want to look at an innovative design, or we've heard a lot about them in some competition or through some trade journal, and so we may purchase them even if they're not exactly similar to our benchmark product. That's why we see those differences there.
In this case here, the red products again were to show our benchmark product here on the — let me see if I can circle it here. Here, this one is in red just to highlight the benchmark high pressure sodium, and then we're highlighting one extreme here, which is our lowest performing product, which is really not doing well, and then we're highlighting in this case two products that have similar illuminance, similar average illuminance, to our benchmark product. So these are the two products of all the SSL products and the induction lamps, these two in red are the two that show the most similar illuminance levels to our high pressure sodium benchmark in this particular application that we're talking about here.
And on slide 11 here, it's the same thing. We're looking at —we've put in red here our benchmark product to look at, and we're under the SSL products that looks like it could be a very similar product. There are some other ones here that have similar distributions as well, but then there's two here that have very, very high average to minimum ratios, so they might not be appropriate for this application that we're talking about here. So they're just highlighted so that people will note that there really are big differences here. Does that answer the question, Ruth?
Ruth Taylor: Yes. Okay, Mia, I'm going to go next to probably the most pervasive question we received. So quite a few people asked this in one way or another, and I think it'd be helpful to explain, especially for our linear replacement study, how we selected those benchmark products and how we selected the SSLs too, but there were questions about why didn't you look at T5s? Why did you select these? So I think there's quite a bit of confusion or wanting to kind of understand that process and where we might be going in future rounds on that.
Mia Paget: Right. That's a very good question. Selecting benchmark products is always a challenge because there's not any, in most applications, there's not any single one benchmark that represents everything. For the linear replacement lamps, in our initial benchmark testing that we did a couple years ago, we looked at a prismatic lens troffer, and that was using T12s because that was kind of the lower performing at the time, making sure that we could at least — solid-state lighting products were at least achieving that lower level of the T12 efficacy in the prismatic lens product. And then we were looking at the parabolic louvered troffer from what some lighting designers suggested was a common type of parabolic louvered troffer, it was 12 celled troffer, from a very well known manufacturer. So a very typical product using T8 lamps. Now we're using T8 lamps because of their market share and because that is where we're seeing the greatest marketing effort and energy coming from the SSL manufacturers. So we're targeting that type of product because of the amount of marketing noise that goes on in that domain and because we do see it as a potential area for dissatisfaction of customers, very, very wide dissatisfaction if they purchased these products thinking that they are one-for-one equivalents. So in that case, we chose what were considered by lighting designers to be very typical, kind of easier to achieve benchmarks, and then as the solid-state lighting market is progressing, we're looking at benchmark products that are actually higher performance and often product that may be more expensive because the SSL products typically tend to be more expensive upfront costs. So they could become competing against these products that tend to be higher end products as well, so that's why we're looking at high performance troffers at this time.
We are looking at the one lamp troffer and the two lamp troffer because in some cases one of the arguments given by SSL manufacturers are that their products don't necessarily put out as much light output, but that's because it's not needed or that's because they are getting the light to the right places. So we are choosing benchmarks that allow us to test those hypotheses, test those claims, and we're making choices of the benchmarks using lighting designers that communicate with us either through our CALiPER Guidance Committee or folks on our team or folks that are involved with some of the other DOE commercialization support efforts. And in some cases, we make — as I said here, we've tested the two lamp high performance troffer with two different ballast levels now, and we are testing a single lamp troffer with another ballast to get that data point if you will in the picture.
It is very difficult to choose benchmark products, and again, our basic principle when we're trying to choose is often to say what is the solid-state lighting manufacturer claiming as an equivalency? And if they're claiming equivalency, then we typically want to have a benchmark that represents what they're claiming, but it is not an easy thing to single out. There are a lot of different products on the market, and it's very easy for manufacturers to find wiggle room by claiming something slightly different in the next six months. And we're always open to suggestions, either directly or through our Guidance Committee members. We're very open to suggestions about what types of benchmarking and equivalency is needed. It's not the focus of the CALiPER testing, and so we have to do it very judiciously, but it's been very helpful for us to do, and we've learned a lot through our benchmark testing.
Ruth Taylor: Yes, I think it does. And I will say we have heard you loud and clear, those of you that have put in your comments/questions about why don't you have a T5? Why don't you have a T5 benchmark there? So obviously, as things progress, we'll be looking at that more and more, where the product claims are and those kind of things to move that direction. Do you want to say anything else about that?
Mia Paget: I think that's a very good point. We are not — we haven't seen a large push or a large market segment talking about T5s and so we haven't focused on it, but we are seeing it more and more. And actually, well, just yesterday, we were talking about some other options for doing some benchmarking in 2x2s and in 2x4s, so that's a very good idea to go down that path and look at now.
Ruth Taylor: Okay. I had a question about if you could explain a little more about the Duv and clarify how that value can be negative, what it means when that's negative. Could you — you might need to go back to that slide and maybe clarify that a little more.
Mia Paget: Well, yes. I don't have actually a very good slide here that shows a chromaticity diagram with a blackbody locus and the chromaticity coordinates. Duv is based on the U prime UV, or U prime, V prime color space, so there's different chromaticity coordinates that define color, and color is defined by where those chromaticity points lie with respect to a blackbody locus, so basically where the white light is pure as white as light is and that describes a curve which we call our blackbody locus. Now for each color temperature there is a target chromaticity point, so chromaticity coordinate value for a given value that's close to that blackbody locus that kind of defines your central target point and then for a given nominal color rating you can deviate from that in a certain direction before it becomes visible to the human eye. Now Duv is basically telling us how far you are in a perpendicular — well, it's not necessarily perpendicular, but how far you are in a distance from the blackbody locus, or from a given point, and that distance can be positive or negative as far as going up above the line or negative down below the line. So it's a distance from that line and that's how it can be negative.
Now it's a tricky value to use. It's a tricky thing to look at because at different color temperatures, what is considered to be a Duv within tolerance or outside of tolerance is not the same, so you need to look at the ANSI standard that is associated with defining solid-state lighting color. And again, that's applicable primarily for indoor applications. It's not originally intended for our outdoor applications, although it's still useful for any form of light to look at that value to know how close to white that light is going to be. The ANSI standard, and we can put that online, we'll put the reference to it online when we answer the question, that gives you the exact value so you can look for a given color — correlated color temperatures, given CCT value, what is considered to be a Duv value which is within tolerance. So that usually ranges from about negative 0.003 to positive 006, but again, that varies depending on where you are in color temperature. So that can be quite complex, and there's not an easy straight forward way to present that at this time, which doesn't help the industry. And that's a good point, and there might be something to do there to help folks with respect to defining color quality in the sense that CCT is not the only value to look at, but the secondary value Duv is complex and not straight forward at this time. It's not intuitive for people who are not experts in looking at this data. It's unfortunate that there isn't an intuitive simple way to define color quality overall.
Ruth Taylor: Great. Thanks. I think we have time for probably just a couple of more questions. I had someone ask how we — whether we do anything to evaluate the consistency or performance of products from sample to sample from the same manufacturer. Do we do anything to try to assess that in CALiPER?
Mia Paget: No. CALiPER doesn't really have the funding or the capabilities to do very, very broad product assessments like that where we would need to study a lot of samples from a given manufacturer. We have done it a few times looking at consistency of production or purchasing six or ten products of a given type, but we really can't afford to focus on any given product. There's so many differences from manufacturer to manufacturer, product line to product line, and from month to month, year to year, and that the industry is moving very fast. The production lines are changing and evolving very fast, so spending time and money looking at that, while it would — might give us insight into a given product and a given product line at a given time, it would only be that single product at that single time and probably not be able to tell us anything about even another product from that same manufacturer. Sometimes they might be produced in another factory. They might be produced entirely different — using entirely different process or different set of equipment, so it's very — it's not information that would be valid across a spectrum of products. It's not going to give us a lot of insight that can be generalizable, and since as you saw from our no commercial use policy, we're not trying to target any given manufacturer, any given product. We don't want to go down the line of spending a lot of energy on topics that can't be generalized and used as general lessons. Now we do a little bit of that, and that's why we do mention reliability when we see issues because we do want people to be — we want manufacturers to be alert to that. We want buyers to be alert to potential for reliability problems, but we don't want what we do to be taken as an indication of future or generalized performance of a given manufacturer or a given product line because these things are changing very, very fast.
Ruth Taylor: Yeah, we did have a few people along those same lines did ask whether we're doing any kind of special testing or other testing for reliability.
Mia Paget: Yes, well we do our long-term testing, and again, it's more indicative where we pick out a couple dozen products and put them in for long-term testing so that we can see whether or not they are meeting the lumen maintenance and color maintenance expectations that we see in the industry. Again, those are not statistically broad sets of tests, but they are tests that allow us to get a general feel for how the industry is doing and where they're going and allow us to communicate and talk about the issues. That's why we're also doing the flicker study. We're looking at that type of issue again to communicate, to help the standards groups, and to help make progress in those areas. So we are looking at reliability in a couple of different ways. We're doing another type of long-term test that's started right now to be able to correlate between LED device data and performance in situ of the products, and that's going to feed into some more standards work as well. So we are doing work where we can with respect to reliability. It's a difficult subject in part because it's a moving target and in part because these products do tend to be very robust, so it takes a lot of time, and there's no simple way to accelerate the testing on them or no consistent way that would be applicable for all products.
Ruth Taylor: Okay. Let's see. I had another question about fan-driven integral lamps. I did quickly check with Heidi on this, and we did have a couple of A-lamps in Round 8 and one MR16 in Round 10, and so if you want to look at the details on those.
There was a question about whether those can get Energy Star certification. We couldn't find anything to preclude that, but you really need to go to the Energy Star website since the EPA is doing that now and see if they had any recent changes in that.
We also had a question about residential products, that there's been more interest in that and whether CALiPER is focusing more on that. Mia, you might want to add to this, but I would say that our focus on A-lamp replacements, and we do a lot of that every single round. Jim Brodrick is quite interested in how those products are progressing, so that's one of the main ways that we address those residential issues. I don't know if you want to add anything to that, Mia.
Mia Paget: Yes. Well we are looking at residential downlights, doing quite a series of downlights right now, and that'll be coming out in our next round of testing. So we don't necessarily focus primarily on commercial or residential, but looking at the residential sector, a lot of the work that we go into that is working with retailers, large retailers, and big box retailers, to try to help them to have specifications that will help to eliminate the poorer performing products. It's one of the reasons that we do the work that we do with our long-term testing and that we do all the work — we're doing — working with Lighting Facts labels so that these labels get out there, and so that the bigger box retailers require them so that we can be sure that the products out there are at least being tested adequately and being reported on adequately. So working on the labeling and the information for the general consumer is very important to the DOE, and the CALiPER Program provides a lot of support to those efforts, but those are separate efforts from the CALiPER Program. But we feed into them and try to provide them as much information as we can to help them move forward. So that's mostly — I would say a lot of that work is on the retail side because those are the stakeholder groups that can have the greatest impact there. We do work with the manufacturers as well, but a lot of it is with the retailers at this time.
Ruth Taylor: Okay. I think that's probably all the questions we can cover in this time slot. We want to be respectful of your time, so I'm going to hand this back over to Terry to wrap up.
Terry Shoemaker: Thank you, everyone, for participating in today's webcast brought to you by the U.S. Department of Energy. You may all now disconnect. Have a nice day.