Solar Inverter Connectivity Problem Reveals Weak Troubleshooting
I like to think I’m decent at troubleshooting, but a recent failure reminded me how important it can be to break recalcitrant problems into their constituent parts. No one will encounter this precise issue, so take this story as a general lesson.
In short, I interpreted a communications failure from our solar panel inverters as a network problem, but it turned out to be a power issue associated with a feature in an uninterruptible power supply (UPS). Embarrassingly, I didn’t figure this out until the solar repair tech came to take a look and walked me through the parts of the system.
The Background
In 2015, we installed 18.9 KWh of solar panels to provide nearly all the electric power our house uses for geothermal-based heating and cooling and one of our two cars, a 2015 Nissan Leaf. (“Nearly all” because we had to anticipate future usage when sizing the system, and we guessed slightly low. NYSEG usually charges us a few hundred dollars in late winter after we’ve used up our generation credit from the summer.)
Overall, the solar panels have worked well, feeding into three SMA Sunny Boy inverters (one for each array of panels) that distribute power to the house and send the excess to the grid for our neighbors. The inverters publish generation statistics to the Web-based Sunny Portal, which emails me the totals daily. I don’t care about the exact numbers, but those email messages alert me when something has gone wrong. We’ve had an inverter fail, the connection with the portal occasionally drops, and I once forgot to turn on the main disconnect switch after some troubleshooting.
We didn’t install any battery-based power storage that could power the house in the event of an outage, partly because when we designed the system, power outages had been infrequent and brief. We did opt to put a standard electrical outlet on each inverter so we could tap into whatever power was being generated during an outage for specific uses. In the winter, that would be sufficient to charge our phones (100–300 watts of output, if the panels aren’t covered with snow, with chargers drawing 5–60 watts); in the summer, it’s more than enough to power our garage chest freezer (3000–5000 watts of output, with the freezer drawing only 150 watts after a startup spike of maybe 300 watts).
Power Outages Compromise Connectivity
From July 10 through July 16, we lost power three times. The first happened during a tornado warning but was actually caused by the local highway department repaving our road, raising it enough that a passing truck caught and broke the overhead electric wires leading to our house. Oops! The remaining two outages stemmed from the severe storms that have become commonplace with climate change. I grew up on a farm 12 miles away, watching the summer weather intently to make hay with my father, and today’s weather patterns are vastly more volatile, as you can see from the alerts that rolled in just while I was writing today. Shortly after I finished writing this article, we lost power again. It’s getting tedious.
After a previous power outage this spring (we’ve had six so far in 2024, a new record), the SMA inverters lost connectivity, and I had to perform a reset sequence on them to get them to talk to the portal again. Happily, Halco, the company that installed them, was happy to tell me via email what to try so I didn’t have to wait for a tech just to flip three switches and circuit breakers in the correct order.
The inverters lost connectivity again after the first of the recent power outages on July 10. I assumed—bad idea!—it was the same problem, so I ran through the reset sequence. But we lost power again on July 11, so I didn’t know if my reset worked. I remembered to reset it again a day or so later, but the inverters didn’t appear on the portal, and I got caught up in weekend activities and failed to check again until July 15. I never saw if that reset worked because we lost power again that day, our longest outage ever at 29 hours.
During that last outage, I used an inverter outlet to power our chest freezer for the first time, so I was fussing with the inverters for much of the day. When disconnected from the grid, they report only the draw, not production, and it took me a while to figure this out and prove to myself that they were producing enough that I didn’t have to worry about browning out the freezer. I flirted with connecting the freezer to my iMac’s unused UPS to eliminate the possibility of a cloud or darkness dropping the power generation to a dangerously low level, but the freezer wouldn’t run from the UPS for reasons I still don’t understand. Perhaps the UPS could sense that the freezer’s power draw was unusual, even though it should have been able to handle 150 watts of draw.
Once the power came up for good, I ran through the reset sequence again on July 17, to no avail. I tried it a few more times before contacting Halco. They agreed that I’d tried all the basics and promised to send a tech out. Luckily, he had some time a few days later, well before the scheduled appointment, and he quickly revealed something I hadn’t known—that you can knock on the front of the inverters in a certain way to get them to cycle through status messages, one of which showed that they had IP addresses and thought they were communicating correctly. Yes, part of the troubleshooting process was to smack the case, just as we occasionally had to do to early computers.
Is It Plugged In?
“It must be something inside,” he declared, so we went in to look at my networking gear. I showed him where the Ethernet cable from the inverters plugged into an Ethernet switch… and why wasn’t that switch showing any activity lights? “Maybe it got fried in the outage,” he suggested, but I pointed out that it was plugged into a UPS. We traced its power cable down to the UPS, where it was plugged in next to the hub for my temperature monitoring system (see “Wireless Sensor Tags Protect Against Freezer Failure,” 25 July 2018), whose activity lights were equally dark. The UPS was working, since the Eero base station that manages my network was online, as was the device that lets me manage my Honeywell thermostat from an iPhone app.
We pulled the UPS out from its dim corner and examined it. That was when I realized what had happened. This particular APC UPS has a “master” outlet (the top left outlet in the photo below) and several “controlled” outlets (in the bottom left). When the device plugged into the master outlet goes into sleep or standby mode, or is switched off, the UPS cuts power to the controlled outlets to save energy. However, this mode isn’t enabled on this UPS by default—there’s a physical Master Enable button you press to turn it on. But the Eero Pro, being the most important of the networking devices, was plugged into the master outlet. Why didn’t that keep the controlled outlets powered up? And what changed during the outage such that the controlled outlets became disabled?
Each time the power went off, I turned the APC UPS off to silence its annoying alert beep. Because it’s in a dark corner and I couldn’t turn on a light, I had to feel around for the power button somewhat blindly. (This button is illuminated, but it’s common for a power switch to turn on a separate status light.) During one of the first two outages, I must have inadvertently pressed the Master Enable button before finding the actual power button. The Eero Pro consumes only about 5 watts of power, presumably below the threshold that the APC UPS needs to power the controlled outlets. It makes sense; if I had a Mac plugged into the master outlet and it dropped to 5 watts, there would be no need to power peripherals necessary only when the Mac is in use.
Once I took the APC UPS out of master/controlled mode, the Ethernet switch and temperature monitoring hub instantly powered up and started working, and the SMA inverters immediately connected to their portal. Problem solved!
Lessons Learned
Why didn’t I figure this out on my own sooner? In my defense, my Wi-Fi network was functional, and because my iMac uses both Ethernet and Wi-Fi, I didn’t notice that the powered-down Ethernet switch was preventing Ethernet from working in my office. In retrospect, Tonya and I did notice some slight network issues that were probably related to the extension Eero in my office being forced to use wireless for backhaul instead of Ethernet.
Plus, two of the four networking devices in the collection were working, so some LEDs were on back there. I had even unplugged and replugged the Ethernet cable from the inverters, so I had looked at the Ethernet switch without realizing it was powered down. I often go months without a notification from the temperature monitoring system, so it wasn’t surprising not to have heard from those sensors during all this. And I very seldom need to control the Honeywell thermostat from my iPhone, so I didn’t notice it was offline (since it was plugged into the disabled Ethernet switch, even though it had power).
In other words, my troubleshooting was sloppy. Because I had solved the inverter connectivity problem in one way previously, I didn’t look beyond that solution to follow the connectivity path upstream. Once the Halco tech made me do that, the problem quickly became apparent.
And that, dear readers, is the moral of the story. If your first attempt at solving a technical problem fails, break the problem into parts and verify that each of those parts is functional on its own. Every problem will be different, but by starting at one end and eliminating every possible variable, you should be able to discern the cause.
Thanks @ace for an excellent article. I’ve been through similar situations and equally stumped. Most of the time I find that somewhere along the way in my debugging I had assumed something about one of the steps that I didn’t actually verify. In your case you thought the UPS was working, but hadn’t actually verified it. This has happened to me countless times. Thanks for the reminder to check every link in the chain!
Really terrific article, @ace Adam! Thank you.
On a side note, and I don’t mean this in a creepy way, it’s a little eerie and encouraging to see your solar and networking setups. I’ve been through so many attempts over the years to keep the different boxes and cables completely out of sight, and finally just decided to embrace it all. I think it’s helped a bit with troubleshooting, especially because the ASUS router and the fiber modem occasionally stop talking to each other.
Our solar setup, which includes a battery, is both simpler and more complex. The panels each have a microinverter (37 all together) and there are three more in the battery box (at left in the photo).
Troubleshooting in this case involves making sure each of the microinverters on the roof is still talking to the combiner. I can see a long list of all the microinverters in the system portal, but I’ve already had one instance where the portal refused to communicate with anyone, including the manufacturer. It required time on the phone while they walked me through rebooting, then installed firmware fixes that they “hoped” would prevent future problems.
I have already had to learn how to establish a direct connection to the combiner through its local WiFi portal (which we don’t use for day-to-day stuff, as it also has a cellular modem and an Ethernet connection cabled back to the rack). I’ve also confirmed that my general approach of treating the support folks politely and with patience gets things fixed fast.
Glad that you found the issue; I would never have guessed that particular thing without a second pair of eyes on the problem!
To expand on this slightly: I’m the one on the team people come to when they’ve exhausted their troubleshooting knowledge and can’t figure out why something isn’t working.
I start asking them basic questions. Was this program recompiled? Did you reset this database? Is this pointing to your override? Is your program really using this test file, or is a different file?
And when they say, “of course it is!”, I say, prove it. And often that’s when we find the problem.
Because, when you have gone through all the normal steps and still can’t find why it doesn’t work, it means one of those “facts” is wrong. You must question all of your assumptions.
"Percussive maintenance "to the rescue!
Your comments about temperature extremes reminds me of my experience with the heat pump we installed in the fall of 2022. Our previous heating system was forced hot air, so the installers fed the heat pump output into the old ventilation system, saving us a bundle. We noticed that in winter the heat pump started pumping cold air through the system, but the installers told us that was normal and it should not be a problem in a normal winter. Then a seriously subzero cold rolled in the Boston area (it may have hit - 15 F in the wee hours of the morning) and the heat pump simply could not pump heat at those temperatures. I never realized how cold a house can feel when it can’t reach 50 °F! I was on the phone with the installers, and they explained that at seriously subzero temperatures, you had to install special wiring to use resistance heating (like a standard electric oven) to warm the heat pump enough so it could pump heat instead of cold. They had not done it because they thought it wouldn’t be necessary, but they hadn’t expected weather that cold. We had them install the special circuit, and thankfully haven’t had another such extreme cold snap. But it’s a reminder that extreme weather can push household equipment beyond its design zone.
Sadly, the installer took advantage of you here.
There’s no reason why they couldn’t install a gas or oil furnace feeding your hot air system’s ducts. Yes, you would need a separate compressor for use in the summer, but eliminating the furnace to buy less equipment in a cold location like Boston should never have been considered. (I am aware of the fact that electric utilities everywhere try to promote heat pumps where they are a bad idea, simply to shift your heating expenses from the gas company to themselves, but it remains a bad idea).
While some modern heat pumps can deal with massively cold temperatures, most can not, and they fail miserably when the weather gets that cold.
The fact that they didn’t think a resistive coil system was necessary in the Boston area is stark raving mad. We have them in our heat pump system in Virginia, and our weather never gets as cold as your weather.
The ideal solution, which should have also been suggested is a dual-fuel system. This is a system where you have a traditional gas/oil furnace, combined with a heat pump, and possibly also resistive coils. So when the temperature remains warm enough for the heat pump to be efficient, it uses that. When it gets too cold for that, it switches to gas/oil. And if that should fail, the coils serve as an “emergency” heat source. This is nothing new - I’ve seen HVAC companies advertise this 20 years ago.
Dual-fuel is a more expensive than a heat pump (or furnace) by itself, but not much more expensive than a furnace plus an air conditioner. You’d think that HVAC installers would promote this, since it is really the best of both worlds, but for some reason, they usually don’t.
I agree that the installer could have done better, but it’s not that simple.
Our Boston suburb has been pushing heat engines, and a local green volunteer helped in discussing the options. In 2022, we were eligible to receive a $10,000 rebate from the local electric company, which covered a little more than half of the total bill. They did mention the option of keeping the old heating system as a backup. However, we could only get the full rebate if we took out the old heating system, which was a 35 year old oil burner close to the end of its useful life, and the installer said there was no room to fit both the heat pump and the old heater. (I can’t remember if there was any mention of the cost or possibility of putting in a new gas heating backup, but I was more interested in getting the full rebate.)
The dual-fuel option wasn’t in the city’s green agenda, but the installer we chose did mention it.
I agree they should have pushed the resistive coil heating element. But it had been a long time since Boston temperatures had hit minus 15 F and they assumed we were interested in saving money. All in all, I regard this as annoying teething problems of a new heating technology. At some point I will probably want to investigate adding more thermostats to get more cooling into the attic where I work, but we can make do by shutting down some of the grates in the system.
re: why your freezer wouldn’t run on the Mac’s UPS.
When the compressor of a freezer starts, the current draw is sudden and extreme. I am not surprised that a UPS wasn’t able to cope.
“Percussive maintenance,” indeed! That reminds me of my first duty assignment in the Air Force, circa 1965. The Special Projects Division of the Global Weather Center was supported by an IBM 7094 mainframe with a whopping 32K words of memory plus eight half-inch tape drives. It was served by an IBM 1410 that converted punched cards to input tape, and output tape to print, both at quite high speeds. Visitors were always impressed by the huge and immaculate computer room, but if they walked around the equipment, they were likely to be puzzled by a grubby-looking length of 4x4 timber that lay in a shadowed corner. It had been salvaged from the nearby motor pool, and was called the “core whomper.” See, the 7094 had an oil-cooled magnetic core memory, and somewhere in that oil was a tiny piece of loose metal that was never captured by the filters. When it got caught on a core, it would cause a parity error that shut down the machine. The remedy was to pick up the 4x4 and whomp on the memory case until that bit of metal was shaken off its resting place and order was restored. Fortunately, that didn’t happen often, and I never saw it actually used. But it added an interesting bit of local legendry to an otherwise drab military environment.
Adam, thank you. I soak up these types of articles like a sponge, since troubleshooting is part of my vocation (like @mschmitt). I have to press myself to think outside the box, and it can be a challenge since we are all set in our ways. I learned a few more tricks from your post.
Matt Clark
Hi Adam, I’d be interested in reading an article about your system.
I was testing all this stuff using a WattsUp Pro that shows power draw, and as far as I could tell, the startup draw from the freezer’s compressor was only about 300 watts, which would have seemed within the UPS’s capabilities. But perhaps it was too fast or something like that?
Alas, we installed it 9 years ago, and while I participated in the planning with Halco, the installer, I didn’t get into the weeds enough to know that much about it. I suspect the technology has changed and improved since then. Overall, though, we’re happy with how it has worked.
My CyberPower UPS has a total power off button, press in to click on or off, RIGHT where you have to put your left hand to pull plugs out as the female is way too tight with right hand. Same for pushing in.
After shutting off all my stuff over and over by accident, I have trained myself to not press that button! In industrial design, this is called poor affordance. Basically, they put a mission critical but all too easy power button in a form factor that is all too easy to use by accident. It actually has me not buying a second one with the same button despite it otherwise being an excellent UPS. They still use the same button. So dumb!
I recently bought a small fridge/freezer for my homebuilt campervan and I was testing on a battery pack to see how it would run. It’s very low-power (45 watts), but even though the battery I was using runs up to 300 watts, it would die on freezer start-up.
At first I thought I was screwed, but then I found the “DC” setting for the freezer that you’re supposed to use when running it off a battery. Apparently that changes how the freezer starts up and once I turned that setting on, literally no issues running the freezer off the battery or the car cigarette lighter.
This particular freezer is made for car/battery use, so it’s designed to work this way. A regular freezer would not be and that startup surge might overwhelm a regular UPS.
(Even with my small battery I can run the freezer at 10 degrees for 8+ hours. It’s been awesome for traveling and camping! I just use it to freeze ice packs which I swap out every couple of days in my larger cooler which I use as my actual fridge. I run the freezer off shore power at night when I’m parked at a campsite and off battery when away from power and on a hike.)
Good one. I learned how to code in that era. What young USERS don’t know would overflow TBytes of HD.
Exactly. What’s the sampling rate of that meter? And then there’s the UPS itself. Just because it’s rated for 300 W (or 300 V-Amps), does not necessarily mean it can cope with very quick draw to that level from zero. Such average power ratings and especially V-Amps are usually DC (or perhaps rms) figures. They don’t tell you how quick the UPS can follow a rapidly increasing load.
Thanks!
It looks like the Watts-Up Pro has a sample rate of 1 per second, which would probably be too slow to capture and display whatever the surge really is.
The UPS claims it can put out 815 watts, but there’s nothing that talks about how quickly it can respond. So that’s likely the issue.
https://www.cyberpowersystems.com/capacity-va/1350-va/
FWIW, the spec sheet for my laser printer (Brother HL-L3270CDW) says that it draws approximately 430W while printing, but the peak power draw is 3x that: 1290W
I assume the peak power is the initial inrush to quick-heat the fuser (which often causes a brief voltage sag where the room lights flicker and the UPS next to it on the same circuit briefly switches over to battery).
The CyberPower UPS I have recommends not using its UPS plugins for a laser printer, for that very reason. It does show the voltage drop when I print, but doesn’t switch to battery.
I don’t know of any UPS from any manufacturer, with any capacity that says it’s OK to to connect a laser printer (or other high-currently device like a kitchen appliance or a sump pump), and I think it’s precisely because of the high current draw and even higher inrush current when it turns on.
As for my UPS switching to battery from the voltage sag, I assume this is its “AVR” (automatic voltage regulation) circuit kicking in. I don’t think it detected an outage during that time. And it only lasts for a fraction of a second.
My solar installer strongly recommended against connecting the blowers for our two gas-fireplace inserts to the backup load for the house. It was a general guideline that discourages connecting any load that includes a motor because of startup surge.
Freezer and refrigerator are connected with no problems. But they considered the little blower motors a bridge too far.