Read-out of Power production of individual solar panels with Enphase microinverters

[FireWizard]

Dear interested reader,

Below find a brief description on how to extract the Power production per solar panel, if you have Enphase micro inverters installed and to present these in Domoticz.

For a brief introduction, please see:
https://www.domoticz.com/forum/viewtopi ... 31&t=29170

In order to get the data from the Enphase Envoy-S or the metered version, you need to install a MQTT broker, like Mosquitto, Node JS and Node Red.
This will give you an excellent companion to Domoticz and for non script gurus or programmers an easy way to get things programmed and configured for use in Domoticz.

Follow the instructions and links under the picture in the following thread:
https://www.domoticz.com/forum/viewtopi ... 28&t=22368

If you can access Node Red flow editor from any webbrowser with http://<IP address>:1880, you are fine. Test also your MQTT broker.

In Domoticz hardware, add MQTT over LAN. Check with a tool that you can subscribe to domoticz/out. I use: https://mqttfx.jensd.de/, but there are many other tools.

Later I will describe, how to read the data from the Envoy-S.

[FireWizard]

This is the second part of the description.

If you have finished the setup of the required programs, it is time to start with the flow.

In order to get the Power production of each individual solar panel/micro inverter as a sensor in Domoticz you have to create the sensors first as follows:

1. Go to "hardware" and create a new hardware device called Dummy (Does nothing, use for virtual switches only).
2. Give it a name, e.g. Enphase inverters.
3. Create virtual sensors.
4. Give the sensor a name, e.g. Solar Panel 1.
5. As sensor type, select Electric (Instant+Counter).
6. Repeat step 3 to 5, as many times as you have number of solar panels.

So, if you have 8 solar panels, you have now created 8 virtual sensors of the type General and subtype kWh.

Go to the devices in the setup tab and look up your newly created devices.
For each created sensor, note their name and Idx.
You will need that later.

Screenshot_IQ7-1.png (44.87 KiB) Viewed 7240 times

OPTIONAL

If you want also the maximum produced power of a solar panel/micro inverter visible in Domoticz, repeat the steps 3 to 6. However the sensor type has to be Usage Electric.
In this case you will have for each solar panel a sensor, indicating the max. power ever produced.
If you have reached a certain maximum on a sunny day, it will not change until you have another and higher maximum.

Screenshot_IQ7-2.png (48.42 KiB) Viewed 7240 times

The next step is that you go to the Utility tab and look up the sensors just created.

Click on Edit and change the Type from Usage to Return.
Leave Energy read as Computed.
Press Update.
Do this for every created sensor.

Screenshot_IQ7-3.png (17.75 KiB) Viewed 7240 times

You will see that the icon changes to a solar.

Your sensors are waiting to receive data from Node Red.

[FireWizard]

@miki8889,

Indeed, I did not publish the node red part, but it will follow as soon as possible.
Currently I don't have access to my node red editor (because of holidays).
It is difficult or even not possible to use some screenshots.

I will make a description later, and all, who don't need screenshots, can make their own flow.
Others have to wait a little.

Regards

[FireWizard]

This is the third part of the description.

I assume that both Mosquitto and Node Red is running fine and that Domoticz can communicate with the Mosquitto broker.

Before we start with it, we should collect all serial numbers, as Enphase is rather "serial number" oriented.
We will need the serial numbers of the Enphase micro inverters, as we should make a table with the relationship between the Enphase inverter and the IDX of the sensors.
So we have to define for ourselves, which panel, e.g.Solar Panel 1, has which serial number and the IDX of the sensor.

For the flow we will need 5 or 6 nodes.

1. The first node in the flow will be the "inject" node. This is one of the core nodes.
See: https://nodered.org/docs/user-guide/nodes
We can leave most of it standard.
Payload has to be the Unix time stamp.
At the end you want to set the poll time of the Enphase Envoy S or Envoy S metered, but during the development I suggest that you leave that box "unticked", as, while building the flow, it is disturbing. Enphase publishes that the values are updated every 5 minutes, but I have seen quicker updates. So I took the same update time, as the Domoticz Enphase hardware does, 30 seconds.

Screenshot_IQ7-NodeRed-1.png (26.59 KiB) Viewed 7240 times

2. The second node is optional.
As I think, that it is not useful to poll the Envoy S during the "dark" period, I inserted a timer which blocks the 30 second time tick between 30 minutes after sunset until 30 minutes before sunrise.
So only during the period between 30 minutes before sunrise until 30 minutes after sunset the Envoy is polled.
I use the within time node in: https://flows.nodered.org/node/node-red ... ithin-time.
As your Envoy is in your own LAN you can omit this node, as their is no limit regarding the number of requests. During the building of the flow leave it out, as you cannot test the flow during the dark hours.

Screenshot_IQ7-NodeRed-2.png (38.14 KiB) Viewed 7240 times

Do not forget to configure the configuration of the timecontrol (The pencil at the right).
Insert your lat/lon co-ordinates.

3. The next node is the http request node from the function nodes block.

The URL you should poll has to be: http://<YOUR ENVOY IP ADDRESS>/api/v1/production/inverters
You have to fill in your own IP address of your Envoy S or Envoy S metered.
This URL is username/password protected.
The fact that using scripts is difficult to poll this URL, is that this URL uses digest authentication, while mostly basic authentication is used.
For those interested in the technical backgound, see: https://en.wikipedia.org/wiki/Digest_ac ... entication and other sources.

The username is "envoy" (without "") and small letters.
The password is the last six digits of your serial number of the Envoy S or Envoy S metered.
I hope you can find that serial number.

If not:
1. See the affixed sticker at you Envoy.
2. In your browser, open: http://<YOUR ENVOY IP ADDRESS>/home
3. In your browser, open: http://<YOUR ENVOY IP ADDRESS>/info.xml (See the first sn)
4. Do something wrong: http://<YOUR ENVOY IP ADDRESS>/error

So, as you see the serial number of your Envoy (and so the password) is displayed almost everywhere.

Screenshot_IQ7-NodeRed-3.png (38.34 KiB) Viewed 7240 times

The results will be shown in the next post.


Regards.

[FireWizard]

In this post you will find the results of the http request.

In order to see the results, you have to connect a "debug" node to the output of the "http request" node.

Press the left square of the inject node. It will present you the following in the "debug" node:

Screenshot_IQ7-NodeRed-4.png (4.83 KiB) Viewed 7238 times

It is an array of objects and each object contains the data of each inverter.

Opening the array, by clicking on the arrow you will see the following:

Screenshot_IQ7-NodeRed-5.png (7.76 KiB) Viewed 7238 times

For each inverter you will see an object. So your array contains as many elements as you have inverters/solar panels.

Screenshot_IQ7-NodeRed-6.png (22.34 KiB) Viewed 7238 times

You will see two variables, we are interested in:

1. lastReportWatts, this is the currently produced power of the inverter, expressed in Watts.
2. maxReportWatts, this is max power the inverter produced in a certain period, also expressed in Watts.

I have no idea when that period is reset, but I have seen changes.
[Edit] Your highest score will be lost as soon as you power down you Envoy-S, so in fact you start a new period
As said, this is an optional sensor, you may or may not want to use.
The first one is the most interesting.

[FireWizard]

As I'm not sure that the object, containing the inverter data, is always the same element in the array, we will not use the position in the array as the "key". It might happen, that because of a power down of the Envoy S, the objects are presented in a different order.

In order to use the serial number of the inverter as the "key", we have to use a split node.
This node will split the different objects in the array as different, single objects.

You can leave the properties of the "split" object as default.
See below

Screenshot_IQ7-NodeRed-7.png (23.28 KiB) Viewed 7233 times

The result should then be as follows:

Screenshot_IQ7-NodeRed-8.png (36.28 KiB) Viewed 7233 times

The next step is to prepare the data in such a way that Domoticz understands it and that the right "lastReportWatts" is presented for the right sensor.
Therefore we need a table where we can see, which inverter (serial number) sends its value to which IDX in Domoticz.
You have prepared this table, haven't you?

[FireWizard]

To prepare the data for Domoticz we will use a function node, which is one of the core nodes of Node Red.
Drag and drop the function node to the canvas and connect it with the output of the previous "split" node.
Double click on it and the node will open.

Screenshot_IQ7-NodeRed-9.png (11.16 KiB) Viewed 7208 times

The background of the syntax of messages, accepted by Domoticz, is described in the following links:
https://www.domoticz.com/wiki/MQTT.
The syntax is similar as described here:
https://www.domoticz.com/wiki/Domoticz_API/JSON_URL%27s

Another good source of information you will find here:
https://piandmore.wordpress.com/

In order to parse the data from the "split" node and to format it, ready for Domoticz, you can use a number of "If-then-else" statements.
However, this is in general less efficient as to use "switch case" statements.

In case, you have chosen, to create a sensor for both lastReportWatts and maxReportsWatts, you have the following code:

Code: Select all

var msg1 = {};
var msg2 = {};

switch (msg.payload.serialNumber) {

    case "12185100xxxx": //Solar Panel 1
    msg1.payload = {"command":"udevice","idx":263,"nvalue":0,"svalue":((msg.payload.lastReportWatts).toString() + ";0")};
    msg2.payload = {"command":"udevice","idx":271,"nvalue":0,"svalue":(msg.payload.maxReportWatts).toString()};
    break;

    case "12185100xxxx": //Solar Panel 2
    msg1.payload = {"command":"udevice","idx":264,"nvalue":0,"svalue":((msg.payload.lastReportWatts).toString() + ";0")};
    msg2.payload = {"command":"udevice","idx":272,"nvalue":0,"svalue":(msg.payload.maxReportWatts).toString()};
    break;

    case "12183402xxxx": //Solar Panel 3
    msg1.payload = {"command":"udevice","idx":265,"nvalue":0,"svalue":((msg.payload.lastReportWatts).toString() + ";0")};
    msg2.payload = {"command":"udevice","idx":273,"nvalue":0,"svalue":(msg.payload.maxReportWatts).toString()};
    break;

    case "12183402xxxx": //Solar Panel 4
    msg1.payload = {"command":"udevice","idx":266,"nvalue":0,"svalue":((msg.payload.lastReportWatts).toString() + ";0")};
    msg2.payload = {"command":"udevice","idx":274,"nvalue":0,"svalue":(msg.payload.maxReportWatts).toString()};
    break;

}
return [[msg1,msg2]];

In this case I have an example of 4 solar panels/inverters.
You should insert the serial numbers of the inverters after "case" and between " ".
Also you should replace the idx. Therefore you need that table.
Text after // is "Comment".

For every panel less than four, you have to delete a block of code between and including case and break;.
(Don't forget the semicolon after break).
In case, you have more than 4 solar panels/inverters, you should add additional blocks of code.

If you have opt to use only lastReportsWatts, the code can be simplified to:

Code: Select all

switch (msg.payload.serialNumber) {

    case "12185100xxxx": //Solar Panel 1
    msg.payload = {"command":"udevice","idx":263,"nvalue":0,"svalue":((msg.payload.lastReportWatts).toString() + ";0")};
    break;

    case "121851xxxx": //Solar Panel 2
    msg.payload = {"command":"udevice","idx":264,"nvalue":0,"svalue":((msg.payload.lastReportWatts).toString() + ";0")};
    break;

    case "12183402xxxx": //Solar Panel 3
    msg.payload = {"command":"udevice","idx":265,"nvalue":0,"svalue":((msg.payload.lastReportWatts).toString() + ";0")};
    break;

    case "12183402xxxx": //Solar Panel 4
    msg.payload = {"command":"udevice","idx":266,"nvalue":0,"svalue":((msg.payload.lastReportWatts).toString() + ";0")};
    break;

}
return msg;

Now it is really time to connect a debug node to the output of this function node and to check carefully that the idx of all sensors is correct and that the idx is not "double" used.
If the output of the function node look like this screenshot you are fine.

Screenshot_IQ7-NodeRed-10.png (33.46 KiB) Viewed 7208 times

Note: The command "udevice" is optional as Domoticz defaults to "udevice", but for completeness and clarity, I have included this command.
If the output looks like the screenshot and you have included your own indexes, you are ready to connect the output to the last node in the flow.

[FireWizard]

The last node of the flow is the output node MQTT.

Drag and drop this node to the canvas and double click to configure it.

Screenshot_IQ7-NodeRed-11.png (20.38 KiB) Viewed 7207 times

The Server name is up to you.
The Topic should be "domoticz/in" (without " ")
QoS, Retain and the Name (of the node) is up to you.

The next step is to press the "pencil", right of the Server name:

Screenshot_IQ7-NodeRed-12.png (28.53 KiB) Viewed 7207 times

Configure the IP address of the MQTT broker. Leave the Port number as default: 1883.
If your Mosquitto broker is installed on the same device as Node Red and Domoticz, you can use localhost.
The other things, you can leave as default.

If you have secured your MQTT broker with a username and password, which I strongly recommend, you should go to the Security tab, otherwise you can quit by pressing "Update".

Screenshot_IQ7-NodeRed-13.png (17.06 KiB) Viewed 7207 times

The Messages tab , you can leave untouched, and you should press "Update".

Press "Done".

[FireWizard]

Final.

If everything has been configured successfully, you will see that the MQTT output node is connected to Domoticz.

Screenshot_IQ7-NodeRed-14.png (4.16 KiB) Viewed 7206 times

If you connect the output of the function node to the input of the MQTT output node the values should be presented in Domoticz.

Screenshot_Zonnepaneel 123 sensors.png (137.63 KiB) Viewed 7206 times

The only thing we need now is sunshine.

I hope this series is complete and clear, even for a starter, and that it will be useful.

I'm open for suggestions and comments.

Regards,

[johnnie]

Hi Firewizard, or other experts,

I'm trying to get the enphase visible in Domoticz with the help of this thread, but I'm stuck at the node split (I think). Any help is much appreciated.

The first capture is the one from you're post,
The two following; is the detailed one by clicking object/{0-09]/object and the last picture the collapsed one. In the later, you're capture is showing details.

How can I troubleshoot?

Allready thanks, Johnnie

[FireWizard]

Hi,

@johnnie

Is it possible to post your flow. so far?

Regards

[johnnie]

Hi FireWizard,

Hereby, I only removed the ip adres and changed my password for the envoy:

Code: Select all

[{"id":"ada30381.aa087","type":"tab","label":"Enphase Metered","disabled":false,"info":""},{"id":"712730fd.6a","type":"inject","z":"ada30381.aa087","name":"Timer (30 sec)","topic":"","payload":"","payloadType":"date","repeat":"30","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":80,"wires":[["8cf90fde.8ebe7"]]},{"id":"256cd435.2fe4dc","type":"http request","z":"ada30381.aa087","name":"Envoy-S Productie/Inverter request","method":"GET","ret":"obj","paytoqs":false,"url":"http://111.111.111.1/api/v1/production/inverters","tls":"","persist":false,"proxy":"","authType":"digest","x":400,"y":140,"wires":[["3f97de95.1383c2","9c88df6b.66e16"]]},{"id":"9c88df6b.66e16","type":"debug","z":"ada30381.aa087","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":690,"y":80,"wires":[]},{"id":"8cf90fde.8ebe7","type":"within-time-switch","z":"ada30381.aa087","name":"Dagtimer","positionConfig":"e24dbd2f.a9137","startTime":"sunrise","startTimeType":"pdsTime","startOffset":"-30","startOffsetType":"num","startOffsetMultiplier":60000,"endTime":"sunset","endTimeType":"pdsTime","endOffset":"30","endOffsetType":"num","endOffsetMultiplier":60000,"timeDays":"*","timeOnlyOddDays":false,"timeOnlyEvenDays":false,"timeMonths":"*","propertyStart":"","propertyStartType":"none","propertyStartCompare":"true","propertyStartThreshold":"","propertyStartThresholdType":"num","startTimeAlt":"","startTimeAltType":"entered","startOffsetAlt":0,"startOffsetAltType":"none","startOffsetAltMultiplier":60000,"propertyEnd":"","propertyEndType":"none","propertyEndCompare":"true","propertyEndThreshold":"","propertyEndThresholdType":"num","endTimeAlt":"","endTimeAltType":"entered","endOffsetAlt":0,"endOffsetAltType":"none","endOffsetAltMultiplier":60000,"tsCompare":"0","lastMsgOnStartOut":false,"lastMsgOnEndOut":false,"x":140,"y":160,"wires":[["256cd435.2fe4dc"],[]]},{"id":"3f97de95.1383c2","type":"split","z":"ada30381.aa087","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":330,"y":220,"wires":[["b55fe34.a5f072"]]},{"id":"b55fe34.a5f072","type":"function","z":"ada30381.aa087","name":"IQ7-IDX","func":"var msg1 = {};\nvar msg2 = {};\n\nswitch (msg.payload.serialNumber) {\n\n    case \"121916077118\": //Solar Panel L.1.1\n    msg1.payload = {\"command\":\"udevice\",\"idx\":48,\"nvalue\":0,\"svalue\":((msg.payload.lastReportWatts).toString() + \";0\")};\n    msg2.payload = {\"command\":\"udevice\",\"idx\":49,\"nvalue\":0,\"svalue\":(msg.payload.maxReportWatts).toString()};\n    break;\n\n    case \"121916072218\": //Solar Panel L.1.2\n    msg1.payload = {\"command\":\"udevice\",\"idx\":50,\"nvalue\":0,\"svalue\":((msg.payload.lastReportWatts).toString() + \";0\")};\n    msg2.payload = {\"command\":\"udevice\",\"idx\":51,\"nvalue\":0,\"svalue\":(msg.payload.maxReportWatts).toString()};\n    break;\n\n    case \"121916073189\": //Solar Panel L.1.3\n    msg1.payload = {\"command\":\"udevice\",\"idx\":52,\"nvalue\":0,\"svalue\":((msg.payload.lastReportWatts).toString() + \";0\")};\n    msg2.payload = {\"command\":\"udevice\",\"idx\":53,\"nvalue\":0,\"svalue\":(msg.payload.maxReportWatts).toString()};\n    break;\n\n    case \"121916072215\": //Solar Panel L.1.4\n    msg1.payload = {\"command\":\"udevice\",\"idx\":54,\"nvalue\":0,\"svalue\":((msg.payload.lastReportWatts).toString() + \";0\")};\n    msg2.payload = {\"command\":\"udevice\",\"idx\":55,\"nvalue\":0,\"svalue\":(msg.payload.maxReportWatts).toString()};\n    break;\n\n}\nreturn [[msg1,msg2]];","outputs":1,"noerr":0,"x":500,"y":220,"wires":[["14675a13.cf6526"]]},{"id":"14675a13.cf6526","type":"mqtt out","z":"ada30381.aa087","name":"Domoticz In","topic":"domoticz/in","qos":"2","retain":"false","broker":"f2d7ce3c.0e928","x":710,"y":220,"wires":[],"icon":"node-red/batch.svg"},{"id":"e24dbd2f.a9137","type":"position-config","z":"","name":"","isValide":"true","longitude":"0","latitude":"0","angleType":"deg","timeZoneOffset":"99","timeZoneDST":"0","stateTimeFormat":"3","stateDateFormat":"12"},{"id":"f2d7ce3c.0e928","type":"mqtt-broker","z":"","name":"RaspiTest","broker":"127.0.0.1","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

Best regards, John

[FireWizard]

Hello,

@johnnie,

Sorry to say, I cannot import he flow. Due to a syntax error.
(SyntaxError: JSON.parse: expected ',' or '}' after property value in object at line 1 column 529 of the JSON data)

Could you try to post it once more?
And I suggest to copy it as code (</>) and not spoiler.

Regards

[FireWizard]

@johnnie,

Not needed anymore, I have the flow imported and will look to it.

Regards

[FireWizard]

Hello,

@johnnie,

I tested your flow and its seems to be OK.

I have noted a couple of things.

1. You have an Envoy Metered, while I have the standard Envoy. While this may produce a different output, I do not believe this is the case, if you try to collect data from the individual inverters.
2. You have 18 inverters, while I have 8. If you have more than 10 elements in an array, the debug output splits it up in "blocks" of 10.
So you have first [0 ... 9], but in addition you will also see (not visible in the screenshot) [10 ...17].
This is not a problem.
3. I removed also the "within time node". In another project, some issues has been discovered, and so I replaced this node by another one, that exactly does, what we need here: node-red-contrib-time-switch. You can install this from the palette in Node Red.
Configuration is self-explaining. Insert your lat/lon co-ordinates and the time you wanted it, to switch.

Screenshot_time_switch.png (19.82 KiB) Viewed 6457 times

4. I inserted my IP address of the Envoy and username (envoy) and password (6 last digits of serial number) and your flow produces exactly the same output as my flow does. So I believe you don't have a problem. The split node should produce 18 individual objects.
In the picture you showed, the devType and lastReportDate are shown in HEX value.
Connect a "debug" node to the output of the "split" node and check that you have 18 individual messages with objects.

Regards

[johnnie]

Hi FireWizard,

Thanks for helping me out, and I will change the node as advised.
Does it matter that I only populated 4 micro's in the function-node after the split-node instead of the full 18?

Best regards, Johnnie

[FireWizard]

Hi,

@johnnie

You wrote:

Does it matter that I only populated 4 micro's in the function-node after the split-node instead of the full 18?

No, it doesn't matter, but you will only have output for the 4 inverters, for which you configured the serial numbers.
But, I think, at the end you want them all 18.

Regards

[AUitdehaag]

Thank you for the node-red flow.

I have one question: How do you get the kWh per inverter in domoticz?
I have found another dzVents script, which uses the (once per) 15 minutes value and adds the Watts divided by 4 to get the Wh.
As I am not a software engineer , do you have that working in node-red?

Any help is appriciated!

[FireWizard]

Hi,

@AUitdehaag

You wrote:

I have one question: How do you get the kWh per inverter in domoticz?

It is described earlier in the thread.

You will find the values for the individual inverters at: http://<IP address of your Envoy S>/api/v1/production/inverters
This URL is username and password protected.
The username is "envoy", without " ".
The password is the last 6 digits of the serial number of your Envoy S.

If you fill in these values in the URL above in your browser, you will get:

Code: Select all

[
  {
    "serialNumber": "123456789123",
    "lastReportDate": 1587314803,
    "devType": 1,
    "lastReportWatts": 44,
    "maxReportWatts": 229
  },

In this example produced the panel 44 Watts

How to do it is described in this thread. Be aware that the URL uses "Digest Authentication".

You have to create new hardware of type Dummy (Does nothing, use for virtual switches only).
The sensor used in Domoticz is a virtual sensor of the type "Usage (Electric)".
You should edit this sensor and change "Type" to "Return" and set "Energy read": to "Computed".

Best regards

[AUitdehaag]

@FireWizard.

Thanks a lot for explaning.
I was "missing" the energy read "computed" setting.
I adjusted it and now it works fine!