how to add APSystems ECU-3 solar inverter monitor

[FireWizard]

Hello Bernhard,

I would like to split this thread in 2 pieces, if you agree:
1. Check, so that we are sure that all the tools are functioning.
2. Try to communicate with APSystems

You said

MQTT, Node JS and node red installed

You said also

Can receive MQTT messages from Domoticz when my broker subscribes to domoticz/out

I wonder, how did you check that?

As you have installed Node Red and Mosquitto (MQTT) you should now be able to open the Node Red workflow editor in any browser
Point your browser to: http://<ip_address of node red>:1880.
Now you should see see the workbench in the middle, at the left the library, with all installed nodes and at the right the info/debug pane,

Now it is time to play a little with node red, in order to get acquainted with this tool.
A good starting point is: https://nodered.org/docs/
Also http://noderedguide.com/ may help you a lot.

With

Connect the (default) Node Red MQTT in node to domoticz/out and connect a debug node and chech that you can see the topics published by Domoticz

I mean that you should make your first small flow.
As follows:

1. Drag and drop the MQTT in Node on the workbench

Screenshot_Node Red MQTT in.png (9.99 KiB) Viewed 3628 times

Configure this node as follows:

Double Click on the node in the workbench
You may give the server a name, e.g. MQTT broker
Enter your topic, you want to subscribe to. In this case domoticz/out
Leave the output as auto-detect
QoS you can leave
Name is optional and up to you to decide
Click on the pencil at the right side of the server name
At Connection you fill in the IP address of the broker and the port stays default (1883).
Tick also the checkbox, called "Use legacy MQTT 3.1 support"
If you have secured your broker (as advised) with a username/password fill in the fields in the "Security" Tab
The rest you can leave for this moment.
Deploy (the red button in the right top corner)

If you did everything correct you will see below the MQTT node a green dot with connected.

2. The next thing we do is that we drag and drop from the output section the debug node.
You may double click on it and give it a name but that is optional

3. Draw a line from the output of the MQTT In node to the debug node, so that you have the following flow:

Screenshot_Node Red MQTT debug.png (6.69 KiB) Viewed 3628 times

Do not forget to press Deploy

3. If you go now to the debug pane
Press Ctrl+ g and the d or click on the debug symbol, right beside info.
If everything is succesfull you will see debug output from domoticz.

Screenshot_Node Red debug output.png (41.33 KiB) Viewed 3628 times

If you can see output from Domoticz you have checked that both Domoticz and Node Red can communicate with your broker.

But you have verified already a couple of things, so maybe you find it already yourself.

I come back to AP Systems issue later.

Regards

[FireWizard]

Hello Bernhard,

If all the tools are working, is the next step to get data from the AP systems ECU.

In my opinion you have 3 options.

1. An api directly from your local ECU unit (preferred)
2. An api through the portal of APSystems
3. Connection using PVoutput

Ad 1.
If I do a quick scan in the manual, I see that you should be able to access your ECU on your local IP adress.
Questions
1. Is that possible?
2. Do you need username and/or password?
3. According to the manual your ECU seems to have the following default address:
IP Address: 192.168.131.228
Subnet Mask: 255.0.0.0
4. If this is possible, can you try to logon, but add .json behind it.
5. Do you have access on the standard port 80, if http, or 443., if https?
6. Can you provide a screen shot if you are logged in with your browser?

Ad 2.
I see in the manual that you have a so called EMA account.
I assume it is username/password protected.
Can you confirm how the authentication is done (not providing your personal details!)
The login page you can access at: https://api.apsystemsema.com/ema/index.action

The link https://medium.com/@rukmalf/extracting- ... 2b8e8942b6 indicates, that you easily extract the data by the command:
curl -H "Content-Type: application/x-www-form-urlencoded" http://api.apsystemsema.com:8073/apsema ... tPowerInfo -d ecuId=<ECU ID> -d filter=power -d date=<date> -o <date>.json.

The first part is exactly the same as login page. So I do not believe that it gives access directly to ECU.

Ad 3.

Also the python script at https://github.com/willemstoker/aps-to- ... voutput.py
uses http://api.apsystemsema.com:8073/apsema ... tPowerInfo.
That link does not work (method not allowed) and I doubt if that script works.

@krizzz. Does the script and your solution work?

The curl command above may work, but you need more info, such as the ECU Id and a date, but you can try.

Summing up.

I think that the first method is worthwhile to focus on, because option 2 and 3 uses the AP Systems portal and you want to extract the data from your ECU directly.

Looked at the script from openenergy but I read I need "EmonPI" no idea what it is.

See: https://wiki.openenergymonitor.org/index.php/EmonPi

Did some searching again and run into this thread

Yes, I have seen this thread, but it doesn't give the crucial information.
I saw also the script at: https://gist.github.com/anonymous/c460d ... c69fd6a1ee.
It is the same as: https://community.openenergymonitor.org ... rters/3491

We should focus on it.

Before you can use the script you have to install an additional node in Node Red, EmonCMS as follows

Go to the menu at the top right (3 bars) and click Manage palette.
Go to the Install tab and search for EmonCMS.

Screenshot_node emoncms.png (19.72 KiB) Viewed 3628 times

If you cannot succeed on this road, you can try to scrape the website.
Another step is to use Wireshark and to capture the data between the ECU and the AP Systems portal.

For now.

Regards

[Benneton]

Hi,

Thank you very much.

You said also

Can receive MQTT messages from Domoticz when my broker subscribes to domoticz/out

I wonder, how did you check that?

I subscribed MQTT.fx to the domoticz/out and checked by sending various messages around with Mosquito.

Yes I managed to open Node Red web interface but had no clue what to do. your instructions are clear on this.

I can see debug messages in Node-red so communication between Domoticz / broker and broker / node-red is working.

Next step is to see what I can do with part 2 of you support.
Thank you so far for the support.
Bernard

[Benneton]

Next step most likely the most complicated one ..

Ad 1.
If I do a quick scan in the manual, I see that you should be able to access your ECU on your local IP adress.
Questions
1. Is that possible?
2. Do you need username and/or password?
3. According to the manual your ECU seems to have the following default address:
IP Address: 192.168.131.228
Subnet Mask: 255.0.0.0
4. If this is possible, can you try to logon, but add .json behind it.
5. Do you have access on the standard port 80, if http, or 443., if https?
6. Can you provide a screen shot if you are logged in with your browser?

1. yes I can connect to the web interface of my ECU-3, there I can see performance and history of generates power.
2. No not needed
3. Correct but changed this to fit my needs of local LAN IP addressing scheme.
4. I can just access the the IP in my case 192.168.2.88

When I navigate from the landing page to realtime data I get directed to:
http://192.168.2.88/index.php/realtimedata/power_graph

Tried with JSON behind it but error 404.
Also tried to telnet into the device, get response on port 23 but need an account, searched internet but could not find username/password.
admin/admin etc. did not work.
5. standard port 80, https://192.168.2.88/ page cannot be loaded.
6. yes see above with point 4.

URL structure:
192.168.2.88 --> landing page
http://192.168.2.88/index.php/realtimedata --> realtime landing page
http://192.168.2.88/index.php/realtimedata/power_graph --> power section
http://192.168.2.88/index.php/realtimedata/energy_graph --> Energy section
http://192.168.2.88/index.php/management --> management
http://192.168.2.88/index.php/managemen ... tercurrent
http://192.168.2.88/index.php/management/datetime
http://192.168.2.88/index.php/management/language
http://192.168.2.88/index.php/management/network
http://192.168.2.88/index.php/management/wlan
http://192.168.2.88/index.php/management/upgrade_ecu

I saved the code of the http://192.168.2.88/index.php/realtimedata page in a text file:
https://mbernard.home.xs4all.nl/youless ... edata..txt
Maybe this helps in understanding where data is stored.

Ad 2. I do not have an EMA account, maybe I need to register, need to check.

Will look into the other options / scripts later this week. again thank you very much for this!

Bernard

[FireWizard]

Hello Bernhard,

Thanks for a lot of information.
I assume that you use a Raspberry Pi, with Raspbian Stretch.
Can you confirm.

I suggest you should follow the way as described in: https://community.openenergymonitor.org ... rters/3491

We only follow the first two steps.

1. Download the attached ZIP file, edit getECUData.py to include your ECU IP address and then place the getECUData.py file into /home/pi/ directory.
2. Import the NodeRED flow…

As the script is a Python 3 script we have to install pip3, as this is not installed by default.

Follow these steps:

1. Open a terminal on your pi with ssh.
2. Execute the command: python3 -V.
It should return Python 3.5.3.
3. Execute the command: sudo apt-get install python3-pip.
This command installs pip3.
4. To check if it has been installed correctly execute the following command: pip3 --version.
It should give something like: pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.5).
Do not upgrade, as this gives a problem with a module 'main'.

Next step is to download the ZIP file from: https://community.openenergymonitor.org ... rters/3491
to your Desktop/Laptop PC.
1. Unpack the ZIP file.
2. Select all ( Ctrl+A) and copy (Ctrl +C).
3. In the terminal of the RPi give the following command: nano getECUData.py.
This will open an empty editor with that name.
4. Paste the clipboard file into the editor ( Ctrl+Shift+V).
5. Edit the following line: ecuIP = "192.168.10.29" and insert your IP address of the ECU.
6. Save the file with Ctrl+X and answer with Y If you are in your home directory it is saved in /home/pi .

Before you can use it you have to install a python3 module (bs4).
See the following line in the file: ### needs the bs4 libary, so "$ pip install bs4".
However this is a python3 script, so you should use: pip3 install bs4 (pip3 instead of pip).

The next step is to copy the Node Red flow (Select and Ctrl+C).

1. Open the Node REd editor.
2. Open the menu in right upper corner and select import.
3. Select clipboard.
4. Paste the file (Node Red flow) and click import.

You will see the following:

Screenshot_APS inverter flow.png (11.47 KiB) Viewed 3623 times

You see at the right you have an unknown node: emoncms. That is because the emoncms node has not been installed and we probably don't need it.

The last step is to remove that unknown node and insert a debug node.
See the picture below.

Screenshot_APS inverter flow2.png (10.48 KiB) Viewed 3623 times

During the development process I suggest to disable the 5 minute time "tick". It is disturbing and gives an error (in my case, because it cannot connect).
Double click the inject node and change "Repeat" to none.
You can reactivate or change that later.

If everything is installed, it's time to activate the inject node and to watch the output in the debug node.

If we know, that there is output and how the output looks like, we can continue.

So far,

Regards

[Benneton]

Hi FireWizard,

Again thanks for your patience with a programing noob.

Indeed I am running this on Raspberry PI 3.
Raspbian Stretch indeed:


Okay, this is what I achieved:
1. Installed pip3 - Oke
2. Installed getECUData.py script with correct IP address in /home/pi - Oke

3. installed pip3 - Oke
4. created Node Red flow and set 5 min tick of and replace emoncms with debug node

5. activated ECU_node - fail


To check the getECUData.py script I did run it from the command line:
$ Python3 getECUData.py

This returned many errors:
pi@BennetonsPI:~ $ python3 getECUData.py
Traceback (most recent call last):
File "getECUData.py", line 16, in <module>
table = BeautifulSoup(urllib.request.urlopen(url),"html.parser").find("table")
File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.5/urllib/request.py", line 472, in open
response = meth(req, response)
File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.5/urllib/request.py", line 510, in error
return self._call_chain(*args)
File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/usr/lib/python3.5/urllib/request.py", line 590, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

From the chocolate I can make of it, I believe the ECU is not returning valid (or any at all) responses.

Hope you understand better what is going on and point me yet again in the right direction.

Your support is much appreciated.

Regards,
Bernard

[FireWizard]

Hi Bernhard,

So 2 problems.

1. Deploy failed: No response from server.
This has nothing to do with the flow, but with the communication between your client and the Node-Red server.
The first step you should do is to restart Node Red as follows:
- Log in with ssh to the terminal.
- Execute to following command: node-red-stop.
- Execute to following command: node-red-start.
Then check your log with the command node-red-log.

it should be possible to deploy the flow. but dependent on the server version I red about some issues with specific versions.
What is your version of Node Red? (Open menu and then the last line)

2. A lot of errors.
You did not mentioned it, but did you install bs4, because it complains with reference to BeautifulSoup.

Before you can use it you have to install a python3 module (bs4).
See the following line in the file: ### needs the bs4 libary, so "$ pip install bs4".
However this is a python3 script, so you should use: pip3 install bs4 (pip3 instead of pip).

FYI; From tomorrow I'm not so frequently online, because of holidays.

Regards

[Benneton]

Hi,

After restart I was able to deploy the flow in node red.
Version node-red v0.20.7
Deployed but not green, with red square Error 1.
9/2/2019, 10:24:14 PMnode: 2d675f41.7765d
msg.payload : string[0]
""

I did install bs4, checked my console and found it:


When I check the console window where node-red is running I see when flow is activated the following errors.
2 Sep 22:35:01 - [info] [exec:APS Inverter Data] error:Error: Command failed: python3 /home/pi/getECUData.py 1567456500376
Traceback (most recent call last):
File "/home/pi/getECUData.py", line 16, in <module>
table = BeautifulSoup(urllib.request.urlopen(url),"html.parser").find("table")
File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.5/urllib/request.py", line 472, in open
response = meth(req, response)
File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.5/urllib/request.py", line 510, in error
return self._call_chain(*args)
File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/usr/lib/python3.5/urllib/request.py", line 590, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

2 Sep 22:35:18 - [info] Stopping flows
2 Sep 22:35:18 - [info] Stopped flows
2 Sep 22:35:18 - [info] Starting flows
2 Sep 22:35:18 - [info] Started flows
2 Sep 22:35:20 - [info] [exec:APS Inverter Data] error:Error: Command failed: python3 /home/pi/getECUData.py 1567456519076
Traceback (most recent call last):
File "/home/pi/getECUData.py", line 16, in <module>
table = BeautifulSoup(urllib.request.urlopen(url),"html.parser").find("table")
File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.5/urllib/request.py", line 472, in open
response = meth(req, response)
File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.5/urllib/request.py", line 510, in error
return self._call_chain(*args)
File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/usr/lib/python3.5/urllib/request.py", line 590, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

I must be missing something or the version of node-red maybe the issue...

Yet again a big thank you!
Do have a lovely holiday! Enjoy it.

Bernard

[FireWizard]

Hi Bernhard,

I do not think that the problem is Node Red or the version.
The "execute" node in the flow simply calls python and the script.
You can check the error1 message in the console by issuing the command node-red-log.

If you executie the python command from the commandline, you will see the same.
So it is the python file.
The problem is that I'm not a python expert and also can' t try and test.
But I think about it.

Regards

[Benneton]

Hi FireWizard,

Looked further into what goes wrong, found another reason why it could not work.
The getECUData.py is written to scrape data from ECU device running version 3.10.10.
My ECU is running 4.1. looking at the errors when running the script I see,
"File "/home/pi/getECUData.py", line 16, in <module>
table = BeautifulSoup(urllib.request.urlopen(url),"html.parser").find("table")"

(url) is the link to the location on the ECU device
- /cgi-bin/parameters?target=main
- /cgi-bin/

When I open the link in my browser I get error 404 not found.

Link for realtime data in v4.1 is:
- /index.php/realtimedata

My guess is that per version 4.x the structure has changed.

I changed the URL part of the script and run it. Get different errors now.

3 Sep 15:51:54 - [info] [exec:APS Inverter Data] error:Error: Command failed: python3 /home/pi/getECUData.py
Traceback (most recent call last):
File "/home/pi/getECUData.py", line 23, in <module>
print(str(counter) + " Serial: " + row.contents[0].get_text().split("\xa0")[0] + ",")
File "/home/pi/.local/lib/python3.5/site-packages/bs4/element.py", line 646, in __getattr__
self.__class__.__name__, attr))
AttributeError: 'NavigableString' object has no attribute 'get_text'

That is to my simple knowledge the result of the table / HTML page layout change.

Version 3.10.10 (according to https://community.openenergymonitor.org ... how/3469/9 )
<center>
<tr>
<td align=center>Inverter ID</td>
<td align=center>Current Power</td>
<td align=center>Grid Frequency</td>
<td align=center>Grid Voltage</td>
<td align=center>Temperature</td>
<td align=center>Date</td>
</tr>
</center>

Version 4.1:
<table class="table table-condensed table-bordered">
<thead>
<tr>
<th scope="col">Inverter ID</th>
<th scope="col">Current Power</th>
<th scope="col">Grid Frequency</th>
<th scope="col">Grid Voltage</th>
<th scope="col">Temperature</th>
<th scope="col">Reporting Time</th>
</tr>
</thead>
<tbody>
<div>
<tr class='active'>
<td>123456789012-A </td>
<td> 23 W </td>
<td rowspan=2 style='vertical-align: middle;'> 50.0 Hz </td>
<td> 240 V </td>
<td rowspan=2 style='vertical-align: middle;'> 31 &#176;C </td>
<td rowspan=2 style='vertical-align: middle;'> 2019-09-03 15:46:05
</td>
</tr>

Did a quick google to see if I could find any reference to a newer version of the script, no luck so far.
Asked Dave if he has an update of the script for scraping version 4.1.
Let's see that he comes back with.
In the meantime I will look into ways to get the script updated.

Will keep you all posted.

Bernard

[FireWizard]

Hi Bernard,

I had seen that link.
Yesterday I already thought that the script was too old (2 years) and that it was not suitable for the current version. If a newer version is not available, I think you should stop that route.

Next option is to scrape the data in Node-Red.

You published already valuable information. Do you know what the difference is between A and B for the Id for the same inverters?
Do you think that you can get such a page for all your 8 inverters?

Regards

[Benneton]

Hi FireWizard,

It is one page with all, in my case, 6 inverters, each inverter has an option to connect 2 panels. It is 2 micro inverters in 1 house.
A / B is to differentiate between the panels connected

I just grabbed a fresh rendered page from realtime window:


via the link below you can download the code of the page.
https://mbernard.home.xs4all.nl/youless ... medata.htm

Kind regards
Bernard

[FireWizard]

Hello Bernhard,

I think it is posssible to scrape the data from that webpage.

Just to verify:

- you have 6 inverters with 12 solarpanels connected.
- you get from each solar panel the produced power in watt.
- you get from each solar panel the grid voltage.
- you get from each inverter the frequency in Hz.
- you get from each inverter the temperature in degrees Celcius.

I think you can delete the getECUData.py script file on your Rasberry Pi and you can delete the flow from your Node Red. You don' t need it anymore.

I will focus on the data scraping from now on.
What do you want to present in Domoticz?
I assume the produced power for each solar panel, the grid voltage for each panel and the frequency and temperature for each inverter.
What about date/time?

Regards

[Benneton]

That is positive news

Just to verify:

- you have 6 inverters with 12 solarpanels connected. - Correct
- you get from each solar panel the produced power in watt. Correct
- you get from each solar panel the grid voltage. Correct
- you get from each inverter the frequency in Hz. Correct
- you get from each inverter the temperature in degrees Celcius.Correct

Cleaned up my system.

It would be great to have per Panel:
- power
- voltage
- frequency
- temperature

This then with time/date stamp so that Domoticz can create graph and historical overview, like we have i.e. for temperature.

Not sure if all is possible so what ever can be done is nice and much better no info in Domoticz at all

Thanks!

[FireWizard]

Hi Bernhard,

On the web page, you showed, I see only the produced power and grid voltage per panel. Temperature and frequency are shown as per inverter, so unless you have another page, it is not possible to get this info per solar panel.

I also read that the data is updated every 5 minutes. Is that correct?

I suggest the following flow:

- inject node with 5 minute tick (can be faster, up to you), followed by timer node.
- timer node (moment), which will pass the flow (5 min. tick) between 30 minutes before sunrise until 30 minutes after sunset). It is useless to poll the inverters at night.
- next node will be the http request node, to poll the (local) url.
- html node to sort the received data.
- function node to prepare the data for domoticz.
- the last node will be mqtt-out to domoticz/in.

If we have the data in domoticz, we will have the graphs in domoticz automatically.

For the moment I wiil use the provided link as url, but we have to change that later.

Regards

[Benneton]

Hi FireWizard,

When this is not possible to derive from the inverter to panel then we have:
Power and Voltage.

Did check and indeed update is every 5 min.

- inject node with 5 minute tick (can be faster, up to you), followed by timer node.
- timer node (moment), which will pass the flow (5 min. tick) between 30 minutes before sunrise until 30 minutes after sunset). It is useless to poll the inverters at night.
- next node will be the http request node, to poll the (local) url.
- html node to sort the received data.
- function node to prepare the data for domoticz.
- the last node will be mqtt-out to domoticz/in.

So with above 5 steps the data is passed on the MQTT broker and can be picked up by Domoticz.
In Domoticz we have to create the 12 panels (in my case) and link to the MQTT feed.

Thanks for your support!
Bernard

[FireWizard]

Hi Bernhard,

Thanks for the info.
You can wait with creating the devices.
I think we will need:

- 12 devices for the power of each panel
- 12 for the voltage
- 6 for the temperature of each inverter
- 6 for the frequency.

You can start thinking about a nice drawing for your solar panels.

Creating the flow will take time, as I do not have easy access to my Node Red server and editting on a tablet is not that easy.

Regarda

[Benneton]

Hi FireWizard,

Please do take it easy, working on tablet is indeed difficult, I do not get further then reading websites and the odd game.
So enjoy your holiday, no rush!

I will think on drawing for Panel and see if I can find or make one, most likely the first.

Bernard

[dheuts]

First of all, sorry for my late reply. I was on vacation.

Using the option Krizzz mentioned before with PVoutput.org
It's working, thanks! But there is only 1 problem. If the output is 0, the script will just use the latest updated value.
So you still don't find out there is a problem in Domoticz. But PVoutput has a notification email option, so it's working.

Would be great if you could grab the info FireWizard !

[FireWizard]

Hi dheuts,

I think it should be possible. Like you, I'm still on vacation.
So no progress this week.

Regards