Page 1 of 2

General usage lost values

Posted: Tuesday 16 July 2024 20:17
by sailmich
Hello,
I'm using 2024.7 stable, installation with docker on a Rpi3.
I have a small photo voltaic on my roof. For nearly two years I put the values from a shelly plug into a dummy device general usage to monitor it. The values are computed not from device.
Bildschirmfoto vom 2024-07-16 19-50-13.png
Bildschirmfoto vom 2024-07-16 19-50-13.png (34.43 KiB) Viewed 1548 times
Yesterday in accident I changed from "computed" to "From device" after that all the values are gone. Even after changing back to "computed" it's still blank.Also the log is empty. I noticed that issue also in former versions of domoticz.
Bildschirmfoto vom 2024-07-16 19-49-16.png
Bildschirmfoto vom 2024-07-16 19-49-16.png (13.69 KiB) Viewed 1548 times
Bildschirmfoto vom 2024-07-16 19-49-40.png
Bildschirmfoto vom 2024-07-16 19-49-40.png (138.54 KiB) Viewed 1548 times

How can I rescue my data?

Re: General usage lost values

Posted: Tuesday 16 July 2024 21:09
by habahabahaba
in home directory there is backups folder with hourly-daily-monthly backups of database.

Re: General usage lost values

Posted: Wednesday 17 July 2024 6:39
by sailmich
I asked the wrong question. How can I get it back to work and rescue my data? And is it a bug?

Re: General usage lost values

Posted: Sunday 21 July 2024 7:12
by sailmich
I'm using a dummy device electricity(actual and counter). The value for actual is coming from node red. When set Energy read to "from device" without having a device support it, this will crash the dummy device like I described above. When switched back the energy read to "computed" all stays blank.
I would say this is a bug.

Re: General usage lost values

Posted: Sunday 21 July 2024 10:13
by PieterS
It is not related to latest version of Domoticz. It is there for years.

In the dashboard shows error NaN and no change to correct it. Only create a new device and start all over again without history is only solution. Why it happens? No idea.

In the log:

Code: Select all

Error converting sValue/sUsage
Indeed, the data is feeded from Node-Red and sometimes it works for weeks. Sometimes it is corrupt after a few days. No modifications in flow. I got about crazy and do not create a new device any more.

IF it is error of NR I could accept that. But it should be nice if there is a check in Domoticz towards the sort data ( integer/string) so that in case of an error the device does not get corrupt. Now it is not usable after that failure.

Re: General usage lost values

Posted: Sunday 21 July 2024 11:59
by FireWizard
Hi,

@PieterS,

You wrote:
In the dashboard shows error NaN and no change to correct it.
NaN stands for Not a Number. So obviously you have received in Domoticz a wrong value.
And as you use Node-RED, it is probably that you send the wrong data.

I use a number of such devices (Electrical, actual and counter) from Node-RED myself for a number of years and I never noticed such issues.
Remember that with MQTT or HTTP you write directly into the database and wrong values may cause corruption of your database.

@PieterS and @sailmich

Can you show us the data string, you send from Node-RED by MQTT or HTTP to Domoticz?

Regards

Re: General usage lost values

Posted: Sunday 21 July 2024 13:03
by PieterS
@Firewizard,

I know about NaN. :x
You asked:
Can you show us the data string, you send from Node-RED by MQTT or HTTP to Domoticz?
I use MQTT as transmission method.

Code: Select all

msg1.payload = { "command": "udevice", "idx": 2577, "nvalue": 0, "svalue": total_power + ";" + total_energy };
It's a pitty that one wrong value ruins the history of that device.. Otherwise there is need for extra check in NR before sending the data.

I will send you the flow in a PB. The cause could be somewhere else in the flow.

Re: General usage lost values

Posted: Sunday 21 July 2024 18:21
by sailmich
@Firewizard I also use mqtt.

Code: Select all

msg.payload = { "command": "udevice", "idx": 636, "nvalue": 0, "svalue":msg.payload.toString()};
return msg;
But I've never had problems, it is working for more than one and a half year. Just because I hit the wrong button it messed up.

Re: General usage lost values

Posted: Sunday 21 July 2024 18:45
by sailmich
My code is just sending the power, so no problems with wrong values sending from Node red. The problem seems to be in domoticz when receiving wrong or no values.

Re: General usage lost values

Posted: Sunday 21 July 2024 21:55
by FireWizard
Hi @sailmich.

I see you use as virtual device Electric (instant + counter)

This virtual device requires two values.
In spite of the fact, that you set the configuration to "Computed" and not "From device", you need to send the cumulative usage.

See: https://piandmore.wordpress.com/tag/domoticz/
Electric

The electric sensor will show the current usage of electricity in Watt and the cumulative usage in kWh. You can add digits after the decimal if needed.

{"command":"udevice", "idx":1234, "svalue":"us;cu"}

where us is usage in Watt and cu is cumulative usage in Wh (even though it is displayed in kWh)
I think you should replace:

Code: Select all

msg.payload = { "command": "udevice", "idx": 636, "nvalue": 0, "svalue":msg.payload.toString()};
return msg;
with

Code: Select all

msg.payload = { "command": "udevice", "idx": 636, "nvalue": 0, "svalue":(msg.payload.toString() + ";0")};
return msg;
I do not know, why it has been messed up, but as you said:
The problem seems to be in domoticz when receiving wrong or no values.
This is exactly, what you were doing. So perhaps it will solve your issue


Regards

Re: General usage lost values

Posted: Monday 22 July 2024 4:53
by habahabahaba
Thats all written

Re: General usage lost values

Posted: Monday 22 July 2024 22:35
by sailmich
@FireWizard I'm pretty sure your solution will work.
Even If it's written, one false code shouldn't destroy the whole device.
Thank's for giving explanation.

Re: General usage lost values

Posted: Tuesday 23 July 2024 11:41
by FlyingDomotic
A possible explanation is a "bad" value stored into Domoticz database. If it's the case, removing this bad value or changing it to something right could be the solution.

To make the diagnostic, you have to :
  • Backup current Domoticz database on your PC (using "Setup" -> "Settings", clicking on "Backup/restore" tab and then "Backup". You'll get Domoticz.db file on your download folder.
  • Open domoticz.db with the SQL lite data browser
  • Find the IDX of your device in Domoticz, clicking on "Edit" on its widget, and getting "IDX" on top of page (here, probably 636)
  • Looking for values into short term table and long term table to find any discontinuity/error on lists
Short and long tables names depends on device type.

By default, you can try "Meter" for short term table (last hours or so) and "Meter_Calendar" for long term (daily data since origin).

Other tables are "MultiMeter", "Percentage", "Rain", "Temperature", "UV" and "Wind" for short term storage. Long term name is composed of short term table name followed by "_Calendar". As you may imagine, this depends on device type.

To reduce displayed data, select only "DeviceRowId" column with the matching IDX.

In you case, as data from few day ago, looking to MeterCalendard table should be sufficient.

If there's some strange data in "Value" column, then we may try to fix it. If table is only starting after the "incident" date, it may means that Domoticz erased data, and recovery would be different (something like extracting it from backup database and reloading it).

In all cases, having a look to a database copy is safe (as described). Changing "production" database value may not be...

Re: General usage lost values

Posted: Thursday 25 July 2024 20:25
by sailmich
@FlyingDomotic thank you very much for your help! But I would spent to much time to find the bad data. I stay with the crashed counter and made a new one with sending the correct values according to the manual.
@FireWizard I took a look into shellies API and found that the plug is also sending energy :) I got a solution with http but would like to do it with mqtt. I still don`t now how to get booth values together and sent it via mqtt. May you help me.
If crashing the dummy device by false values isn't a bug, could someone please move the tread.
Thanks
Spoiler: show

Code: Select all

[
    {
        "id": "b4ece2bd784ad56a",
        "type": "join",
        "z": "449126130c89dd8a",
        "name": "",
        "mode": "custom",
        "build": "object",
        "property": "payload",
        "propertyType": "msg",
        "key": "topic",
        "joiner": "\\n",
        "joinerType": "str",
        "accumulate": false,
        "timeout": "",
        "count": "2",
        "reduceRight": false,
        "reduceExp": "",
        "reduceInit": "",
        "reduceInitType": "",
        "reduceFixup": "",
        "x": 870,
        "y": 1220,
        "wires": [
            [
                "8c791a5f7d32afa6"
            ]
        ]
    },
    {
        "id": "b62a35e1c4344295",
        "type": "change",
        "z": "449126130c89dd8a",
        "name": "get power",
        "rules": [
            {
                "t": "set",
                "p": "topic",
                "pt": "msg",
                "to": "Watt_Erzeugung",
                "tot": "str"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 380,
        "y": 1200,
        "wires": [
            [
                "b4ece2bd784ad56a"
            ]
        ]
    },
    {
        "id": "9b53bfdac90aed76",
        "type": "change",
        "z": "449126130c89dd8a",
        "name": "get energy",
        "rules": [
            {
                "t": "set",
                "p": "topic",
                "pt": "msg",
                "to": "kWh_Erzeugung",
                "tot": "str"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 390,
        "y": 1260,
        "wires": [
            [
                "b4ece2bd784ad56a"
            ]
        ]
    },
    {
        "id": "8c791a5f7d32afa6",
        "type": "http request",
        "z": "449126130c89dd8a",
        "name": "set SUW variable",
        "method": "POST",
        "ret": "txt",
        "paytoqs": "ignore",
        "url": "http://xxxxxxxxxxxxx/json.htm?type=command&param=udevice&idx=785&nvalue=0&svalue={{payload.Watt_Erzeugung}};{{payload.kWh_Erzeugung}}",
        "tls": "",
        "persist": false,
        "proxy": "",
        "insecureHTTPParser": false,
        "authType": "",
        "senderr": false,
        "headers": [],
        "x": 1090,
        "y": 1220,
        "wires": [
            []
        ]
    }
]

Re: General usage lost values

Posted: Thursday 25 July 2024 21:40
by FireWizard
Good evening @sailmich

You wrote:
I took a look into shellies API and found that the plug is also sending energy
That is correct, but it depends on the type of plug (generation 1 or generation 2), how to format the request.
it can be done and then it is a simple conversion.

Can you tell me, what the type of your plug is?. Is it a Shelly Plug or PlugS (Generation 1) or is it a Shelly Plus Plug S (Generation 2)
keep in mind that the value you receive from the Shelly for "Energy" is expressed in Wm (Watt-minute) So 1 Wm = 1/60 Wh
I got a solution with http but would like to do it with mqtt.
That is possible. I created something quick with an "Inject" node, (as I do not know the Generation of the Shelly yet)
See the flow below in order to give you an idea.

Code: Select all

[
    {
        "id": "de9d282626577580",
        "type": "inject",
        "z": "919eeef6718077fa",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "Watt_Erzeugung",
        "payload": "123",
        "payloadType": "num",
        "x": 320,
        "y": 340,
        "wires": [
            [
                "92de407702b6a1c7"
            ]
        ]
    },
    {
        "id": "16fc407458de7947",
        "type": "inject",
        "z": "919eeef6718077fa",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "kWh_Erzeugung",
        "payload": "600",
        "payloadType": "num",
        "x": 320,
        "y": 400,
        "wires": [
            [
                "92de407702b6a1c7"
            ]
        ]
    },
    {
        "id": "92de407702b6a1c7",
        "type": "join",
        "z": "919eeef6718077fa",
        "name": "Join Watt and Watt-minute",
        "mode": "custom",
        "build": "object",
        "property": "payload",
        "propertyType": "msg",
        "key": "topic",
        "joiner": "\\n",
        "joinerType": "str",
        "useparts": false,
        "accumulate": true,
        "timeout": "",
        "count": "2",
        "reduceRight": false,
        "reduceExp": "",
        "reduceInit": "",
        "reduceInitType": "",
        "reduceFixup": "",
        "x": 610,
        "y": 360,
        "wires": [
            [
                "48ea9677282ac733",
                "7102f68aedf7232e"
            ]
        ]
    },
    {
        "id": "48ea9677282ac733",
        "type": "debug",
        "z": "919eeef6718077fa",
        "name": "debug 1",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 880,
        "y": 440,
        "wires": []
    },
    {
        "id": "7102f68aedf7232e",
        "type": "function",
        "z": "919eeef6718077fa",
        "name": "function 1",
        "func": "msg.payload = { \"command\": \"udevice\", \"idx\": 528, \"svalue\": msg.payload.Watt_Erzeugung.toString() + \";\" + (msg.payload.kWh_Erzeugung / 60).toString() };\nreturn msg;",
        "outputs": 1,
        "timeout": 0,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 860,
        "y": 300,
        "wires": [
            [
                "a66757038f23b540"
            ]
        ]
    },
    {
        "id": "a66757038f23b540",
        "type": "debug",
        "z": "919eeef6718077fa",
        "name": "debug 2",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 1040,
        "y": 300,
        "wires": []
    }
]
If crashing the dummy device by false values isn't a bug, could someone please move the tread.
I do not see it as a bug, but injecting wrong or no values in the Domoticz database is not a good idea.
But I agrre it would be better, that it would reject the value.
Personally I have never been able to crash Domoticz by injecting something wrong.

So to send it by MQTT, you need

1. Configure your Plug, Plug/S Plug Plus to use MQTT
2. Use to MQTT Explorer to check the communication
3. Let me know the type of plug.

Have a look at the example.

Regards

Re: General usage lost values

Posted: Friday 26 July 2024 17:05
by PieterS
sailmich wrote: Monday 22 July 2024 22:35 @FireWizard I'm pretty sure your solution will work.
Even If it's written, one false code shouldn't destroy the whole device.
I agree that the concept along that device in Domoticz is not failproof. But we have to deal with that...
Two solutions:
1. Ask an enhancement to the team of Domoticz
2. Build a check along NaN in the flow in Node-Red before sending data.

@Firewizard gave me ultimate help with that last option. Thnx and all credits to him.

Re: General usage lost values

Posted: Friday 26 July 2024 18:04
by sailmich
Hello @FireWizard I have a plug S, with your function node, all is working well. Thank you very much! I had to open the plug s on my fritz box, without internet access it just sending the power. I will compare "computed and from device" for a while, when diference isn't big I will go back to computed.

@PieterS the solution for NaN would be interesting for me and maybe others too, may you post it?

Cheers

Re: General usage lost values

Posted: Friday 26 July 2024 20:38
by PieterS
I understand. I will do that next week. Not easy to reach the flow at this moment. Hotspot, laptop, vpn and the right mood.. yes, it's possible. 8-)

Maybe is @Firewizard able to do that earlier.

Re: General usage lost values

Posted: Friday 26 July 2024 21:10
by FireWizard
I will try tomorrow and will also add a brief explanation.

Re: General usage lost values

Posted: Saturday 27 July 2024 19:08
by FireWizard
Hi,

As promised I will show a Node-RED example, in order to make to data flow to Domoticz more robust.

I see in several forum posts discussions about the fact that Domoticz cannot handle inputs with errors
It will always be a point of discussion, who is responsible that an application will not crash.

By request I will show a quick solution, which I made for @PieterS

He has 4 different Solar inverters, a Goodwe, Omnik, Growat and a SMA inverter. All inverters can communicate their own way with Node_RED.
In order to display the total produced power and energy, we have to add these values.
It might happen that one ore more values are not (yet) available.

In this case, every inverter communicate with Node-RED and the value of "Power" and "Energy" are stored in a so called "flow variable".
To avoid that some value is not available and the variable will contain undefined we have to take care that you will retrieve always a value.
If the value is not available, we use 0.

Code: Select all

let growatt_power = flow.get("growatt_power") || 0;
let sma_power = flow.get("sma_power") || 0;
let goodwe_power = flow.get("goodwe_power") || 0;
let omnik_power = flow.get("omnik_power") || 0;

let growatt_energy = flow.get("growatt_energy") || 0;
let sma_energy = flow.get("sma_energy") || 0;
let goodwe_etoday = flow.get("goodwe_etoday") || 0;
let omnik_etoday = flow.get("omnik_etoday") || 0;
So the value is either the valu from the "flow variable" or 0.

Normally this will do, but to avoid that a value is indicated as NaN (Not a Number) we will test both power and energy for NaN.

Code: Select all

if (isNaN(total_energy) || isNaN(total_power)) {
 return null // halt flow
};
If, either total_energy or total_power are equal to NaN the flow will stop and not push a NaN to Domoticz.

Total flow, please find below.

Code: Select all

let msg1 = {};
let msg2 = {};
let msg3 = {};

let growatt_power = flow.get("growatt_power") || 0;
let sma_power = flow.get("sma_power") || 0;
let goodwe_power = flow.get("goodwe_power") || 0;
let omnik_power = flow.get("omnik_power") || 0;

let growatt_energy = flow.get("growatt_energy") || 0;
let sma_energy = flow.get("sma_energy") || 0;
let goodwe_etoday = flow.get("goodwe_etoday") || 0;
let omnik_etoday = flow.get("omnik_etoday") || 0;

let total_energy = (growatt_energy * 100) + (sma_energy * 1000) + (goodwe_etoday * 1000) + (omnik_etoday * 1000); //v1
let total_power = (growatt_power / 10) + sma_power + goodwe_power + omnik_power; //v2

if (isNaN(total_energy) || isNaN(total_power)) {
 return null // halt flow
};

msg1.payload = { "command": "udevice", "idx": 2578, "nvalue": 0, "svalue": total_power.toString() + ";" + total_energy.toString() };
msg2.payload = total_energy.toString();
msg3.payload = total_power.toString();

return [msg1, msg2, msg3];
Regards