General usage lost values

Please use template to report bugs and problems. Post here your questions when not sure where else to post
Only for bugs in the Domoticz application! other problems go in different subforums!

Moderators: leecollings, remb0

Forum rules
Before posting here, make sure you are on the latest Beta or Stable version.
If you have problems related to the web gui, clear your browser cache + appcache first.

Use the following template when posting here:

Version: xxxx
Platform: xxxx
Plugin/Hardware: xxxx
Description:
.....

If you are having problems with scripts/blockly, always post the script (in a spoiler or code tag) or screenshots of your blockly

If you are replying, please do not quote images/code from the first post

Please mark your topic as Solved when the problem is solved.
sailmich
Posts: 232
Joined: Wednesday 17 February 2016 22:55
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Germany
Contact:

General usage lost values

Post by sailmich »

Hello,
I'm using 2024.7 stable, installation with docker on a Rpi3.
I have a small photo voltaic on my roof. For nearly two years I put the values from a shelly plug into a dummy device general usage to monitor it. The values are computed not from device.
Bildschirmfoto vom 2024-07-16 19-50-13.png
Bildschirmfoto vom 2024-07-16 19-50-13.png (34.43 KiB) Viewed 1027 times
Yesterday in accident I changed from "computed" to "From device" after that all the values are gone. Even after changing back to "computed" it's still blank.Also the log is empty. I noticed that issue also in former versions of domoticz.
Bildschirmfoto vom 2024-07-16 19-49-16.png
Bildschirmfoto vom 2024-07-16 19-49-16.png (13.69 KiB) Viewed 1027 times
Bildschirmfoto vom 2024-07-16 19-49-40.png
Bildschirmfoto vom 2024-07-16 19-49-40.png (138.54 KiB) Viewed 1027 times

How can I rescue my data?
User avatar
habahabahaba
Posts: 192
Joined: Saturday 18 March 2023 14:44
Target OS: Windows
Domoticz version: 2024.4
Contact:

Re: General usage lost values

Post by habahabahaba »

in home directory there is backups folder with hourly-daily-monthly backups of database.
sailmich
Posts: 232
Joined: Wednesday 17 February 2016 22:55
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Germany
Contact:

Re: General usage lost values

Post by sailmich »

I asked the wrong question. How can I get it back to work and rescue my data? And is it a bug?
sailmich
Posts: 232
Joined: Wednesday 17 February 2016 22:55
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Germany
Contact:

Re: General usage lost values

Post by sailmich »

I'm using a dummy device electricity(actual and counter). The value for actual is coming from node red. When set Energy read to "from device" without having a device support it, this will crash the dummy device like I described above. When switched back the energy read to "computed" all stays blank.
I would say this is a bug.
PieterS
Posts: 195
Joined: Wednesday 31 May 2017 16:06
Target OS: NAS (Synology & others)
Domoticz version: 2024.7
Location: NL
Contact:

Re: General usage lost values

Post by PieterS »

It is not related to latest version of Domoticz. It is there for years.

In the dashboard shows error NaN and no change to correct it. Only create a new device and start all over again without history is only solution. Why it happens? No idea.

In the log:

Code: Select all

Error converting sValue/sUsage
Indeed, the data is feeded from Node-Red and sometimes it works for weeks. Sometimes it is corrupt after a few days. No modifications in flow. I got about crazy and do not create a new device any more.

IF it is error of NR I could accept that. But it should be nice if there is a check in Domoticz towards the sort data ( integer/string) so that in case of an error the device does not get corrupt. Now it is not usable after that failure.
Synology with Domoticz build (V2024.7) in Docker
User avatar
FireWizard
Posts: 1745
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: General usage lost values

Post by FireWizard »

Hi,

@PieterS,

You wrote:
In the dashboard shows error NaN and no change to correct it.
NaN stands for Not a Number. So obviously you have received in Domoticz a wrong value.
And as you use Node-RED, it is probably that you send the wrong data.

I use a number of such devices (Electrical, actual and counter) from Node-RED myself for a number of years and I never noticed such issues.
Remember that with MQTT or HTTP you write directly into the database and wrong values may cause corruption of your database.

@PieterS and @sailmich

Can you show us the data string, you send from Node-RED by MQTT or HTTP to Domoticz?

Regards
PieterS
Posts: 195
Joined: Wednesday 31 May 2017 16:06
Target OS: NAS (Synology & others)
Domoticz version: 2024.7
Location: NL
Contact:

Re: General usage lost values

Post by PieterS »

@Firewizard,

I know about NaN. :x
You asked:
Can you show us the data string, you send from Node-RED by MQTT or HTTP to Domoticz?
I use MQTT as transmission method.

Code: Select all

msg1.payload = { "command": "udevice", "idx": 2577, "nvalue": 0, "svalue": total_power + ";" + total_energy };
It's a pitty that one wrong value ruins the history of that device.. Otherwise there is need for extra check in NR before sending the data.

I will send you the flow in a PB. The cause could be somewhere else in the flow.
Synology with Domoticz build (V2024.7) in Docker
sailmich
Posts: 232
Joined: Wednesday 17 February 2016 22:55
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Germany
Contact:

Re: General usage lost values

Post by sailmich »

@Firewizard I also use mqtt.

Code: Select all

msg.payload = { "command": "udevice", "idx": 636, "nvalue": 0, "svalue":msg.payload.toString()};
return msg;
But I've never had problems, it is working for more than one and a half year. Just because I hit the wrong button it messed up.
sailmich
Posts: 232
Joined: Wednesday 17 February 2016 22:55
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Germany
Contact:

Re: General usage lost values

Post by sailmich »

My code is just sending the power, so no problems with wrong values sending from Node red. The problem seems to be in domoticz when receiving wrong or no values.
User avatar
FireWizard
Posts: 1745
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: General usage lost values

Post by FireWizard »

Hi @sailmich.

I see you use as virtual device Electric (instant + counter)

This virtual device requires two values.
In spite of the fact, that you set the configuration to "Computed" and not "From device", you need to send the cumulative usage.

See: https://piandmore.wordpress.com/tag/domoticz/
Electric

The electric sensor will show the current usage of electricity in Watt and the cumulative usage in kWh. You can add digits after the decimal if needed.

{"command":"udevice", "idx":1234, "svalue":"us;cu"}

where us is usage in Watt and cu is cumulative usage in Wh (even though it is displayed in kWh)
I think you should replace:

Code: Select all

msg.payload = { "command": "udevice", "idx": 636, "nvalue": 0, "svalue":msg.payload.toString()};
return msg;
with

Code: Select all

msg.payload = { "command": "udevice", "idx": 636, "nvalue": 0, "svalue":(msg.payload.toString() + ";0")};
return msg;
I do not know, why it has been messed up, but as you said:
The problem seems to be in domoticz when receiving wrong or no values.
This is exactly, what you were doing. So perhaps it will solve your issue


Regards
User avatar
habahabahaba
Posts: 192
Joined: Saturday 18 March 2023 14:44
Target OS: Windows
Domoticz version: 2024.4
Contact:

Re: General usage lost values

Post by habahabahaba »

Thats all written
sailmich
Posts: 232
Joined: Wednesday 17 February 2016 22:55
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Germany
Contact:

Re: General usage lost values

Post by sailmich »

@FireWizard I'm pretty sure your solution will work.
Even If it's written, one false code shouldn't destroy the whole device.
Thank's for giving explanation.
FlyingDomotic
Posts: 303
Joined: Saturday 27 February 2016 0:30
Target OS: Raspberry Pi / ODroid
Domoticz version: 2020.2
Contact:

Re: General usage lost values

Post by FlyingDomotic »

A possible explanation is a "bad" value stored into Domoticz database. If it's the case, removing this bad value or changing it to something right could be the solution.

To make the diagnostic, you have to :
  • Backup current Domoticz database on your PC (using "Setup" -> "Settings", clicking on "Backup/restore" tab and then "Backup". You'll get Domoticz.db file on your download folder.
  • Open domoticz.db with the SQL lite data browser
  • Find the IDX of your device in Domoticz, clicking on "Edit" on its widget, and getting "IDX" on top of page (here, probably 636)
  • Looking for values into short term table and long term table to find any discontinuity/error on lists
Short and long tables names depends on device type.

By default, you can try "Meter" for short term table (last hours or so) and "Meter_Calendar" for long term (daily data since origin).

Other tables are "MultiMeter", "Percentage", "Rain", "Temperature", "UV" and "Wind" for short term storage. Long term name is composed of short term table name followed by "_Calendar". As you may imagine, this depends on device type.

To reduce displayed data, select only "DeviceRowId" column with the matching IDX.

In you case, as data from few day ago, looking to MeterCalendard table should be sufficient.

If there's some strange data in "Value" column, then we may try to fix it. If table is only starting after the "incident" date, it may means that Domoticz erased data, and recovery would be different (something like extracting it from backup database and reloading it).

In all cases, having a look to a database copy is safe (as described). Changing "production" database value may not be...
sailmich
Posts: 232
Joined: Wednesday 17 February 2016 22:55
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Germany
Contact:

Re: General usage lost values

Post by sailmich »

@FlyingDomotic thank you very much for your help! But I would spent to much time to find the bad data. I stay with the crashed counter and made a new one with sending the correct values according to the manual.
@FireWizard I took a look into shellies API and found that the plug is also sending energy :) I got a solution with http but would like to do it with mqtt. I still don`t now how to get booth values together and sent it via mqtt. May you help me.
If crashing the dummy device by false values isn't a bug, could someone please move the tread.
Thanks
Spoiler: show

Code: Select all

[
    {
        "id": "b4ece2bd784ad56a",
        "type": "join",
        "z": "449126130c89dd8a",
        "name": "",
        "mode": "custom",
        "build": "object",
        "property": "payload",
        "propertyType": "msg",
        "key": "topic",
        "joiner": "\\n",
        "joinerType": "str",
        "accumulate": false,
        "timeout": "",
        "count": "2",
        "reduceRight": false,
        "reduceExp": "",
        "reduceInit": "",
        "reduceInitType": "",
        "reduceFixup": "",
        "x": 870,
        "y": 1220,
        "wires": [
            [
                "8c791a5f7d32afa6"
            ]
        ]
    },
    {
        "id": "b62a35e1c4344295",
        "type": "change",
        "z": "449126130c89dd8a",
        "name": "get power",
        "rules": [
            {
                "t": "set",
                "p": "topic",
                "pt": "msg",
                "to": "Watt_Erzeugung",
                "tot": "str"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 380,
        "y": 1200,
        "wires": [
            [
                "b4ece2bd784ad56a"
            ]
        ]
    },
    {
        "id": "9b53bfdac90aed76",
        "type": "change",
        "z": "449126130c89dd8a",
        "name": "get energy",
        "rules": [
            {
                "t": "set",
                "p": "topic",
                "pt": "msg",
                "to": "kWh_Erzeugung",
                "tot": "str"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 390,
        "y": 1260,
        "wires": [
            [
                "b4ece2bd784ad56a"
            ]
        ]
    },
    {
        "id": "8c791a5f7d32afa6",
        "type": "http request",
        "z": "449126130c89dd8a",
        "name": "set SUW variable",
        "method": "POST",
        "ret": "txt",
        "paytoqs": "ignore",
        "url": "http://xxxxxxxxxxxxx/json.htm?type=command&param=udevice&idx=785&nvalue=0&svalue={{payload.Watt_Erzeugung}};{{payload.kWh_Erzeugung}}",
        "tls": "",
        "persist": false,
        "proxy": "",
        "insecureHTTPParser": false,
        "authType": "",
        "senderr": false,
        "headers": [],
        "x": 1090,
        "y": 1220,
        "wires": [
            []
        ]
    }
]
User avatar
FireWizard
Posts: 1745
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: General usage lost values

Post by FireWizard »

Good evening @sailmich

You wrote:
I took a look into shellies API and found that the plug is also sending energy
That is correct, but it depends on the type of plug (generation 1 or generation 2), how to format the request.
it can be done and then it is a simple conversion.

Can you tell me, what the type of your plug is?. Is it a Shelly Plug or PlugS (Generation 1) or is it a Shelly Plus Plug S (Generation 2)
keep in mind that the value you receive from the Shelly for "Energy" is expressed in Wm (Watt-minute) So 1 Wm = 1/60 Wh
I got a solution with http but would like to do it with mqtt.
That is possible. I created something quick with an "Inject" node, (as I do not know the Generation of the Shelly yet)
See the flow below in order to give you an idea.

Code: Select all

[
    {
        "id": "de9d282626577580",
        "type": "inject",
        "z": "919eeef6718077fa",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "Watt_Erzeugung",
        "payload": "123",
        "payloadType": "num",
        "x": 320,
        "y": 340,
        "wires": [
            [
                "92de407702b6a1c7"
            ]
        ]
    },
    {
        "id": "16fc407458de7947",
        "type": "inject",
        "z": "919eeef6718077fa",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "kWh_Erzeugung",
        "payload": "600",
        "payloadType": "num",
        "x": 320,
        "y": 400,
        "wires": [
            [
                "92de407702b6a1c7"
            ]
        ]
    },
    {
        "id": "92de407702b6a1c7",
        "type": "join",
        "z": "919eeef6718077fa",
        "name": "Join Watt and Watt-minute",
        "mode": "custom",
        "build": "object",
        "property": "payload",
        "propertyType": "msg",
        "key": "topic",
        "joiner": "\\n",
        "joinerType": "str",
        "useparts": false,
        "accumulate": true,
        "timeout": "",
        "count": "2",
        "reduceRight": false,
        "reduceExp": "",
        "reduceInit": "",
        "reduceInitType": "",
        "reduceFixup": "",
        "x": 610,
        "y": 360,
        "wires": [
            [
                "48ea9677282ac733",
                "7102f68aedf7232e"
            ]
        ]
    },
    {
        "id": "48ea9677282ac733",
        "type": "debug",
        "z": "919eeef6718077fa",
        "name": "debug 1",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 880,
        "y": 440,
        "wires": []
    },
    {
        "id": "7102f68aedf7232e",
        "type": "function",
        "z": "919eeef6718077fa",
        "name": "function 1",
        "func": "msg.payload = { \"command\": \"udevice\", \"idx\": 528, \"svalue\": msg.payload.Watt_Erzeugung.toString() + \";\" + (msg.payload.kWh_Erzeugung / 60).toString() };\nreturn msg;",
        "outputs": 1,
        "timeout": 0,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 860,
        "y": 300,
        "wires": [
            [
                "a66757038f23b540"
            ]
        ]
    },
    {
        "id": "a66757038f23b540",
        "type": "debug",
        "z": "919eeef6718077fa",
        "name": "debug 2",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 1040,
        "y": 300,
        "wires": []
    }
]
If crashing the dummy device by false values isn't a bug, could someone please move the tread.
I do not see it as a bug, but injecting wrong or no values in the Domoticz database is not a good idea.
But I agrre it would be better, that it would reject the value.
Personally I have never been able to crash Domoticz by injecting something wrong.

So to send it by MQTT, you need

1. Configure your Plug, Plug/S Plug Plus to use MQTT
2. Use to MQTT Explorer to check the communication
3. Let me know the type of plug.

Have a look at the example.

Regards
PieterS
Posts: 195
Joined: Wednesday 31 May 2017 16:06
Target OS: NAS (Synology & others)
Domoticz version: 2024.7
Location: NL
Contact:

Re: General usage lost values

Post by PieterS »

sailmich wrote: Monday 22 July 2024 22:35 @FireWizard I'm pretty sure your solution will work.
Even If it's written, one false code shouldn't destroy the whole device.
I agree that the concept along that device in Domoticz is not failproof. But we have to deal with that...
Two solutions:
1. Ask an enhancement to the team of Domoticz
2. Build a check along NaN in the flow in Node-Red before sending data.

@Firewizard gave me ultimate help with that last option. Thnx and all credits to him.
Synology with Domoticz build (V2024.7) in Docker
sailmich
Posts: 232
Joined: Wednesday 17 February 2016 22:55
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Germany
Contact:

Re: General usage lost values

Post by sailmich »

Hello @FireWizard I have a plug S, with your function node, all is working well. Thank you very much! I had to open the plug s on my fritz box, without internet access it just sending the power. I will compare "computed and from device" for a while, when diference isn't big I will go back to computed.

@PieterS the solution for NaN would be interesting for me and maybe others too, may you post it?

Cheers
PieterS
Posts: 195
Joined: Wednesday 31 May 2017 16:06
Target OS: NAS (Synology & others)
Domoticz version: 2024.7
Location: NL
Contact:

Re: General usage lost values

Post by PieterS »

I understand. I will do that next week. Not easy to reach the flow at this moment. Hotspot, laptop, vpn and the right mood.. yes, it's possible. 8-)

Maybe is @Firewizard able to do that earlier.
Synology with Domoticz build (V2024.7) in Docker
User avatar
FireWizard
Posts: 1745
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: General usage lost values

Post by FireWizard »

I will try tomorrow and will also add a brief explanation.
User avatar
FireWizard
Posts: 1745
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: General usage lost values

Post by FireWizard »

Hi,

As promised I will show a Node-RED example, in order to make to data flow to Domoticz more robust.

I see in several forum posts discussions about the fact that Domoticz cannot handle inputs with errors
It will always be a point of discussion, who is responsible that an application will not crash.

By request I will show a quick solution, which I made for @PieterS

He has 4 different Solar inverters, a Goodwe, Omnik, Growat and a SMA inverter. All inverters can communicate their own way with Node_RED.
In order to display the total produced power and energy, we have to add these values.
It might happen that one ore more values are not (yet) available.

In this case, every inverter communicate with Node-RED and the value of "Power" and "Energy" are stored in a so called "flow variable".
To avoid that some value is not available and the variable will contain undefined we have to take care that you will retrieve always a value.
If the value is not available, we use 0.

Code: Select all

let growatt_power = flow.get("growatt_power") || 0;
let sma_power = flow.get("sma_power") || 0;
let goodwe_power = flow.get("goodwe_power") || 0;
let omnik_power = flow.get("omnik_power") || 0;

let growatt_energy = flow.get("growatt_energy") || 0;
let sma_energy = flow.get("sma_energy") || 0;
let goodwe_etoday = flow.get("goodwe_etoday") || 0;
let omnik_etoday = flow.get("omnik_etoday") || 0;
So the value is either the valu from the "flow variable" or 0.

Normally this will do, but to avoid that a value is indicated as NaN (Not a Number) we will test both power and energy for NaN.

Code: Select all

if (isNaN(total_energy) || isNaN(total_power)) {
 return null // halt flow
};
If, either total_energy or total_power are equal to NaN the flow will stop and not push a NaN to Domoticz.

Total flow, please find below.

Code: Select all

let msg1 = {};
let msg2 = {};
let msg3 = {};

let growatt_power = flow.get("growatt_power") || 0;
let sma_power = flow.get("sma_power") || 0;
let goodwe_power = flow.get("goodwe_power") || 0;
let omnik_power = flow.get("omnik_power") || 0;

let growatt_energy = flow.get("growatt_energy") || 0;
let sma_energy = flow.get("sma_energy") || 0;
let goodwe_etoday = flow.get("goodwe_etoday") || 0;
let omnik_etoday = flow.get("omnik_etoday") || 0;

let total_energy = (growatt_energy * 100) + (sma_energy * 1000) + (goodwe_etoday * 1000) + (omnik_etoday * 1000); //v1
let total_power = (growatt_power / 10) + sma_power + goodwe_power + omnik_power; //v2

if (isNaN(total_energy) || isNaN(total_power)) {
 return null // halt flow
};

msg1.payload = { "command": "udevice", "idx": 2578, "nvalue": 0, "svalue": total_power.toString() + ";" + total_energy.toString() };
msg2.payload = total_energy.toString();
msg3.payload = total_power.toString();

return [msg1, msg2, msg3];
Regards
Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest