Python Plugin: MqttMapper

[FlyingDomotic]

Few words to explain how it works.

MQTT topics are organized around topics. A topic can have sub-topics, which in turn can have sub-topics...

A topic (as well as a sub-topic) can have a value.

This value could be raw, or formatted as JSON.

When using MQTT Explorer, topics are displayed on the left side, sub-topics folded just under them.When clicking on a topic.sub-topic, value (content) is displayed on the right. It's possible to display it raw and even in JSON or XML format, should value been of this structure.

With that, MqttMapper needs to get the tree specification (topic/sub-topic/...) on "topic" content.

If data is raw, meaning to be inserted as it is, use ("item":""). On the other side, if data is JSON formatted, you should specify the JSON path into "Item".

One of the problems to fix, is how to use the same topic multiple times, as Domoticz plug-in need a unique key, and topic is used as this key. In this case, specifying "Key":"xxx" don't use topic as internal key, but the specified key, allowing the same topic to be used multiple times in the same configuration file. As key is not used to anything else but having a unique value, you can put whatever you want here, provided this is not used elsewhere in configuration file, either in "topic" or in "key" values.

That said, you're right, I was wrong, you have to use "glowmarkt/elecconsumptiontoday" as topic, and an empty "Item". And as every topic is unique, you can remove "key".

Concerning the incremental counter, idea is to send permanently deltas, that will be added. This is useful for things that report consumption for a period (minute, hour...) and reset it to zero just after sending it. In your case, unless you use it daily, just before it's reset, this is definitively not the right type.

Concerning electrical counters, internal values are generally in Wh, but display in kWh. Details are available at https://www.domoticz.com/wiki/Domoticz_API/JSON_URL%27s.

Concerning names, you may also rename it after creation (for example to remove plug-in name automatically added by Domoticz), to make it looking as humanly understandable device name, having nothing related to it's creator plug-in.

Lastly, to be sure, what kind of counter did you choose or "Counter" device? Script may have to be adapted depending on your choice. Currently, script is written for Custom counter (both for MqttMapper device AND virtual "counter").

I tested the script with manual values, and it was working. I'll redo tests deeper, to see if it's working as well as I think.

[tarmacalastair]

Thanks again. So it looks like my MqttMapper bit is ok if I understand you correctly.

On the issue of the counter, what I did was, create a hardware device of type "Dummy", then I clicked Create Virtual Sensors next to the new hardware and then I created a counter device of type "Counter". The types that I am offered are:

• Counter
• Counter Incremental
• Electric (Instant+Counter)
• Managed Counter
• P1 Smart Meter (Electric)
• Usage (Electric)

With some of the types the dzvents script returns attempt to perform arithmetic on a nil value (local 'counter')
There is also a sensor type "Custom Sensor" but no Custom Counter. I'm really not sure which type to use to fit in with your script. Sorry if I'm being really stupid!

[FlyingDomotic]

You're not stupid! Not asking question or trying to understand would have been...

Device type is 243 and subtype 31, with "custom sensor" as "official" name (and not "custom counter" as I (stupidly) said

MqttMapper device is created with these type and subtype, as described in JSON configuration file. "Counter" virtual device you have to create manually should also be that type. If this is not the case, just delete it and recreate it as "custom sensor".

Click on "edit" on device widget, and check that "type" is properly set to "Energy", with "multiplier" equal to 1. If different, just set right values and click on "Update".

With that, everything should work fine!

[tarmacalastair]

Sorry been away a few days. I've just tried creating the "Counter" virtual device as custom sensor and it asks for a name and an Axis Label. Then when I click on the edit button for the widget I can still only change the name, axis label and icon. There is no option to set the type to Energy or set a multiplier. (none of the counters seem to have a multiplier option but some have a "counter divider" option)
Thanks for your patience.

[FlyingDomotic]

Sorry: Let's try with "Counter" virtual device

[tarmacalastair]

Right, I'm trying that now. Let's see what happens. I've set the "Divider" to "1", default is "0". Is that correct?

[FlyingDomotic]

Divider set to zero means "Default value", which seems to depend on type. Setting it to a non null value helps understanding. You can then change it if you see that counter is not increasing properly.

This divider is used to ... divide counter, when internal unit is not the same as displayed. For example, when you cont liter and display m3, or W and display kW. It's often either 1 or 1000, but can be anything, if you device returns inches and you want to display centimeters.

Here, as value has already been converted by MqttMapper's device, using 1 should keep the value the same. If each Wh increase counter as 1 kWh, set it to 1000 to fix it.

[tarmacalastair]

Thank you. The counter seems to be counting now. I'll keep an eye on it for a couple of days and see what happens.

[tarmacalastair]

Hi again. It seems like the counter (I've called it PlainCounter here because I tried so many types) is sort of working but....

Except for the time periods when our electric vehicle is plugged in and charging (like it was today), the Usage Last 24 Hours seems to show lots of bars which are all the same value - 1kWh

Screenshot 2024-05-11 at 22-02-30 Domoticz.png (104.98 KiB) Viewed 1367 times

Looking back over the week, most days seem to show slightly lower totals than the actual. Last night (Friday), just after midnight, when the counter had zeroed, I took a screenshot showing the MQTT mapper value at 11.9222:

Screenshot 2024-05-11 at 00-01-22 Domoticz.png (42.1 KiB) Viewed 1367 times

but in the previous screenshot for the last week, Friday shows a value of 11.

Just wondering if you have any thoughts/tweaks on these 2 points please?
Thanks.

[FlyingDomotic]

Adding some traces into script, displaying Mapper Elec and Plain counter may help to understand what's happening.

If you don't known how to do, post here the script you're using, I'll modify it.

[tarmacalastair]

Not a clue, sorry. Therefore very grateful for your help.
My MqttMapper.json:

Code: Select all

{
        "Elec": {
                "topic": "glowmarkt/elecconsumptiontoday",
                "key": "elec",
                "type": "243", "subtype": "31", "switchtype": "0",
                "options": {"Custom":"1;Electricity"},
                "mapping": {"item": ""}
        }
}

And my eleccounter.lua file (as you posted except for the counter name):

Code: Select all

- Example managing daily counter reset every day into cumulative counter
return {
        on = {devices = {'Mapper - Elec'}},                                                         -- Device containing today's value
        data={previousValue={initial=0}},                                                               -- Permanent data to save precious value
        execute = function(domoticz, device)
                local counterDevice = domoticz.devices("PlainCounter")                                       -- Load counter device
                local counter = counterDevice.counter                                                   -- Load counter value
                local currentValue = device.sensorValue                                                 -- Load elec value
                local delta = currentValue - domoticz.data.previousValue                                -- Compute delta since last pass
                print(counter, currentValue, domoticz.data.previousValue, delta)
                domoticz.data.previousValue = currentValue                                              -- Save previous value for last pass
                if delta >= 0 then                                                                      -- Value incremented
                        counter = counter + delta                                                       -- Add delta
                else
                        counter = counter + currentValue                                                -- Reinitialized, add value
                end
                print(counter)
                counterDevice.updateCounter(counter)                                                    -- Update counter
        end
}

Cheers.

[FlyingDomotic]

Trace was already partly implemented. Here's a new version with a clean trace.

Code: Select all

-- Example managing daily counter reset every day into cumulative counter
return {
    on = {devices = {'Mapper - Elec'}},                                         -- Device containing today's value
    data={previousValue={initial=0}},                                           -- Permanent data to save precious value
    execute = function(domoticz, device)
        local counterDevice = domoticz.devices("PlainCounter")                  -- Load counter device
        local counter = counterDevice.counter                                   -- Load counter value
        local currentValue = device.sensorValue                                 -- Load elec value
        local delta = currentValue - domoticz.data.previousValue                -- Compute delta since last pass
        domoticz.data.previousValue = currentValue                              -- Save previous value for last pass
        if delta >= 0 then                                                      -- Value incremented
            newCounter = counter + delta                                        -- Add delta
        else
            newCounter = counter + currentValue                                 -- Reinitialized, add value
        end
        print("Electricity: Counter="..tostring(counter)..", currentValue="..tostring(currentValue)..", previousValue="..tostring(domoticz.data.previousValue)..", delta="..tostring(delta))
        counterDevice.updateCounter(newCounter)                                 -- Update counter
    end
}

You should have a look at Domoticz log to find lines containing "Electricity: ". You'll then see content of variable before and after change.

[tarmacalastair]

Does this look right to you?

Code: Select all

 2024-05-14 03:20:04.792 Status: dzVents: Electricity: Counter=146.283, currentValue=1.2988, previousValue=1.2988, delta=0.078150000000001
2024-05-14 03:40:04.182 Status: dzVents: Electricity: Counter=146.361, currentValue=1.3339666666667, previousValue=1.3339666666667, delta=0.035166666666668
2024-05-14 04:00:05.869 Status: dzVents: Electricity: Counter=146.396, currentValue=1.4673833333333, previousValue=1.4673833333333, delta=0.13341666666663
2024-05-14 04:20:04.233 Status: dzVents: Electricity: Counter=146.529, currentValue=1.5340666666667, previousValue=1.5340666666667, delta=0.066683333333368
2024-05-14 04:40:04.259 Status: dzVents: Electricity: Counter=146.596, currentValue=1.57475, previousValue=1.57475, delta=0.040683333333301
2024-05-14 05:00:05.788 Status: dzVents: Electricity: Counter=146.637, currentValue=1.6703, previousValue=1.6703, delta=0.095550000000001
2024-05-14 05:20:04.039 Status: dzVents: Electricity: Counter=146.733, currentValue=1.7543666666667, previousValue=1.7543666666667, delta=0.084066666666668
2024-05-14 05:40:04.140 Status: dzVents: Electricity: Counter=146.817, currentValue=1.8009, previousValue=1.8009, delta=0.046533333333301
2024-05-14 06:00:09.954 Status: dzVents: Electricity: Counter=146.864, currentValue=1.9036333333333, previousValue=1.9036333333333, delta=0.10273333333333
2024-05-14 06:20:05.008 Status: dzVents: Electricity: Counter=146.967, currentValue=2.0016666666667, previousValue=2.0016666666667, delta=0.098033333333367
2024-05-14 06:40:04.629 Status: dzVents: Electricity: Counter=147.065, currentValue=2.0609333333333, previousValue=2.0609333333333, delta=0.059266666666634
2024-05-14 07:00:05.779 Status: dzVents: Electricity: Counter=147.124, currentValue=2.3306, previousValue=2.3306, delta=0.2696666666667
2024-05-14 07:20:04.536 Status: dzVents: Electricity: Counter=147.394, currentValue=2.5760833333333, previousValue=2.5760833333333, delta=0.24548333333333
2024-05-14 07:40:04.280 Status: dzVents: Electricity: Counter=147.639, currentValue=2.672, previousValue=2.672, delta=0.0959166666667
2024-05-14 08:00:05.906 Status: dzVents: Electricity: Counter=147.735, currentValue=2.9845833333333, previousValue=2.9845833333333, delta=0.31258333333333
2024-05-14 08:20:04.652 Status: dzVents: Electricity: Counter=148.048, currentValue=3.2120166666667, previousValue=3.2120166666667, delta=0.22743333333337
2024-05-14 08:40:03.970 Status: dzVents: Electricity: Counter=148.275, currentValue=3.2982166666667, previousValue=3.2982166666667, delta=0.086199999999967
2024-05-14 09:00:05.907 Status: dzVents: Electricity: Counter=148.361, currentValue=3.5019333333333, previousValue=3.5019333333333, delta=0.20371666666663
2024-05-14 09:20:04.363 Status: dzVents: Electricity: Counter=148.565, currentValue=3.7605833333333, previousValue=3.7605833333333, delta=0.25865000000003
2024-05-14 09:40:04.477 Status: dzVents: Electricity: Counter=148.824, currentValue=3.8410333333333, previousValue=3.8410333333333, delta=0.080450000000034
2024-05-14 10:00:05.993 Status: dzVents: Electricity: Counter=148.904, currentValue=4.14855, previousValue=4.14855, delta=0.3075166666667
2024-05-14 10:20:04.780 Status: dzVents: Electricity: Counter=149.212, currentValue=4.3060666666667, previousValue=4.3060666666667, delta=0.15751666666667
2024-05-14 10:40:04.202 Status: dzVents: Electricity: Counter=149.37, currentValue=4.4296666666667, previousValue=4.4296666666667, delta=0.12359999999997
2024-05-14 11:00:05.229 Status: dzVents: Electricity: Counter=149.494, currentValue=4.6752, previousValue=4.6752, delta=0.2455333333333 

The currentValue and previousValue on each line are the same.
Thanks

[FlyingDomotic]

Oups ...
A made a istake, loading previousValue too early. Try with the following script, which display it before updating it.

Code: Select all

-- Example managing daily counter reset every day into cumulative counter
return {
    on = {devices = {'Mapper - Elec'}},                                         -- Device containing today's value
    data={previousValue={initial=0}},                                           -- Permanent data to save precious value
    execute = function(domoticz, device)
        local counterDevice = domoticz.devices("PlainCounter")                  -- Load counter device
        local counter = counterDevice.counter                                   -- Load counter value
        local currentValue = device.sensorValue                                 -- Load elec value
        local delta = currentValue - domoticz.data.previousValue                -- Compute delta since last pass
        if delta >= 0 then                                                      -- Value incremented
            newCounter = counter + delta                                        -- Add delta
        else
            newCounter = counter + currentValue                                 -- Reinitialized, add value
        end
        print("Electricity: Counter="..tostring(counter)..", currentValue="..tostring(currentValue)..", previousValue="..tostring(domoticz.data.previousValue)..", delta="..tostring(delta))
        domoticz.data.previousValue = currentValue                              -- Save previous value for last pass
        counterDevice.updateCounter(newCounter)                                 -- Update counter
    end
}

Part of log that will be interesting is at midnight. However, it seems that data is loaded every 20 minutes. Very probably, power consumption between 23:40 and 0.00 will be lost as never sent. Reducing the interval to 1 minute will reduce missing data (you'll loose only 1 minute instead of 20).

[tarmacalastair]

Thanks, I'll try the amended script shortly. And I'll need to enable Domoticz OS logging so I can see the entries around midnight.

When I first started querying the remote data centre I tried requesting updates every minute but I kept getting locked out so I changed my crontab to run the script every 20 minutes.

I'll try reducing the interval to 10 minutes and then do a separate crontab entry for 23:59, 00:00 & 00:01 just to cover that interval. Although, having said that I believe that the smart meters only upload their data every 30 minutes to the national database which is why I chose 20 minute intervals.

Sorry that this is taking up your time but I am very grateful to you for your help. Must do a wiki entry once I have it working.

[tiga]

clearly i have something wrong....but i dont know what.

this is what i have setup:

Code: Select all

7-7-2024 13:11:09node: debug wtw
itho/ithostatus : msg.payload : Object
object
Requested fanspeed (%): 0
Balance (%): 99
Supply fan (RPM): 742
Supply fan actual (RPM): 740
Exhaust fan (RPM): 750
Exhaust fan actual (RPM): 750
Supply temp (°C): 21.76
Exhaust temp (°C): 20.3
Status: 0
Room temp (°C): 21.76
Outdoor temp (°C): 20.3
Valve position: 0
Bypass position: 0
Summercounter: 127.7
Summerday (K_min): 0
Frost timer: 0
Boiler timer: 177
Frost block: 120
Current position: 0
VKKswitch: 0
GHEswitch: 0
Airfilter counter: 276
Global fault code: 0
Actual Mode: 1
Pir fan speed level: -1
Highest received CO2 value (Ppm): 32767
Highest received RH value (%RH): 239
Air Quality (%): 119.5
Remaining override timer (Sec): 0
Fallback speed timer (Sec): 0
Label out of bound error: 0

this is what i have in mqttmapper.json:

Code: Select all

{
        "itho ruimte temperatuur": {
        "topic": "itho/ithostatus",
        "type": "80", "subtype": "5", "switchtype": "0",
        "mapping": {"item": "Room temp (°C)"}
        },

        "itho buiten temperatuur": {
        "topic": "itho/ithostatus",
        "type": "80", "subtype": "5", "switchtype": "0",
        "mapping": {"item": "Outdoor temp (°C)"}
        }
}

it updates strange in domoticz like this:

Code: Select all

 2024-07-07 13:02:09.670 mqttmapper: onMQTTConnected found itho/ithostatus, Device 'itho ruimte temperatuur'
2024-07-07 13:02:09.670 mqttmapper: Setting itho ruimte temperatuur to >21.33<
2024-07-07 13:02:09.680 mqttmapper: onMQTTConnected found itho/ithostatus, Device 'itho ruimte temperatuur'
2024-07-07 13:02:09.680 mqttmapper: Setting itho ruimte temperatuur to >20.82< 

it updates 'itho ruimte temperatuur' with the correct value of 21.33 and then directly overwrites it with outdoor temp value of 20.82
what am i doing wrong?i have tried several versions but no succes



solved it by using "key"

[tiga]

i have an other little problem, i want the bypass position: 0= closed 1= open to turn a switch on/off

this is what i have in my .json:

Code: Select all

        "itho bypassklep": {
        "topic": "itho/ithostatus",
        "key": "itho/ithostatus/Bypass position",
        "type": "244", "subtype": "73", "switchtype": "0",
        "mapping": {"item": "Bypass position", "default": "0", "values": {"Off": "0", "On": "1"}}
        }

this is what i have in the log in domoticz:

Code: Select all

 2024-07-07 16:03:13.539 mqttmapper: onMQTTConnected found itho/ithostatus, Device 'mqttmapper - itho bypassklep'
2024-07-07 16:03:13.539 mqttmapper: testValue="Off" (<class 'str'>), readValue="1" (<class 'str'>)
2024-07-07 16:03:13.539 mqttmapper: testValue="On" (<class 'str'>), readValue="1" (<class 'str'>)
2024-07-07 16:03:13.539 mqttmapper: Setting mqttmapper - itho bypassklep to 0/0 

what do i need to change to get this bit working?

[FlyingDomotic]

clearly i have something wrong....but i dont know what.

this is what i have setup:

Code: Select all

7-7-2024 13:11:09node: debug wtw
itho/ithostatus : msg.payload : Object
object
Requested fanspeed (%): 0
Balance (%): 99
Supply fan (RPM): 742
Supply fan actual (RPM): 740
Exhaust fan (RPM): 750
Exhaust fan actual (RPM): 750
Supply temp (°C): 21.76
Exhaust temp (°C): 20.3
Status: 0
Room temp (°C): 21.76
Outdoor temp (°C): 20.3
Valve position: 0
Bypass position: 0
Summercounter: 127.7
Summerday (K_min): 0
Frost timer: 0
Boiler timer: 177
Frost block: 120
Current position: 0
VKKswitch: 0
GHEswitch: 0
Airfilter counter: 276
Global fault code: 0
Actual Mode: 1
Pir fan speed level: -1
Highest received CO2 value (Ppm): 32767
Highest received RH value (%RH): 239
Air Quality (%): 119.5
Remaining override timer (Sec): 0
Fallback speed timer (Sec): 0
Label out of bound error: 0

this is what i have in mqttmapper.json:

Code: Select all

{
        "itho ruimte temperatuur": {
        "topic": "itho/ithostatus",
        "type": "80", "subtype": "5", "switchtype": "0",
        "mapping": {"item": "Room temp (°C)"}
        },

        "itho buiten temperatuur": {
        "topic": "itho/ithostatus",
        "type": "80", "subtype": "5", "switchtype": "0",
        "mapping": {"item": "Outdoor temp (°C)"}
        }
}

it updates strange in domoticz like this:

Code: Select all

 2024-07-07 13:02:09.670 mqttmapper: onMQTTConnected found itho/ithostatus, Device 'itho ruimte temperatuur'
2024-07-07 13:02:09.670 mqttmapper: Setting itho ruimte temperatuur to >21.33<
2024-07-07 13:02:09.680 mqttmapper: onMQTTConnected found itho/ithostatus, Device 'itho ruimte temperatuur'
2024-07-07 13:02:09.680 mqttmapper: Setting itho ruimte temperatuur to >20.82< 

it updates 'itho ruimte temperatuur' with the correct value of 21.33 and then directly overwrites it with outdoor temp value of 20.82
what am i doing wrong?i have tried several versions but no succes



solved it by using "key"

This is probably due to the fact that you had twice the same topic, while Python only keeps one. You may fix it using 2 different "key" items, with the same "topic". In this case, Python will use the 2 different "key", and do the job properly.

I'll add a warning when loading json file if this kind of problem occurs.

[tiga]

thanks i used the key and it works.

only thing i can not get working is the bypass position to operate a switch.

see my later post.

[FlyingDomotic]

i have an other little problem, i want the bypass position: 0= closed 1= open to turn a switch on/off

this is what i have in my .json:

Code: Select all

        "itho bypassklep": {
        "topic": "itho/ithostatus",
        "key": "itho/ithostatus/Bypass position",
        "type": "244", "subtype": "73", "switchtype": "0",
        "mapping": {"item": "Bypass position", "default": "0", "values": {"Off": "0", "On": "1"}}
        }

this is what i have in the log in domoticz:

Code: Select all

 2024-07-07 16:03:13.539 mqttmapper: onMQTTConnected found itho/ithostatus, Device 'mqttmapper - itho bypassklep'
2024-07-07 16:03:13.539 mqttmapper: testValue="Off" (<class 'str'>), readValue="1" (<class 'str'>)
2024-07-07 16:03:13.539 mqttmapper: testValue="On" (<class 'str'>), readValue="1" (<class 'str'>)
2024-07-07 16:03:13.539 mqttmapper: Setting mqttmapper - itho bypassklep to 0/0 

what do i need to change to get this bit working?

Let's try with:

Code: Select all

{
"itho bypassklep": {
        "topic": "itho/ithostatus",
        "key": "itho/ithostatus/Bypass position",
        "type": "244", "subtype": "73", "switchtype": "0",
        "mapping": {"item": "Bypass position", "default": "0", "values": {"0": "0", "1": "100"}}
        }
}

The correct mapping syntax structure is "values":{"[mqtt external value]": "[domoticz internal value]"}.

Here, if I correctly guess, MQTT value is either 0 or 1, while Domoticz values for switch are 0 or 100. Using "values": {"0": "0", "1": "100"}, we say : if you receive "0" in "Bypass position", then put "0" in Domoticz (Off), if "1" in "Bypass position", then put "100" in Domoticz (On).