copy a api/telegram into a local text file [Solved]

[fonger]

Hi There

I have a P1 dongle into my smart meter. I want to capture the values and use them into Domoticz.
The values I got from the P1 dongle is in a telegram style. on a local ip 192.168.178.16/api/v1/telegram, and it looks like this:

/ISK5\2M550T-1013

1-3:0.2.8(50)
0-0:1.0.0(210309202253W)
0-0:96.1.1(4530303534303037363733333231333230)
1-0:1.8.1(000733.488*kWh)
1-0:1.8.2(000839.845*kWh)
1-0:2.8.1(000061.240*kWh)
1-0:2.8.2(000145.582*kWh)
0-0:96.14.0(0002)
1-0:1.7.0(02.441*kW)
1-0:2.7.0(00.000*kW)
0-0:96.7.21(00004)
0-0:96.7.9(00004)
1-0:99.97.0(2)(0-0:96.7.19)(200529212626S)(0000000228*s)(201025114028W)(0000000776*s)
1-0:32.32.0(00000)
1-0:52.32.0(00000)
1-0:72.32.0(00000)
1-0:32.36.0(00001)
1-0:52.36.0(00001)
1-0:72.36.0(00001)
0-0:96.13.0()
1-0:32.7.0(226.4*V)
1-0:52.7.0(224.7*V)
1-0:72.7.0(225.9*V)
1-0:31.7.0(001*A)
1-0:51.7.0(009*A)
1-0:71.7.0(000*A)
1-0:21.7.0(00.212*kW)
1-0:41.7.0(02.221*kW)
1-0:61.7.0(00.004*kW)
1-0:22.7.0(00.000*kW)
1-0:42.7.0(00.000*kW)
1-0:62.7.0(00.000*kW)
0-1:24.1.0(003)
0-1:96.1.0(4730303634303032303130373737393230)
0-1:24.2.1(210309202003W)(00865.754*m3)
!

I found a topic (Domoticz + P1 monitor share device) about this and the solution was a script that writes the values to the devices, but that script was based on a file who got filled bij a p1 monitor.
My question is: is there a possibility to copy the telegram (frequently) from the site and write it down to a local text file on my raspberry?
So I can use the script wrote for the p1 monitor.

Grtz Fonger

[FireWizard]

Hi, @fonger

The published telegram is a correct telegram of a Smart Meter, except that it doesn't contain a CRC.
So disable the CRC check.

Why not use the native hardware in Domoticz? (P1 Smart Meter with LAN interface)?

See: https://www.domoticz.com/wiki/Dutch_DSM ... th_P1_port

Regards

[fonger]

The way that you describe isn't working, I don't get the information from the p1 meter, Domoticz doesn't recognize the meter.
Thats why I want to do it the way I describe above.
But I don't know how to copy the telegram and write it down to a local .txt file

Grtz Fonger

[waaren]

The way that you describe isn't working, I don't get the information from the p1 meter, Domoticz doesn't recognize the meter.
Thats why I want to do it the way I describe above.
But I don't know how to copy the telegram and write it down to a local .txt file

Happy to help but to determine the best approach I would like to know your domoticz / dzVents Version.

What devices (type/subtype/idx ) do you want to use for this data? Maybe the data can be directly used by dzVents to update these devices.

[fonger]

Thanks

I copied your last script from the topic:

viewtopic.php?p=255949#p255949

That works fine, if I manually make a copy of the telegram, and write it to the text file.
"/tmp/p1Data/p1msg.txt" (change from the original script)

Al the devices are in Domoticz, and get that values from that file.

So I think I need a script that makes a copy from the telegram on the local IP (timebased) and and write it down to the p1msg.txt file.

log:

2021-03-11 17:37:20.329 Status: EventSystem: Script event triggered: /home/pi/domoticz/dzVents/runtime/dzVents.lua
2021-03-11 17:37:30.296 Status: dzVents: Info: Handling Domoticz custom event for: "readTelegram"
2021-03-11 17:37:30.296 Status: dzVents: Info: readTelegram: ------ Start internal script: p1msg-file: Custom event: "readTelegram"
2021-03-11 17:37:30.308 Status: dzVents: Debug: readTelegram: Processing device-adapter for Power: P1 smart meter energy device adapter
2021-03-11 17:37:30.309 Status: dzVents: Debug: readTelegram: Processing device-adapter for Gas: Gas device adapter
2021-03-11 17:37:30.309 Status: dzVents: Debug: readTelegram: Processing device-adapter for Input Voltage: Voltage device adapter
2021-03-11 17:37:30.310 Status: dzVents: Debug: readTelegram: Processing device-adapter for Usage l1: Electric usage device adapter
2021-03-11 17:37:30.310 Status: dzVents: Debug: readTelegram: Processing device-adapter for Usage l2: Electric usage device adapter
2021-03-11 17:37:30.310 Status: dzVents: Debug: readTelegram: Processing device-adapter for Usage l3: Electric usage device adapter
2021-03-11 17:37:30.311 Status: dzVents: Debug: readTelegram: Processing device-adapter for amperageTotal: 1-phase Ampere device adapter
2021-03-11 17:37:30.311 Status: dzVents: Debug: readTelegram: Processing device-adapter for amperageCombined: 3-phase Ampere device adapter
2021-03-11 17:37:30.312 Status: dzVents: > usage1: 737630
2021-03-11 17:37:30.312 Status: dzVents: > powerUsageL2: 220
2021-03-11 17:37:30.312 Status: dzVents: > tariff: 2000
2021-03-11 17:37:30.312 Status: dzVents: > voltageL3: 230200
2021-03-11 17:37:30.312 Status: dzVents: > usage2: 848595
2021-03-11 17:37:30.312 Status: dzVents: > l1Power: 0
2021-03-11 17:37:30.312 Status: dzVents: > powerReceived: 1684
2021-03-11 17:37:30.312 Status: dzVents: > amperageL1: 1000
2021-03-11 17:37:30.312 Status: dzVents: > voltageL1: 230200
2021-03-11 17:37:30.312 Status: dzVents: > returned1: 61240
2021-03-11 17:37:30.312 Status: dzVents: > powerDelivery: 0
2021-03-11 17:37:30.312 Status: dzVents: > powerUsageL3: 0
2021-03-11 17:37:30.312 Status: dzVents: > gas: 875933
2021-03-11 17:37:30.312 Status: dzVents: > amperageL3: 9000
2021-03-11 17:37:30.312 Status: dzVents: > powerUsageL1: 242
2021-03-11 17:37:30.312 Status: dzVents: > voltageL2: 229800
2021-03-11 17:37:30.312 Status: dzVents: > returned2: 149959
2021-03-11 17:37:30.312 Status: dzVents: > amperageL2: 1000


Grtz Fonger

[waaren]

So I think I need a script that makes a copy from the telegram on the local IP (timebased) and and write it down to the p1msg.txt file.

And the other question was to tell which domoticz / dzVents version you use.

That will influence the approach to take with the scripts

[fonger]

Over Domoticz
Version: 2020.2
Build Hash: b63341bc0
Compile Date: 2020-04-26 13:47:55
dzVents Version: 3.0.2

[waaren]

dzVents Version: 3.0.2

Can you try below script?

if the combination of the two scripts works a next step should be to combine these scripts. It does not make much sense to me to get data from an (for domoticz) external source, write it to a file with script A and use script B to read the same file and process the data.
A more common approach would be to collect the data with a script and use the same script to process it. It should not be needed to use an OS file to store the temp data if dzVents can get the data directly by itself.

Code: Select all

local scriptVar = 'readTelegram'

return
{
    on =
    {
        timer =
        {
            'every minute',
        },
        httpResponses =
        {
            scriptVar,
        },
    },

    logging =
    {
        level = domoticz.LOG_DEBUG, -- set to domoticz.LOG_ERROR when all OK
        marker = scriptVar,
    },

    execute = function(dz, item)

        local smartFile = '/tmp/p1Data/p1msg.txt' --  full qualified file name

        local function openURL(delay)
            dz.openURL(
            {
                url = 'http://192.168.178.16/api/v1/telegram',
                callback = scriptVar,
            }).afterSec(delay or 0)
        end

        local function writeFile(fn, str)
            local f = io.open(fn, 'w' )
            f:write(str)
            f:close()
        end

        -- main code
        if item.isTimer  then
            openURL()
        elseif item.isHTTPResponse and item.ok then
            writeFile(smartFile, item.data)
        else
            dz.log('Problem reading the telegram', dz.LOG_ERROR)
            dz.log(tostring(item.data), dz.LOG_DEBUG)
        end
    end
}

[waltervl]

Would it be something to use the standard http poller for this?
So with http poller get the telegram and with a Lua telegram parser put the data in domotica devices?

[waaren]

Would it be something to use the standard http poller for this?
So with http poller get the telegram and with a Lua telegram parser put the data in domotica devices?

You can tackle this in many different ways. To name a few.

node-red + MQTT
Classic Lua scripting
Python event script
Python plugin
Extend current P1 hardware
Bash scripting
HTTP poller + Lua parser
dzVents

My personal preference is dzVents because I know my way around with it but a large part of the fun in domoticz is that you can choose what best suits you.

[fonger]

Yes! it works properly.

[waaren]

Yes! it works properly.

OK.

can you also try below version? It does the job without temp. OS files. (And using only this script. So no separate read and write scripts)

Code: Select all

local scriptVar = 'readTelegram'

return
{
    on =
    {
        timer =
        {
            'every minute',
        },
        httpResponses =
        {
            scriptVar,
        },
    },

    logging =
    {
        level = domoticz.LOG_DEBUG,
        marker = scriptVar,
    },

    execute = function(dz, item)

        local debug = _G.logLevel == dz.LOG_DEBUG

        local power = dz.devices('Power') -- change to name of your virtual smart meter (P1 smart Meter (Electric))
        local gas = dz.devices('Gas') -- change to name of your virtual smart meter (Gas)
        local volt = dz.devices('Input Voltage') -- change to name of your virtual voltage device

        local l1 = dz.devices('Usage l1')  -- change to name of your virtual Usage L1 (Usage Electric)
        local l2 = dz.devices('Usage l2')  -- change to name of your virtual Usage L1 (Usage Electric)
        local l3 = dz.devices('Usage l3')  -- change to name of your virtual Usage L1 (Usage Electric)

        local ampere1Phase = dz.devices('amperageTotal') -- change to name of your virtual amperage device (1 phase)
        local ampere3Phases = dz.devices('amperageCombined') -- change to name of your virtual amperage device (3 phases)
        local amperage1, amperage2, amperage3 = 0, 0, 0

        local delivered = dz.devices('Delivered') -- change to name of your virtual counter 
        local received = dz.devices('Received') -- change to name of your virtual counter

        local frequency = 6 -- every 60 / frequency seconds
        local P1Reference =
        {
             l1Power = '22.7.0',
             tariff = '96.14.0',
             usage2 = '1.8.2',
             usage1 = '1.8.1',
             returned1 = '2.8.1',
             returned2 = '2.8.2',
             powerReceived = '2.7.0',
             gas = '24.2.1',
             powerUsageL1 = '21.7.0',
             powerUsageL2 = '41.7.0',
             powerUsageL3 = '61.7.0',
             powerDelivery = '1.7.0',
             amperageL1 = '31.7.0',
             amperageL2 = '51.7.0',
             amperageL3 = '71.7.0',
             voltageL1 = '32.7.0',
             voltageL2 = '52.7.0',
             voltageL3 = '72.7.0',
        }

        local function openURL(delay)
            dz.openURL(
            {
                url = 'http://192.168.178.16/api/v1/telegram',
                callback = scriptVar,
            }).afterSec(delay or 0)
        end

        function table.invert(t)
           local it = {}
           for k,v in pairs(t) do
             it[v]=k
           end
           return it
        end

        local function readResult( telegram, it )

            local function crLines(s)
                return s:gsub('\r\n?', '\n'):gmatch('(.-)\n') -- Normalize CR/LF
            end

            local lines = {}
            for line in crLines(telegram) do
                if line:find(':') then
                    key = it[line:match(':(%d+.%d+.%d+)')]
                    if key then
                        lines[key] = math.floor((line:match('%b()%s*%c*$'):match('%d+%.*%d*') or -1) * 1000)
                    end
                end
            end

            if debug then dz.utils.dumpTable(lines) end

            return lines

        end

        local function updateDevices(t)
            power.updateP1  (
                                t.usage1 or 0,
                                t.usage2  or 0,
                                t.returned1  or 0,
                                t.returned2  or 0,
                                t.powerUsageL1 + t.powerUsageL2 + t.powerUsageL3,
                                t.powerDelivery  or 0
                            )

            if t.gas then gas.updateGas(t.gas) end

            if t.powerUsageL1 then
                l1.updateEnergy(t.powerUsageL1)
                amperage1 = dz.utils.round( t.powerUsageL1 / t.voltageL1 , 2)
            end

            if t.powerUsageL2 then
                l2.updateEnergy(t.powerUsageL2)
                amperage2 = dz.utils.round( t.powerUsageL2 / t.voltageL2 , 2)
            end

            if t.powerUsageL3 then
                l3.updateEnergy(t.powerUsageL3 )
                amperage3 = dz.utils.round( t.powerUsageL3 / t.voltageL3 , 2)
            end

            ampere1Phase.updateCurrent( dz.utils.round( amperage1 + amperage2 + amperage3 , 2 ) )
            ampere3Phases.updateCurrent( amperage1, amperage2, amperage3 )

            volt.updateVoltage( dz.utils.round( t.voltageL1 / 1000 ,1 ) )

            delivered.updateCounter(t.powerDelivery or 0) 
            received.updateCounter(t.powerReceived or 0) 
        end

        local function showResults(t, it)
            if not(debug) then return end
            for code, value in pairs(t) do
                if it[code] then
                    dz.log(it[code] .. ' ==>> ' .. value, dz.LOG_DEBUG)
                end
            end
        end

        -- main code
        if item.isTimer then
            openURL()

            local interval = math.floor(60/frequency)
            for secondsDelay = interval, (60 - interval), interval do
                openURL(secondsDelay)
            end
        elseif item.isHTTPResponse and item.ok then
            local invertedTable = table.invert(P1Reference)

            local actual = readResult(item.data, invertedTable)
            showResults(actual, invertedTable)
            updateDevices(actual, invertedTable)
        else
            dz.log('Problem reading the telegram', dz.LOG_ERROR)
            dz.log(tostring(item.data), dz.LOG_DEBUG)
        end
    end
}

[fonger]

Sorry it take a while, I had a crash because of a corrupted sd card, I had to make a complete reinstall and lost everything.
But I'am on air again...

I get an error

LOG info:

Spoiler: show

Code: Select all

2021-03-13 15:29:20.331 Status: dzVents: Info: Handling httpResponse-events for: "readTelegram"
2021-03-13 15:29:20.331 Status: dzVents: Info: readTelegram: ------ Start internal script: P1 meter: HTTPResponse: "readTelegram"
2021-03-13 15:29:20.354 Status: dzVents: Debug: readTelegram: Processing device-adapter for Power: P1 smart meter energy device adapter
2021-03-13 15:29:20.355 Status: dzVents: Debug: readTelegram: Processing device-adapter for Gas: Gas device adapter
2021-03-13 15:29:20.356 Status: dzVents: Debug: readTelegram: Processing device-adapter for Input Voltage: Voltage device adapter
2021-03-13 15:29:20.356 Status: dzVents: Debug: readTelegram: Processing device-adapter for Usage l1: Electric usage device adapter
2021-03-13 15:29:20.356 Status: dzVents: Debug: readTelegram: Processing device-adapter for Usage l2: Electric usage device adapter
2021-03-13 15:29:20.357 Status: dzVents: Debug: readTelegram: Processing device-adapter for Usage l3: Electric usage device adapter
2021-03-13 15:29:20.357 Status: dzVents: Debug: readTelegram: Processing device-adapter for amperageTotal: 1-phase Ampere device adapter
2021-03-13 15:29:20.358 Status: dzVents: Debug: readTelegram: Processing device-adapter for amperageCombined: 3-phase Ampere device adapter
2021-03-13 15:29:20.358 Status: dzVents: Debug: readTelegram:
2021-03-13 15:29:20.358 Status: dzVents: Info: readTelegram: ------ Finished P1 meter
2021-03-13 15:29:20.230 Error: Error opening url: dz:8084/json.htm?type=command&param=getversion
2021-03-13 15:29:20.331 Error: dzVents: Error: (3.0.2) readTelegram: HTTP/1.1 response: 6 ==>> Couldn't resolve host name
2021-03-13 15:29:20.358 Error: dzVents: Error: (3.0.2) readTelegram: Problem reading the telegram

[waaren]

Sorry it take a while, I had a crash because of a corrupted sd card, I had to make a complete reinstall and lost everything.
But I'am on air again...

I get an error

Sorry, was a left behind from my tests. I updated the script I posted.
Can you try again after removing line

Code: Select all

url = "dz:8084/json.htm?type=command&param=getversion" 

[fonger]

Yes the devices getting their values, only the amperageTotal and amperageCombined getting a NAN value.
Log:
2021-03-14 09:45:00.543 Error: UpdateMeter: Error converting sValue/sUsage! (IDX: 57, sValue: 'nan', sUsage: '0', dType: 243, sType: 23)

How I add the 1.7.0 ( powerDelivery) and the 2.7.0 (powerRecieved) to the devices?

[waaren]

Yes the devices getting their values, only the amperageTotal and amperageCombined getting a NAN value.
Log:
2021-03-14 09:45:00.543 Error: UpdateMeter: Error converting sValue/sUsage! (IDX: 57, sValue: 'nan', sUsage: '0', dType: 243, sType: 23)

How I add the 1.7.0 ( powerDelivery) and the 2.7.0 (powerRecieved) to the devices?

For the amperages please try again using the updated posted version.

The powerDelivery is already send to the virtual smart meter but can also be split out to another device.
Do you want type counter devices for delivery and received (= used and produced) or should these values be send to other type devices?

[fonger]

I would like to get the actual (used and proceed) values in different devices.
Now I have noticed that the input Voltage doesn't update.

[waaren]

I would like to get the actual (used and proceed) values in different devices.
Now I have noticed that the input Voltage doesn't update.

The voltage device is in this script for input into it but because your meter already supplies the voltage it's no longer needed for input.
Should the script send the voltage of L1 to this device or the average of voltageL1, voltageL2 and voltageL3 ?

[fonger]

Voltage L1 is fine thanks.

[waaren]

Voltage L1 is fine thanks.

I updated the posted version.