Enphase script stopt working

[Fredom]

Dear members,
I've been using the script below for over a year and never had a problem with it until now.
It stopped overnight with a strange error message. The same thing happened on my backup raspberry. So I suspect it comes from outside. My solar panels work perfectly, it's not that.
Who understands this and can help me to get it working again

Code: Select all

--[[

    This dzVents script is one approach to collect data from your local Enphase system

    It is inspired by the various postings on the domoticz forum about getting details
    from your own solar panels.

    The script uses the installer login for which an Android password generator tool
    is available. Full process description about the development of this tool can be found at

    https://thecomputerperson.wordpress.com/2016/08/28/reverse-engineering-the-enphase-installer-toolkit/

    the Android tool described can be downloaded from
    https://www.dropbox.com/s/xc40op8eqfrykaa/AndroidXam.AndroidXam.Signed2019.apk?dl=0

    The script uses wget calls and interprets the JSON returns.
    for Windows you can get a wget executable via https://eternallybored.org/misc/wget/

    If the createDevices switch is set to true. The script will create ALL the individual devices 
    if they do not exist on the system yet. (1 total and 1 per panel (so if you own 20 panels it creates 21 devices)

    Before activating the script:
    Please read the GETTING STARTED section of the dzVents wiki.
        For windows users download and install the wget utility
        Define a dummy hardware in domoticz if not already done
        Enter your settings at the appropriate place
        Have fun !!

    Please use the domoticz forum fore reporting bugs, asking questions for clarification, tips for improvement, etc..

    History
    0.201909210900 -- Start design / coding
    0.201909251100 -- First version for forum
    0.201909261300 -- Add automated change of inverter device Type to delivery
    0.201910122300 -- Prepared for windows
    
]]--

local scriptVersion = '0.201910122300'
local scriptVar  =  'Enphase_' .. scriptVersion
local frequency = 1 -- -- As far as I know Enphase reporting is only once / 15 minutes

return
{
    on =
    {
        timer = { 'every ' .. frequency .. ' minutes between sunrise and 45 minutes after sunset' }, 
    },

    logging =
    {
        level = domoticz.LOG_DEBUG, -- Change to info or Error if everything works as expected.
        marker = scriptVar,
    },

    data = { inverters = { initial = {} }},

    execute = function(dz)
        -- ==============================
        -- Your setting here
        -- ==============================
        -- Change to reflect location of your wget and remove -- ( comment ) in front of the declaration for your OS  
         wgetExecutable = '/usr/bin/wget'                 -- Linux
        -- wgetExecutable = 'c:\\"Program files"\\wget.exe' -- Windows: double back slashes where one is in the path

                local Enphase =    {
                                ip = '192.168.1.XXX',       -- IP address of your local enphase hub
                                password = 'eXXXXC3b',      -- Change to your calculated password. See description above on how to get this.
                                dummyHardwareIDX = 42,      -- Change to ID of your virtual hardware
                                useProduction = true,
                                productionDeviceName = 933,
                                useInverters = true,
                                inverterDevicePrefix = 'micro inverter: ',  -- script use this prefix + serial number of inverter.
                                createDevices = true,                       -- Should the script create devices if they are not defined yet.

                                -- please note that the counter display will only show a rough estimate
                                -- of the produced energy over the period the device lives in domoticz and the script is active.

        -- ==========================================
        -- == No changes required below this line ==
        -- ==========================================
                                user = 'installer',                                     -- I think this is a fixed username
                                inverterDeviceType = 'Electric (instant+Counter)',      -- script use this type for inverters
                                productionDeviceType = 'Electric (instant+Counter)',    -- script use this type for total production
                                productionAPI = '/api/v1/production',
                                invertersAPI = '/api/v1/production/inverters',

                            }

        local function createEnphaseDevice(deviceType, deviceName)
            local sensorMappedTypes =   {
                                            ['Electric (instant+Counter)'] = '0xF31D',
                                            ['Youless'] = '0xFC01',
                                            ['Usage (electric)'] = '0xF801',
                                        }
            Enphase.openURLDelay = Enphase.openURLDelay or 0                 -- This wil lprevent domoticz to get overloaded by repetative createDevice calls
            local deviceName = dz.utils.urlEncode(deviceName)
            url =   dz.settings['Domoticz url'] .. '/json.htm?type=createdevice&idx=' ..
                    Enphase.dummyHardwareIDX .. '&sensorname=' ..
                    deviceName .. '&sensormappedtype='.. sensorMappedTypes[deviceType]
            dz.openURL(url).afterSec(Enphase.openURLDelay)
            Enphase.openURLDelay = Enphase.openURLDelay + 1
        end

        local function changeInverter2DeliveryType(inverter)
            Enphase.openURLDelay = Enphase.openURLDelay or 0
            url =   dz.settings['Domoticz url'] .. '/json.htm?type=setused&description=&switchtype=4&EnergyMeterMode=0&used=true' .. 
                    '&idx=' ..  inverter.id  ..
                    '&name=' .. dz.utils.urlEncode(inverter.name)
            dz.openURL(url).afterSec(Enphase.openURLDelay)
            dz.log('Changing inverter ' .. inverter.name ..' to delivery type: \n' .. url,dz.LOG_FORCE)
            Enphase.openURLDelay = Enphase.openURLDelay + 1
        end

        local function osCommand(cmd)
            dz.log(cmd,LOG_DEBUG)
            local fileHandle = assert(io.popen(cmd, 'r'))
            local commandOutput = assert(fileHandle:read('*a'))
            local returnTable = {fileHandle:close()}

            if returnTable[3] ~= 0 then
                dz.log("ReturnCode: " .. returnTable[3] .. "\ncommandOutput:\n" .. commandOutput, dz.LOG_ERROR)
            else
                return commandOutput
            end
            dz.log("ReturnCode: " .. returnTable[3] .. "\ncommandOutput:\n" .. commandOutput, dz.LOG_ERROR)
        end

        local function getEnphaseData(api)
            local baseCommand = wgetExecutable .. ' -T 3 --user ' .. Enphase.user .. ' --password ' .. Enphase.password ..' -O - '
            local cmd = baseCommand .. Enphase.ip .. api
            return osCommand(cmd)
        end

        local function updateInverter(device, newInverter, deviceName )
            if device then
                for _, oldInverter in ipairs(dz.data.inverters) do
                    if oldInverter['serialNumber'] == newInverter['serialNumber'] then -- ignore inverter if no history (will be there next run)
                        device.updateElectricity(newInverter['lastReportWatts'],device.WhTotal + tonumber(newInverter['lastReportWatts']) / ( 60 / frequency ) )
                        if device.switchTypeValue ~= 4 then changeInverter2DeliveryType(device) end
                        if dz.time.matchesRule('between 25 minutes after sunset and 23:59') then device.updateElectricity(0,device.WhTotal) end
                    end
                end
            else
                dz.log('Device ' .. deviceName .. ' does not exist. ', dz.LOG_ERROR)
            end
        end

        local function getInverters()
            return dz.utils.fromJSON(getEnphaseData(Enphase.invertersAPI) or {})
        end

        local function procesInverters(inverterTable)
            if inverterTable[1] == nil then
                dz.log('Empty return from Enphase. Go check it out', dz.LOG_ERROR)
                return
            end

            for _, inverter in ipairs(inverterTable) do
                local deviceName = Enphase.inverterDevicePrefix .. inverter.serialNumber
                if not(dz.utils.deviceExists(deviceName)) and Enphase.createDevices then
                    dz.log('Device ' .. deviceName .. ' will be created. ', dz.LOG_FORCE)
                    createEnphaseDevice(Enphase.inverterDeviceType, deviceName)
                else
                    updateInverter(dz.devices(deviceName), inverter, deviceName )
                end
           end
            dz.data.inverters = inverterTable -- store data for next run
        end

        local function getProduction()
           return dz.utils.fromJSON(getEnphaseData(Enphase.productionAPI) or {})
        end

        local function procesProduction(productionTable)
            if productionTable.wattsNow == nil then
                dz.log('Empty return from Enphase. Go check it out', dz.LOG_ERROR)
                return
            end

            local noDevice = not(dz.utils.deviceExists(Enphase.productionDeviceName))
            if noDevice and Enphase.createDevices then
                 dz.log('Device ' .. Enphase.productionDeviceName .. ' will be created. ', dz.LOG_FORCE)
                 createEnphaseDevice(Enphase.productionDeviceType, Enphase.productionDeviceName)
            elseif noDevice then
                dz.log('Device ' .. Enphase.productionDeviceName .. ' does not exist. ', dz.LOG_ERROR)
            else
                dz.devices(Enphase.productionDeviceName).updateElectricity(productionTable.wattsNow, productionTable.wattHoursLifetime )
            end
        end

        -- main
        if Enphase.useProduction then procesProduction(getProduction()) end
        if Enphase.useInverters then procesInverters(getInverters()) end
    end
}

error

Code: Select all

2023-07-07 20:44:00.851 Error: dzVents: Error: (3.1.8) Enphase_0.201910122300: ReturnCode: 5
2023-07-07 20:44:00.851 commandOutput:
2023-07-07 20:44:00.851
2023-07-07 20:44:00.851 Error: dzVents: Error: (3.1.8) Enphase_0.201910122300: ReturnCode: 5
2023-07-07 20:44:00.851 commandOutput:
2023-07-07 20:44:00.851
2023-07-07 20:44:00.851 Error: dzVents: Error: (3.1.8) Enphase_0.201910122300: An error occurred when calling event handler Enphase zonnepanelen
2023-07-07 20:44:00.851 Error: dzVents: Error: (3.1.8) Enphase_0.201910122300: ...ripts/dzVents/generated_scripts/Enphase zonnepanelen.lua:176: attempt to index a nil value (local 'productionTable')

[boum]

Maybe your Enphase hub has an update, because it looks like the API changed for production. That's where the nil value happens. (i just know lua/dzvents, no idea about why this fails now, but the wget does not return a valid JSON, it happens before the JSON is parsed)

[bert]

Most likely, like me a day before yesterday apparently, you received an update of the Envoy from D5x to D7x and now a token is required along with https in the script.

From various sources I found the following to work: (mainly https://gathering.tweakers.net/forum/li ... 0#75876110)

you will need to get a token from enphase, and use that to get the data; the following will generate the token (of a little over 400 characters) which is valid for one year:
1. Log in to Enphase Cloud (https://enlighten.enphaseenergy.com) with the system owner Enphase credentials.
2. Paste the token retrieval URL into the web browser’s address bar:

Code: Select all

https://enlighten.enphaseenergy.com/entrez-auth-token?serial_num=<IQ Gateway/envoy_serial_number> 

Put that token in place of the password in the script (line 68 approx)

As far as I know, wget does not work, you will need curl which you may need to install. When that is done, modify line 63 to reflect your setup:

Code: Select all

wgetExecutable = '/usr/bin/curl' 

then lastly on line 131
replace

Code: Select all

local baseCommand = wgetExecutable .. ' -T 3 --user ' .. Enphase.user .. ' --password ' .. Enphase.password ..' -O - '

with

Code: Select all

 local baseCommand = wgetExecutable .. ' -f -k -H \"Authorization: Bearer ' .. Enphase.password .. '\"' ..' -o - https://'

[Fredom]

Maybe your Enphase hub has an update, because it looks like the API changed for production. That's where the nil value happens. (i just know lua/dzvents, no idea about why this fails now, but the wget does not return a valid JSON, it happens before the JSON is parsed)

Thanks for your response I'll look into it

[Fredom]

Most likely, like me a day before yesterday apparently, you received an update of the Envoy from D5x to D7x and now a token is required along with https in the script.

From various sources I found the following to work: (mainly https://gathering.tweakers.net/forum/li ... 0#75876110)

you will need to get a token from enphase, and use that to get the data; the following will generate the token (of a little over 400 characters) which is valid for one year:
1. Log in to Enphase Cloud (https://enlighten.enphaseenergy.com) with the system owner Enphase credentials.
2. Paste the token retrieval URL into the web browser’s address bar:

Code: Select all

https://enlighten.enphaseenergy.com/entrez-auth-token?serial_num=<IQ Gateway/envoy_serial_number> 

Put that token in place of the password in the script (line 68 approx)

As far as I know, wget does not work, you will need curl which you may need to install. When that is done, modify line 63 to reflect your setup:

Code: Select all

wgetExecutable = '/usr/bin/curl' 

then lastly on line 131
replace

Code: Select all

local baseCommand = wgetExecutable .. ' -T 3 --user ' .. Enphase.user .. ' --password ' .. Enphase.password ..' -O - '

with

Code: Select all

 local baseCommand = wgetExecutable .. ' -f -k -H \"Authorization: Bearer ' .. Enphase.password .. '\"' ..' -o - https://'

Thank you for your response and clear explanation. I will work on it as soon as possible and hope to solve it

[gizmocuz]

Why not using the native Enphase implementation in Domoticz? (beta)

[bert]

because:
1- beta
2- when trying to fix this last night, I did not know that the native implementation was doing individual panels already.. Good job !!
3- if that native implementation breaks for some reason, I am at the mercy of someone willing and able to fix that quickly; like I am now with the HomeAssistant instance I have running in parallel which is now missing data; pending some fix in the mainline release. This dzvents script is easy to follow and adapt; so prefer that.. (Domoticz rulez!!)
4 - most likely, I will end up with a bunch of new devices which need to be replaced, and I know how I work and for sure I am going to mess up the homeplan I made, and consequently the export to influx and the Grafana querries. - too many little strings carefully tied together to risk on a new integration.

But that's just me; I guess for anyone else it might just be a better option indeed

[Fredom]

Why not using the native Enphase implementation in Domoticz? (beta)

Thanks for the tip.
I'll take a look at it

[gizmocuz]

Just implemented the option to disable/enable power generation

@bert, why would the native implementation break? I think we got V7 firmware working pretty early!
Next to this, I am also using it

Replacing the bunch of devices is very easy, press edit, press replace, select the new device, and done.

Another reason, stability and memory usage, especially with python scripts.

[arnout]

Just implemented the option to disable/enable power generation

@bert, why would the native implementation break? I think we got V7 firmware working pretty early!
Next to this, I am also using it

Replacing the bunch of devices is very easy, press edit, press replace, select the new device, and done.

Another reason, stability and memory usage, especially with python scripts.

I'm using the native Enphase implementation with my Enphase still on version R4.10.35. After updating to the newest Beta, I get an error and no data in my Domoticz-Enphase-device:
'2023-07-11 16:59:18.185 Error: Enphase Envoy: Error getting http data! (info)'

I think it has somtething to do with the latest change.

(yes, I'm in the process of updating my Enphase to the newest firmware)

[Fredom]

Why not using the native Enphase implementation in Domoticz? (beta)

Hi gizmocuz,
I got it working. But is it also possible to see all inverters as well?
That would complete it again for me

[gizmocuz]

@Fredom, please update to the latest beta version
In your hardware setup, enable the checkbox to enable individual inverter details.
(Also make sure you have enabled 'Accent new hardware devices' in your settings for a few minutes, and they will show up under 'Devices' where you can add them

[gizmocuz]

@arnout, please update to the latest beta version. Seems < V7 needs HTTP instead of HTTPS...
It should work again, but time to update There could be big security holes in lower firmwares, they won't make them without reason

[Fredom]

@Fredom, please update to the latest beta version
In your hardware setup, enable the checkbox to enable individual inverter details.
(Also make sure you have enabled 'Accent new hardware devices' in your settings for a few minutes, and they will show up under 'Devices' where you can add them

Hi gizmocuz
I found it and it works great.

How and where can I update the plugin in an older version. Domoticz v2022.2 with Debian. Here the plugin does not yet have the option to enter user and password. I have 2 systems old is in use and the new one is still in testing phase. The plugin works great on my test system.

[gizmocuz]

For your old systems, make a backup. (make a backup of the domoticz folder and place this on another system)
Export the database (via the settings page)
Then under Settings, under Software Updates (main page) enable the beta option and apply
Then you can do Setup->Check for updates

Faster,maybe easier, login via ssh, go the domoticz folder and issue

./updatebeta

[HSR3]

Most likely, like me a day before yesterday apparently, you received an update of the Envoy from D5x to D7x and now a token is required along with https in the script.

From various sources I found the following to work: (mainly https://gathering.tweakers.net/forum/li ... 0#75876110)

you will need to get a token from enphase, and use that to get the data; the following will generate the token (of a little over 400 characters) which is valid for one year:
1. Log in to Enphase Cloud (https://enlighten.enphaseenergy.com) with the system owner Enphase credentials.
2. Paste the token retrieval URL into the web browser’s address bar:

This did the trick for me, thanks. Additional to your procedure I had to remove the "HTTP" prefix of the IP address in line 67

When we installed the solar panels last year I added the native Enphase implementation but this somehow didn't generate the gateway and inverters. The LUA script however did work.
After the 2023.1 release beginning this year I was afraid that my Somfy screens wouldn't work. Now the sun is shining I found that they are not working anymore anyway with the 2022.1, probably got a firmware update as well.

When I am in the mood I will start updating to the latest version (possibly beta) to see if this fixes both issues

[Fredom]

For your old systems, make a backup. (make a backup of the domoticz folder and place this on another system)
Export the database (via the settings page)
Then under Settings, under Software Updates (main page) enable the beta option and apply
Then you can do Setup->Check for updates

Faster,maybe easier, login via ssh, go the domoticz folder and issue

./updatebeta

Thanks everything works