Domoticz database check script

Moderator: leecollings

User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Domoticz database check script

Post by waaren »

* Edit: 20190228 ; Include official sqlite3 error messages on failure
* Edit: 20190808 ; Include optional trigger of folllow up OS command
* Edit: 20191005 ; add Telegram as notification option
* Edit: 20191005 ; Include optional set of an Alert device with result of check
* Edit: 20191019 ; Include RC message in alert device
* Edit: 20200108 ; fixed bug(ss) in aftersucces / afterSuccess (case matters and double s also!) -- Thx. to MikeF for reporting

Triggered by a forum member experiencing unexpected behavior of domoticz, that after some root cause analysis turned out to be caused by a malformed (= corrupt) database, I created this dzVents script to check the state of the database.
It will not prevent a corruption but will signal you via Email and/or notification as soon as it find one. For now there are only four checks which should catch the 'database disk image is malformed Error.
If you have an idea for more and better domoticz database checks please let me know. If they can be implemented without consuming too much CPU cycles, I will try to add them.

When not yet familiar with dzVents please start with reading Get started before implementing (~ 5 minutes). Special attention please for "In Domoticz go to Setup > Settings > Other and in the section EventSystem make sure the checkbox 'dzVents disabled' is not checked. Also make sure that in the Security section in the settings you allow 127.0.0.1 to not need a password. dzVents uses that port to send certain commands to Domoticz. Finally make sure you have set your current location in Setup > Settings > System > Location, otherwise there is no way to determine nighttime/daytime state."

Have Fun !

Code: Select all

--[[ -- DB check
        
        Will give you a warning when something is wrong with the (domoticz) sqlite database
        requires sqlite3 
            install command on linux:    sudo apt install sqlite3 
            install command on openwrt:  opkg install sqlite3-cli
            install command on synology: sudo /opt/bin/opkg install sqlite3-cli
            
            History:
            20190125: first public release  
            20190228: Include official sqlite3 error messages on failure
            20190808: Include optional trigger of follow up OS command
            20191005: add Telegram as notification option
            20191005: Include optional set of an Alert device with result of check          
            20191019: Include RC message in alert device 
            20200108; Fixed bug(ss) in aftersucces / afterSuccess (case matters and double s also ! )

]]--
return {
            on =    {  
                        timer   =   { 
                                        "at 20:07",             -- change to a time that suits you
                                        "at 07:07",
                                    },            
                    },
    
        logging =   {  
                        level     =   domoticz.LOG_DEBUG,       -- change to LOG_ERROR after you tested  the script
                        marker    =   "DB check" 
                    },

    execute = function(dz)
        -- =======================  Settings below this line =================
        local email               = true                        -- set to false if you don't want an Email when something wrong with database
        local notify              = true                        -- set to false if you don't want a notification when something wrong with database
        local afterSuccessCommand = '/bashdir/copyDB.sh'        -- set to full qualified path of your followUp script or false if you don't need a followUp after success
        local afterFailCommand    = false                       -- set to full qualified path of your followUp script or false if you don't need a followUp after fail
     
        local followUpAfterFail   = false                       -- set to false if you don't need a followUpAfterFail
        
        local alert = {
                        active   = true,                             -- set to false if you don't want an Alert device set. 
                        device   = "dbCheck result",                 -- Name of your alert device (only needed when you set active = true )
                        failText = "Problem in database",
                        OKText   = "database OK",    
                      }
                      
        local subject             = "Domoticz database check"   -- Free text
        local emailaddress        = "[email protected]"        -- Your Email address
        local path                = "/opt/domoticz/"            -- full qualified path to your database 
        local database            = "domoticz.db"               -- database filename + extension
        -- local database         = "corrupt.db"                -- test database filename + extension
                                                                    -- you can corrupt a test database by just load it in an editor
                                                                    -- and remove a couple of bytes
        local sqlite              = "/usr/bin/sqlite3"          -- location of your sqlite3 tool (use the command 'which sqlite3' to find location)
  
        local myNotificationTable     =     {
                                             -- table with one or more notification systems. 
                                             -- uncomment the notification systems that you want to be used
                                             -- Can be one or more of
                                             
                                             dz.NSS_GOOGLE_CLOUD_MESSAGING, 
                                             -- dz.NSS_HTTP, 
                                             -- dz.NSS_KODI, 
                                             -- dz.NSS_LOGITECH_MEDIASERVER, 
                                             -- dz.NSS_NMA,
                                             -- dz.NSS_PROWL, 
                                             -- dz.NSS_PUSHALOT, 
                                             -- dz.NSS_PUSHBULLET, 
                                             dz.NSS_PUSHOVER, 
                                             -- dz.NSS_PUSHSAFER,
                                             dz.NSS_TELEGRAM,
                                            }
        -- =======================  No modification needed below this line ==================

         local function logWrite(str,level)             -- Support function for shorthand debug log statements
            dz.log(tostring(str),level or dz.LOG_DEBUG)
        end
    
        local space                   = " " 
        local baseCommand             = "sudo" .. space .. sqlite .. space .. path .. database .. space
        local checks                  = {} 
              checks                  = {
                                                  "\'select count(id) from deviceStatus;\'",
                                                   "\'.schema\'",
                                                  "\'pragma integrity_check;\'",
                                                  "\'pragma foreign_key_check;\'",
                                        }

        local function rc2Text(rc)
            local errorMessages =   {
                                          [0] = "database OK",
                                          [1] = "Generic error",
                                          [2] = "Internal logic error in SQLite",
                                          [3] = "Access permission denied",
                                          [4] = "Callback routine requested an abort",
                                          [5] = "The database file is locked",
                                          [6] = "A table in the database is locked",
                                          [7] = "memory allocation failed",
                                          [8] = "Attempt to write a readonly database",
                                          [9] = "Operation terminated by sqlite3_interrupt",
                                         [10] = "Some kind of disk I/O error occurred",
                                         [11] = "The database disk image is malformed",
                                         [12] = "Unknown opcode in sqlite3_file_control",
                                         [13] = "Insertion failed because database is full",
                                         [14] = "Unable to open the database file",
                                         [15] = "Database lock protocol error",
                                         [16] = "Internal use only",
                                         [17] = "The database schema changed",
                                         [18] = "String or BLOB exceeds size limit",
                                         [19] = "Abort due to constraint violation",
                                         [20] = "Data type mismatch",
                                         [21] = "Library used incorrectly",
                                         [22] = "Uses OS features not supported on host",
                                         [23] = "Authorization denied",
                                         [24] = "Not used",
                                         [25] = "2nd parameter to sqlite3_bind out of range",
                                         [26] = "File opened that is not a database file",
                                     }
            return(errorMessages[rc] or "Unknown error")
        end

        local function followUp(cmd)
            os.execute('sudo ' .. cmd ..' &')
        end

        local function osExecute(base,check)
            local fileHandle     = assert(io.popen(base .. check, 'r'))
            local commandOutput  = assert(fileHandle:read('*a'))
            local returnTable    = {fileHandle:close()}
            check = check:gsub("'","") .." result ==>> " ..  ( returnTable[3] ~= 0 and "Failed: " .. rc2Text(returnTable[3]) .. commandOutput .. " (".. returnTable[3] .. ")" or true and "OK" )
            logWrite("Command " .. check )
            return check,returnTable[3]            -- rc[3] contains returnCode
        end

        local function checkDatabase()
            if dz.utils.fileExists(path .. database) then
                if dz.utils.fileExists(sqlite) then
                    for _,check in ipairs (checks) do
                        local result,rc = osExecute(baseCommand,check)
                        if rc ~= 0 then 
                            return rc, result  
                        end
                    end
                else
                    return -1,"sqlite3 not installed"
                end
            else
                return -1,"wrong path to database"
            end
            return 0
        end

        local function updateAlert(rc)
            if alert and alert.active then
                local now = dz.time.rawDate .. ', ' .. dz.time.rawTime .. ': '
                local alertLevel = dz.ALERTLEVEL_RED
                if rc == 0 then  alertLevel = dz.ALERTLEVEL_GREEN end
                dz.devices(alert.device).updateAlertSensor(alertLevel, now .. rc2Text(rc))
            end
        end

        -- main program
        local rc, result = checkDatabase()
        if rc ~= 0 then
            logWrite(result,dz.LOG_ERROR)
            if email then dz.email(subject,result,emailaddress) end
            if notify then 
                dz.notify(subject, result or "Dbase check encountered unknown error", dz.PRIORITY_NORMAL, dz.SOUND_INTERMISSION,"",  myNotificationTable ) 
            end
            if afterFailCommand and ( rc > 0 ) then followUp(afterFailCommand) end
        else
            if afterSuccessCommand then followUp(afterSuccessCommand) end
        end
        updateAlert(rc) 
    end
}
Last edited by waaren on Thursday 28 February 2019 11:13, edited 2 times in total.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
User avatar
EdwinK
Posts: 1820
Joined: Sunday 22 January 2017 21:46
Target OS: Raspberry Pi / ODroid
Domoticz version: BETA
Location: Rhoon
Contact:

Re: Domoticz database check script

Post by EdwinK »

Thanks :)
Running latest BETA on a Pi-3 | Toon® Thermostat (rooted) | Hue | Tuya | IKEA tradfri | Dashticz V3 on Lenovo Huawei Tablet | Conbee
Snowtiger
Posts: 120
Joined: Tuesday 18 October 2016 13:15
Target OS: Raspberry Pi / ODroid
Domoticz version: 4.10915
Contact:

Re: Domoticz database check script

Post by Snowtiger »

I will have it up and running tomorrow - thx
User avatar
jvdz
Posts: 2189
Joined: Tuesday 30 December 2014 19:25
Target OS: Raspberry Pi / ODroid
Domoticz version: 4.107
Location: Netherlands
Contact:

Re: Domoticz database check script

Post by jvdz »

Shouldn't this command:

Code: Select all

"\'pragma check_integrity;\'",
be?:

Code: Select all

"\'pragma integrity_check;\'",
Jos
New Garbage collection scripts: https://github.com/jvanderzande/GarbageCalendar
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Domoticz database check script

Post by waaren »

jvdz wrote: Sunday 27 January 2019 15:21 Shouldn't this command:

Code: Select all

"\'pragma check_integrity;\'",
be?:

Code: Select all

"\'pragma integrity_check;\'",
Jos
Yes, you are right. Pragma commands do not give a warning or error when using an incorrect command so I did not notice that before. I updated the script in post #1.
Thanks !
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
imdos
Posts: 34
Joined: Thursday 03 August 2017 21:50
Target OS: Raspberry Pi / ODroid
Domoticz version: 2023.2
Contact:

Re: Domoticz database check script

Post by imdos »

I have the script running as well and am noticing that (up until now) every other day one of the 4 checks is failing. This one to be specific.

Code: Select all

"\'select count(id) from deviceStatus;\'",
To be precise; I implemented this on the 24th or 25th and it gave an error on the 25'th and not on the 26'th but did again yesterday.

Code: Select all

2019-02-27 17:07:00.491  Status: dzVents: Error (2.4.14): DB check: select count(id) from deviceStatus; result ==>> Failed:   (11)
Should I start worrying or change it to another moment in time which is less likely to fail?
And what information does this error provides me with?
Making use of: Raspbian(SSD), dz Beta, Woonveilig, Z-Wave(alarm+multi-sensor), RFLink(blinds), P1, Yeelight, Xiaomi temp sensors, Tasmota(SonoFF, Blitzwolf SHP2, Shelly1)
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Domoticz database check script

Post by waaren »

imdos wrote: Thursday 28 February 2019 9:30 I have the script running as well and am noticing that (up until now) every other day one of the 4 checks is failing. This one to be specific.

Code: Select all

"\'select count(id) from deviceStatus;\'",
To be precise; I implemented this on the 24th or 25th and it gave an error on the 25'th and not on the 26'th but did again yesterday.

Code: Select all

2019-02-27 17:07:00.491  Status: dzVents: Error (2.4.14): DB check: select count(id) from deviceStatus; result ==>> Failed:   (11)
Should I start worrying or change it to another moment in time which is less likely to fail?
And what information does this error provides me with?
Result code 11 does tell you that sqlite3 encountered a corrupt database trying to execute the sql command. The script will not continue with the other tests but send a mail, notification and/or message in the logfile.
My reaction to such a message would be to:
- double check if I have recent database backups and tested recovery procedures;
- stop domoticz
- copy domoticz.db to check.db
- manually perform the checks from the script on check.db

Any further actions would depend on the outcome of this action but result code 11 is to be taken serious. At least it is a warning that something is not OK on your system. Hard to tell if it is application or hardware (including SD card wear out) related. Do you have any external programs / scripts (not from the domoticz program itself) accessing the domoticz database directly ?
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Domoticz database check script

Post by waaren »

imdos wrote: Thursday 28 February 2019 9:30 And what information does this error provides me with?
Good question !
I modified the script in post #1 to present more (hopefully useful) information when an sqlite3 error occur.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
Snowtiger
Posts: 120
Joined: Tuesday 18 October 2016 13:15
Target OS: Raspberry Pi / ODroid
Domoticz version: 4.10915
Contact:

Domoticz database check script

Post by Snowtiger »

I modified the script in post #1 to present more (hopefully useful) information when an sqlite3 error occur.
thanks a lot Image
imdos
Posts: 34
Joined: Thursday 03 August 2017 21:50
Target OS: Raspberry Pi / ODroid
Domoticz version: 2023.2
Contact:

Re: Domoticz database check script

Post by imdos »

waaren wrote: Thursday 28 February 2019 10:14 Result code 11 does tell you that sqlite3 encountered a corrupt database trying to execute the sql command. The script will not continue with the other tests but send a mail, notification and/or message in the logfile.
My reaction to such a message would be to:
- double check if I have recent database backups and tested recovery procedures;
- stop domoticz
- copy domoticz.db to check.db
- manually perform the checks from the script on check.db

Any further actions would depend on the outcome of this action but result code 11 is to be taken serious. At least it is a warning that something is not OK on your system. Hard to tell if it is application or hardware (including SD card wear out) related. Do you have any external programs / scripts (not from the domoticz program itself) accessing the domoticz database directly ?
Pfewh; I fixed it. It had to do with my actions to cleanup my database after upgrading and dumping a lot of unused IDX'es from devices due to the large number of broadcasts from 433Mhz stuff.

After stopping domoticz the first check was ok and the error came from the pragma integrity_check.
This is the guide which I followed to correct my issue.

Long story short; I copied all tables and rows as sql inserts into a blank table and am up and running again.

Thanks for your good job! :D
Making use of: Raspbian(SSD), dz Beta, Woonveilig, Z-Wave(alarm+multi-sensor), RFLink(blinds), P1, Yeelight, Xiaomi temp sensors, Tasmota(SonoFF, Blitzwolf SHP2, Shelly1)
megamarco83
Posts: 107
Joined: Friday 21 September 2018 15:07
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Domoticz database check script

Post by megamarco83 »

waaren wrote: Friday 25 January 2019 2:40 * Edit: 20190228 ; Include official sqlite3 error messages on failure

Triggered by a forum member experiencing unexpected behaviour of domoticz, that after some root cause analysis turned out to be caused by a malformed (= corrupt) database. I created this dzVents script to check the state of the database.
It will not prevent a corruption but will signal you via Email and/or notification as soon as it find one. For now there are only four checks which should catch the 'database disk image is malformed Error.
If you have an idea for more and better domoticz database checks please let me know. If they can be implemented without consuming too much CPU cycles, I will try to add them.

Have Fun !
hi, thanks as always for ideas and contribution!
just an idea, could be an opportunity / possibility with this dzvent code to implement: after the checking of integrity of domoticz.db, the copy/uploading of the database to a external FTP? in this way we will have always a backup copy that is working :-)
thanks
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Domoticz database check script

Post by waaren »

megamarco83 wrote: Thursday 08 August 2019 14:47 just an idea, could be an opportunity / possibility with this dzvent code to implement: after the checking of integrity of domoticz.db, the copy/uploading of the database to a external FTP? in this way we will have always a backup copy that is working :-)
@megamarco83, hi Marco,

there are a lot of different possibilities to copy the domoticz database. I use the internal backup command, an hourly snapshot on my Nas of the complete system and a daily copy to a dropbox clone.
There is no way to get even close to implement all the different methods in the dzVents script so I took a slightly different approach. In the latest version of the script (see initial post ) you can optionally define an afterSuccess and/or an afterFail command. If declared, the script will trigger an OS program or script as sudo and in the background so the event system will not get to a halt.
With that option you can just keep your preferred backup method but triggered based on the success / failure of the database check.
Please let me know how that works for you.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
megamarco83
Posts: 107
Joined: Friday 21 September 2018 15:07
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Domoticz database check script

Post by megamarco83 »

waaren wrote: Thursday 08 August 2019 18:28
megamarco83 wrote: Thursday 08 August 2019 14:47 just an idea, could be an opportunity / possibility with this dzvent code to implement: after the checking of integrity of domoticz.db, the copy/uploading of the database to a external FTP? in this way we will have always a backup copy that is working :-)
@megamarco83, hi Marco,

there are a lot of different possibilities to copy the domoticz database. I use the internal backup command, an hourly snapshot on my Nas of the complete system and a daily copy to a dropbox clone.
There is no way to get even close to implement all the different methods in the dzVents script so I took a slightly different approach. In the latest version of the script (see initial post ) you can optionally define an afterSuccess and/or an afterFail command. If declared, the script will trigger an OS program or script as sudo and in the background so the event system will not get to a halt.
With that option you can just keep your preferred backup method but triggered based on the success / failure of the database check.
Please let me know how that works for you.
that's great, thanks so much, as usual! :)
BOverdevest
Posts: 23
Joined: Wednesday 28 March 2018 20:56
Target OS: Raspberry Pi / ODroid
Domoticz version: 4.11307
Location: Nederland
Contact:

Re: Domoticz database check script

Post by BOverdevest »

Hi Waaren

Thanks for another great script!!

Potential additions/ideas.

1) Adding NSS_TELEGRAM as notification route.
2) Applying an alert sensor, with the 'result' as the text. Green when 'OK', red if other.

See
https://www.domoticz.com/wiki/DzVents:_ ... ert_sensor

greetings
Bart
HUE, Tradfri, Zwave, RFXCOM, rooted TOON, PS4
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Domoticz database check script

Post by waaren »

BOverdevest wrote: Friday 04 October 2019 23:07 Potential additions/ideas.
1) Adding NSS_TELEGRAM as notification route.
2) Applying an alert sensor, with the 'result' as the text. Green when 'OK', red if other.
@Boverdevest, Hi Bart,

Thx for the compliment and a good idea.
Can you please test this amended script and revert with your findings. If ok I will update the script in the topic start.

Code: Select all

--[[ -- DB check
        
        Will give you a warning when something is wrong with the (domoticz) sqlite database
        requires sqlite3 
            install command on linux:    sudo apt install sqlite3 
            install command on openwrt:  opkg install sqlite3-cli
            install command on synology: sudo /opt/bin/opkg install sqlite3-cli
            
            History:
            20190125: first public release  
            20190228: Include official sqlite3 error messages on failure
            20190808: Include optional trigger of folllow up OS command
            20191005: add Telegram as notification option
            20191005: Include optional set of an Alert device with result of check            
]]--
return {
            on =    {  
                        timer   =   { 
                                        "at 20:07",             -- change to a time that suits you
                                        "at 07:07",
                                    },            
                    },
    
        logging =   {  
                        level     =   domoticz.LOG_DEBUG,       -- change to LOG_ERROR after you tested  the script
                        marker    =   "DB check" 
                    },

    execute = function(dz)
        -- =======================  Settings below this line =================
        local email               = true                        -- set to false if you don't want an Email when something wrong with database
        local notify              = true                        -- set to false if you don't want a notification when something wrong with database
        local aftersuccessCommand = '/bashdir/copyDB.sh'        -- set to full qualified path of your followUp script or false if you don't need a followUp after success
        local afterFailCommand    = false                       -- set to full qualified path of your followUp script or false if you don't need a followUp after fail
     
        local followUpAfterFail   = false                       -- set to false if you don't need a followUpAfterFail
        
        local alert = {
                        active   = true,                             -- set to false if you don't want an Alert device set. 
                        device   = "dbCheck result",                 -- Name of your alert device (only needed when you set active = true )
                        failText = "Problem in database",
                        OKText   = "database OK",    
                      }
                      
        local subject             = "Domoticz database check"   -- Free text
        local emailaddress        = "[email protected]"        -- Your Email address
        local path                = "/opt/domoticz/"            -- full qualified path to your database 
        local database            = "domoticz.db"               -- database filename + extension
        -- local database         = "corrupt.db"                -- test database filename + extension
                                                                    -- you can corrupt a test database by just load it in an editor
                                                                    -- and remove a couple of bytes
        local sqlite              = "/usr/bin/sqlite3"          -- location of your sqlite3 tool (use the command 'which sqlite3' to find location)
  
        local myNotificationTable     =     {
                                             -- table with one or more notification systems. 
                                             -- uncomment the notification systems that you want to be used
                                             -- Can be one or more of
                                             
                                             dz.NSS_GOOGLE_CLOUD_MESSAGING, 
                                             -- dz.NSS_HTTP, 
                                             -- dz.NSS_KODI, 
                                             -- dz.NSS_LOGITECH_MEDIASERVER, 
                                             -- dz.NSS_NMA,
                                             -- dz.NSS_PROWL, 
                                             -- dz.NSS_PUSHALOT, 
                                             -- dz.NSS_PUSHBULLET, 
                                             dz.NSS_PUSHOVER, 
                                             -- dz.NSS_PUSHSAFER,
                                             dz.NSS_TELEGRAM,
                                            }
        -- =======================  No modification needed below this line ==================

         local function logWrite(str,level)             -- Support function for shorthand debug log statements
            dz.log(tostring(str),level or dz.LOG_DEBUG)
        end
    
        local space                   = " " 
        local baseCommand             = "sudo" .. space .. sqlite .. space .. path .. database .. space
        local checks                  = {} 
              checks                  = {
                                                  "\'select count(id) from deviceStatus;\'",
                                                   "\'.schema\'",
                                                  "\'pragma integrity_check;\'",
                                                  "\'pragma foreign_key_check;\'",
                                        }

        local function rc2Text(rc)
            local errorMessages =   {
                                          [1] = "Generic error",
                                          [2] = "Internal logic error in SQLite",
                                          [3] = "Access permission denied",
                                          [4] = "Callback routine requested an abort",
                                          [5] = "The database file is locked",
                                          [6] = "A table in the database is locked",
                                          [7] = "memory allocation failed",
                                          [8] = "Attempt to write a readonly database",
                                          [9] = "Operation terminated by sqlite3_interrupt",
                                         [10] = "Some kind of disk I/O error occurred",
                                         [11] = "The database disk image is malformed",
                                         [12] = "Unknown opcode in sqlite3_file_control",
                                         [13] = "Insertion failed because database is full",
                                         [14] = "Unable to open the database file",
                                         [15] = "Database lock protocol error",
                                         [16] = "Internal use only",
                                         [17] = "The database schema changed",
                                         [18] = "String or BLOB exceeds size limit",
                                         [19] = "Abort due to constraint violation",
                                         [20] = "Data type mismatch",
                                         [21] = "Library used incorrectly",
                                         [22] = "Uses OS features not supported on host",
                                         [23] = "Authorization denied",
                                         [24] = "Not used",
                                         [25] = "2nd parameter to sqlite3_bind out of range",
                                         [26] = "File opened that is not a database file",
                                     }
            return(errorMessages[rc] or "Unknown error")
        end

        local function followUp(cmd)
            os.execute('sudo ' .. cmd ..' &')
        end

        local function osExecute(base,check)
            local fileHandle     = assert(io.popen(base .. check, 'r'))
            local commandOutput  = assert(fileHandle:read('*a'))
            local returnTable    = {fileHandle:close()}
            check = check:gsub("'","") .." result ==>> " ..  ( returnTable[3] ~= 0 and "Failed: " .. rc2Text(returnTable[3]) .. commandOutput .. " (".. returnTable[3] .. ")" or true and "OK" )
            logWrite("Command " .. check )
            return check,returnTable[3]            -- rc[3] contains returnCode
        end
        
        local function checkDatabase()
            if dz.utils.fileExists(path .. database) then
                if dz.utils.fileExists(sqlite) then
                    for _,check in ipairs (checks) do
                        local result,rc = osExecute(baseCommand,check)
                        if rc ~= 0 then 
                            return rc, result  
                        end
                    end
                else
                    return -1,"sqlite3 not installed"
                end
            else
                return -1,"wrong path to database"
            end
            return 0
        end
        
        local function updateAlert(rc)
            if alert and alert.active then
                local now = dz.time.rawDate .. ', ' .. dz.time.rawTime .. ': '
                if rc == 0 then
                    dz.devices(alert.device).updateAlertSensor(dz.ALERTLEVEL_GREEN, now .. alert.OKText)
                else
                    dz.devices(alert.device).updateAlertSensor(dz.ALERTLEVEL_RED, now .. alert.failText)
                end 
            end
        end
        
        -- main program
        local rc, result = checkDatabase()
        if rc ~= 0 then
            logWrite(result,dz.LOG_ERROR)
            if email then dz.email(subject,result,emailaddress) end
            if notify then 
                dz.notify(subject, result or "Dbase check encountered unknown error", dz.PRIORITY_NORMAL, dz.SOUND_INTERMISSION,"",  myNotificationTable ) 
            end
            if afterFailCommand and ( rc > 0 ) then followUp(afterFailCommand) end
        else
            if afterSuccesCommand then followUp(afterSuccesCommand) end
        end
        updateAlert(rc) 
    end
}
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
BOverdevest
Posts: 23
Joined: Wednesday 28 March 2018 20:56
Target OS: Raspberry Pi / ODroid
Domoticz version: 4.11307
Location: Nederland
Contact:

Re: Domoticz database check script

Post by BOverdevest »

Hi Waaren

Seems to work ok, thanks for picking this up.
1 addition, how about passing the error message into the text of the alert button?

greetings
Bart
HUE, Tradfri, Zwave, RFXCOM, rooted TOON, PS4
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Domoticz database check script

Post by waaren »

waaren wrote: Saturday 19 October 2019 1:10
BOverdevest wrote: Friday 18 October 2019 22:48 1 addition, how about passing the error message into the text of the alert button?
Like this ?

Code: Select all

--[[ -- DB check
        
        Will give you a warning when something is wrong with the (domoticz) sqlite database
        requires sqlite3 
            install command on linux:    sudo apt install sqlite3 
            install command on openwrt:  opkg install sqlite3-cli
            install command on synology: sudo /opt/bin/opkg install sqlite3-cli
            
            History:
            20190125: first public release  
            20190228: Include official sqlite3 error messages on failure
            20190808: Include optional trigger of folllow up OS command
            20191005: add Telegram as notification option
            20191005: Include optional set of an Alert device with result of check          
            20191019: Include RC message in alert device 
]]--
return {
            on =    {  
                        timer   =   { 
                                        "at 20:07",             -- change to a time that suits you
                                        "at 07:07",
                                    },            
                    },
    
        logging =   {  
                        level     =   domoticz.LOG_DEBUG,       -- change to LOG_ERROR after you tested  the script
                        marker    =   "DB check" 
                    },

    execute = function(dz)
        -- =======================  Settings below this line =================
        local email               = true                        -- set to false if you don't want an Email when something wrong with database
        local notify              = true                        -- set to false if you don't want a notification when something wrong with database
        local aftersuccessCommand = '/bashdir/copyDB.sh'        -- set to full qualified path of your followUp script or false if you don't need a followUp after success
        local afterFailCommand    = false                       -- set to full qualified path of your followUp script or false if you don't need a followUp after fail
     
        local followUpAfterFail   = false                       -- set to false if you don't need a followUpAfterFail
        
        local alert = {
                        active   = true,                             -- set to false if you don't want an Alert device set. 
                        device   = "dbCheck result",                 -- Name of your alert device (only needed when you set active = true )
                        failText = "Problem in database",
                        OKText   = "database OK",    
                      }
                      
        local subject             = "Domoticz database check"   -- Free text
        local emailaddress        = "[email protected]"        -- Your Email address
        local path                = "/opt/domoticz/"            -- full qualified path to your database 
        local database            = "domoticz.db"               -- database filename + extension
        -- local database         = "corrupt.db"                -- test database filename + extension
                                                                    -- you can corrupt a test database by just load it in an editor
                                                                    -- and remove a couple of bytes
        local sqlite              = "/usr/bin/sqlite3"          -- location of your sqlite3 tool (use the command 'which sqlite3' to find location)
  
        local myNotificationTable     =     {
                                             -- table with one or more notification systems. 
                                             -- uncomment the notification systems that you want to be used
                                             -- Can be one or more of
                                             
                                             dz.NSS_GOOGLE_CLOUD_MESSAGING, 
                                             -- dz.NSS_HTTP, 
                                             -- dz.NSS_KODI, 
                                             -- dz.NSS_LOGITECH_MEDIASERVER, 
                                             -- dz.NSS_NMA,
                                             -- dz.NSS_PROWL, 
                                             -- dz.NSS_PUSHALOT, 
                                             -- dz.NSS_PUSHBULLET, 
                                             dz.NSS_PUSHOVER, 
                                             -- dz.NSS_PUSHSAFER,
                                             dz.NSS_TELEGRAM,
                                            }
        -- =======================  No modification needed below this line ==================

         local function logWrite(str,level)             -- Support function for shorthand debug log statements
            dz.log(tostring(str),level or dz.LOG_DEBUG)
        end
    
        local space                   = " " 
        local baseCommand             = "sudo" .. space .. sqlite .. space .. path .. database .. space
        local checks                  = {} 
              checks                  = {
                                                  "\'select count(id) from deviceStatus;\'",
                                                   "\'.schema\'",
                                                  "\'pragma integrity_check;\'",
                                                  "\'pragma foreign_key_check;\'",
                                        }

        local function rc2Text(rc)
            local errorMessages =   {
                                          [0] = "database OK",
                                          [1] = "Generic error",
                                          [2] = "Internal logic error in SQLite",
                                          [3] = "Access permission denied",
                                          [4] = "Callback routine requested an abort",
                                          [5] = "The database file is locked",
                                          [6] = "A table in the database is locked",
                                          [7] = "memory allocation failed",
                                          [8] = "Attempt to write a readonly database",
                                          [9] = "Operation terminated by sqlite3_interrupt",
                                         [10] = "Some kind of disk I/O error occurred",
                                         [11] = "The database disk image is malformed",
                                         [12] = "Unknown opcode in sqlite3_file_control",
                                         [13] = "Insertion failed because database is full",
                                         [14] = "Unable to open the database file",
                                         [15] = "Database lock protocol error",
                                         [16] = "Internal use only",
                                         [17] = "The database schema changed",
                                         [18] = "String or BLOB exceeds size limit",
                                         [19] = "Abort due to constraint violation",
                                         [20] = "Data type mismatch",
                                         [21] = "Library used incorrectly",
                                         [22] = "Uses OS features not supported on host",
                                         [23] = "Authorization denied",
                                         [24] = "Not used",
                                         [25] = "2nd parameter to sqlite3_bind out of range",
                                         [26] = "File opened that is not a database file",
                                     }
            return(errorMessages[rc] or "Unknown error")
        end

        local function followUp(cmd)
            os.execute('sudo ' .. cmd ..' &')
        end

        local function osExecute(base,check)
            local fileHandle     = assert(io.popen(base .. check, 'r'))
            local commandOutput  = assert(fileHandle:read('*a'))
            local returnTable    = {fileHandle:close()}
            check = check:gsub("'","") .." result ==>> " ..  ( returnTable[3] ~= 0 and "Failed: " .. rc2Text(returnTable[3]) .. commandOutput .. " (".. returnTable[3] .. ")" or true and "OK" )
            logWrite("Command " .. check )
            return check,returnTable[3]            -- rc[3] contains returnCode
        end

        local function checkDatabase()
            if dz.utils.fileExists(path .. database) then
                if dz.utils.fileExists(sqlite) then
                    for _,check in ipairs (checks) do
                        local result,rc = osExecute(baseCommand,check)
                        if rc ~= 0 then 
                            return rc, result  
                        end
                    end
                else
                    return -1,"sqlite3 not installed"
                end
            else
                return -1,"wrong path to database"
            end
            return 0
        end

        local function updateAlert(rc)
            if alert and alert.active then
                local now = dz.time.rawDate .. ', ' .. dz.time.rawTime .. ': '
                local alertLevel = dz.ALERTLEVEL_RED
                if rc == 0 then  alertLevel = dz.ALERTLEVEL_GREEN end
                dz.devices(alert.device).updateAlertSensor(alertLevel, now .. rc2Text(rc))
            end
        end

        -- main program
        local rc, result = checkDatabase()
        if rc ~= 0 then
            logWrite(result,dz.LOG_ERROR)
            if email then dz.email(subject,result,emailaddress) end
            if notify then 
                dz.notify(subject, result or "Dbase check encountered unknown error", dz.PRIORITY_NORMAL, dz.SOUND_INTERMISSION,"",  myNotificationTable ) 
            end
            if afterFailCommand and ( rc > 0 ) then followUp(afterFailCommand) end
        else
            if afterSuccesCommand then followUp(afterSuccesCommand) end
        end
        updateAlert(rc) 
    end
}
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
MikeF
Posts: 350
Joined: Sunday 19 April 2015 0:36
Target OS: Raspberry Pi / ODroid
Domoticz version: V2022.2
Location: UK
Contact:

Re: Domoticz database check script

Post by MikeF »

Hi waaren,

In your script, you have 'afterSuccesCommand' in 'main program', but 'aftersuccessCommand in 'Settings'
snellejellep
Posts: 241
Joined: Tuesday 16 May 2017 13:05
Target OS: Raspberry Pi / ODroid
Domoticz version: 2020.2
Location: The Neterlands
Contact:

Re: Domoticz database check script

Post by snellejellep »

just installed the script. works like a charm. many thanks!
raspberry pi | xiaomi vacuum | yeelight | philips hue | zwave | ubiquiti unifi | harmony | sonoff | zigbee2mqtt | https://www.youtube.com/channel/UC2Zidl ... m1OLuNldfQ
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Domoticz database check script

Post by waaren »

MikeF wrote: Wednesday 08 January 2020 17:07 In your script, you have 'afterSuccesCommand' in 'main program', but 'aftersuccessCommand in 'Settings'
You are right ! Thx for reporting. Changed now in first post.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest