DzVents script to create CSV file for further processing

Moderator: leecollings

Post Reply
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

DzVents script to create CSV file for further processing

Post by waaren »

I created this script for a forum member who had the need to create a rolling csv file to publish the data as a graph using Google charts.

Have Fun !!

Code: Select all

--[[
    script to create a CSV file with following layout.
    
    now,inside,target,outside,lounge,greenhouse,heating  [header row]
    22:10,21.2,7.0,12.7,21.3,11.8,0   [oldest data: 24 hours minus 10 mins. ago]
    22:20,21.1,7.0,12.7,21.3,11.8,0   [next data: 10 mins. later]
    ...
    22:00,21.1,7.0,9.4,22.0,8.0,0     [latest data]

        History:
        20200108: Start coding
        20200110: Start testing on life situation (thanks to MikeF)
        20200122: First public release

]]--

return
{
    on = 
    { 
        timer = 
        {
            -- already quite busy at round minutes so shift a couple of minutes. Choose the one you prefer by uncommenting the line
            'at *:03', 'at *:13', 'at *:23', 'at *:33', 'at *:43', 'at *:53', 
            -- 'every 10 minutes', 
        }, 
    }, 
        
    data = 
    {  
        rows = 
        { 
            history = true, maxItems = 300, maxHours = 48, -- max is whatever is reached first (what will be written to csv is a subset of this) 
        }
    }, 
    
    logging =   
    {
        level = domoticz.LOG_DEBUG, -- set to LOG_ERROR when tested and OK
        marker = 'createCSV', 
    }, 
       
    execute = function(dz)
        
        local dataTable = {}

        -----------------------------------------------------------------------------------------------------------  
        -- Enter header line of your csv text file (or nil if you don't need one ) 
        -- Enter devicenames / methods in the required order. This will determine what attribute-values will be written to the csvFile 
        -- Enter location/name of your csv text file 
        -- Enter time-span to be written to csv file
        -----------------------------------------------------------------------------------------------------------  
        local header =  'now,inside,target,outside,lounge,greenhouse,heating' 
        
        dataTable[1] = dz.time.rawTime:sub(1, 4) .. '0' -- consistent times even if script is not triggered precisely every x0 minutes
        dataTable[2] = dz.utils.round(dz.devices('inside').temperature, 1) 
        dataTable[3] = dz.utils.round(dz.devices('target').temperature, 1)
        dataTable[4] = dz.utils.round(dz.devices('outside').temperature, 1)
        dataTable[5] = dz.utils.round(dz.devices('lounge').temperature, 1)
        dataTable[6] = dz.utils.round(dz.devices('greenhouse').temperature, 1)
        dataTable[7] = dz.devices('heating').active and 5 or 0
        
        local output = '/tmp/myFile.csv' -- location/name of your csv text file
        local span   = '23:59:00'  -- time-span to be written to csv file
        -----------------------------------------------------------------------------------------------------------  
        -- No changes required below these lines
        -----------------------------------------------------------------------------------------------------------  
        
        local function makePersistentData()    
            local row = {}
            for index, value in ipairs(dataTable) do
                row[index] = value
            end          
            return row 
        end

        local function writeCSV()
            local relevantRows = dz.data.rows.subsetSince(span)
            local csvFile = assert(io.open(output, "w"))
            if header then csvFile:write(header ..'\n') end
           
            for index = #relevantRows, 1, -1 do 
                dz.log(table.concat(relevantRows[index].data, ','), dz.LOG_DEBUG) 
                csvFile:write(table.concat(relevantRows[index].data, ',') ..'\n')  
            end

            csvFile:close()
            dz.log((#relevantRows) .. ' data-rows written.', dz.LOG_DEBUG) 
        end
 
        -- main
        dz.data.rows.add(makePersistentData())  -- store in persistent historical data
        writeCSV()
    end
}
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
Mibeus
Posts: 8
Joined: Sunday 29 November 2020 20:29
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: DzVents script to create CSV file for further processing

Post by Mibeus »

this is excellent.

Please can you help me for modification? I need log one month. I dont know what I need change for log more as 24 hours. Now I have this modification for more data limits but still after 24 hours stop recording.

Code: Select all

return
{
    on = { timer = {'every 1 hours'}, 
    }, 
        
    data = 
    {  
        rows = 
        { 
            history = true, maxItems = 3000, maxHours = 3000, -- max is whatever is reached first (what will be written to csv is a subset of this) 
        }
    }, 
    
    logging =   
    {
        level = domoticz.LOG_DEBUG, -- set to LOG_ERROR when tested and OK
        marker = 'createCSV', 
    }, 
       
    execute = function(dz)
        
        local dataTable = {}

        -----------------------------------------------------------------------------------------------------------  
        -- Enter header line of your csv text file (or nil if you don't need one ) 
        -- Enter devicenames / methods in the required order. This will determine what attribute-values will be written to the csvFile 
        -- Enter location/name of your csv text file 
        -- Enter time-span to be written to csv file
        -----------------------------------------------------------------------------------------------------------  
        local header =  'Date,Time,SMT Humidity,SMT Temperature, THT Humidity,THT Temperature, Operators Humidity, Operators Temperature, CC1 Humidity, CC1 Temperature, CC2 Humidity, CC2 Temperature, Storage Humidity, Storage Temperature' 
        
        dataTable[1] = dz.time.rawDate
        dataTable[2] = dz.time.rawTime:sub(1, 4) .. '0' -- consistent times even if script is not triggered precisely every x0 minutes
        dataTable[3] = dz.utils.round(dz.devices('SMTsensor').humidity, 1) 
        dataTable[4] = dz.utils.round(dz.devices('SMTsensor').temperature, 1)
        dataTable[5] = dz.utils.round(dz.devices('THTsensor').humidity, 1) 
        dataTable[6] = dz.utils.round(dz.devices('THTsensor').temperature, 1)
        dataTable[7] = dz.utils.round(dz.devices('Operatorssensor').humidity, 1) 
        dataTable[8] = dz.utils.round(dz.devices('Operatorssensor').temperature, 1)
        dataTable[9] = dz.utils.round(dz.devices('CC1sensor').humidity, 1) 
        dataTable[10] = dz.utils.round(dz.devices('CC1sensor').temperature, 1)
        dataTable[11] = dz.utils.round(dz.devices('CC2sensor').humidity, 1) 
        dataTable[12] = dz.utils.round(dz.devices('CC2sensor').temperature, 1)
        dataTable[13] = dz.utils.round(dz.devices('Storagesensor').humidity, 1) 
        dataTable[14] = dz.utils.round(dz.devices('Storagesensor').temperature, 1)
        
        local output = '/home/pi/Desktop/GPV_data/Humidity'..dz.time.rawDate..'.csv' -- location/name of your csv text file
        local span   = '23:59:00'  -- time-span to be written to csv file
        -----------------------------------------------------------------------------------------------------------  
        -- No changes required below these lines
        -----------------------------------------------------------------------------------------------------------  
        
        local function makePersistentData()    
            local row = {}
            for index, value in ipairs(dataTable) do
                row[index] = value
            end          
            return row 
        end

        local function writeCSV()
            local relevantRows = dz.data.rows.subsetSince(span)
            local csvFile = assert(io.open(output, "w"))
            if header then csvFile:write(header ..'\n') end
           
            for index = #relevantRows, 1, -1 do 
                dz.log(table.concat(relevantRows[index].data, ','), dz.LOG_DEBUG) 
                csvFile:write(table.concat(relevantRows[index].data, ',') ..'\n')  
            end

            csvFile:close()
            dz.log((#relevantRows) .. ' data-rows written.', dz.LOG_DEBUG) 
        end
 
        -- main
        dz.data.rows.add(makePersistentData())  -- store in persistent historical data
        writeCSV()
    end
}
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: DzVents script to create CSV file for further processing

Post by waaren »

Mibeus wrote: Monday 03 May 2021 19:22 this is excellent.

Please can you help me for modification? I need log one month. I dont know what I need change for log more as 24 hours. Now I have this modification for more data limits but still after 24 hours stop recording.
does it really stops recording? you can check by looking at first and last entry in the data file.
(<domoticz dir>/scripts/dzVents/data/__data_<scriptname>.lua)

Can you try changing line

Code: Select all

            local relevantRows = dz.data.rows.subsetSince(span)
to

Code: Select all

            local relevantRows = dz.data.rows.subset(1)
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
Mibeus
Posts: 8
Joined: Sunday 29 November 2020 20:29
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: DzVents script to create CSV file for further processing

Post by Mibeus »

This is file

Code: Select all

Date,Time,SMT Humidity,SMT Temperature, THT Humidity,THT Temperature, Operators Humidity, Operators Temperature, CC1 Humidity, CC1 Temperature, CC2 Humidity, CC2 Temperature, Storage Humidity, Storage Temperature
2021-05-02,20:00,43.0,19.0,39.0,19.9,41.0,21.9,41.0,21.8,41.0,22.0,38.0,22.8
2021-05-02,21:00,43.0,18.8,39.0,19.6,41.0,21.7,41.0,21.5,41.0,21.9,38.0,22.7
2021-05-02,22:00,44.0,18.6,40.0,19.5,41.0,21.6,41.0,21.3,42.0,21.7,39.0,22.5
2021-05-02,23:00,44.0,18.4,40.0,19.3,41.0,21.4,41.0,21.1,42.0,21.5,39.0,22.4
2021-05-03,00:00,44.0,18.2,40.0,19.0,41.0,21.3,42.0,20.9,42.0,21.4,39.0,22.3
2021-05-03,01:00,44.0,18.0,40.0,18.9,41.0,21.2,42.0,20.8,42.0,21.2,39.0,22.2
2021-05-03,02:00,44.0,17.7,40.0,18.6,41.0,21.0,41.0,20.6,41.0,21.1,39.0,22.1
2021-05-03,03:00,44.0,17.6,39.0,18.5,41.0,20.9,41.0,20.5,41.0,21.0,38.0,22.0
2021-05-03,04:00,44.0,17.1,40.0,18.1,41.0,20.6,41.0,20.2,41.0,20.8,38.0,21.8
2021-05-03,05:00,44.0,17.0,40.0,18.0,41.0,20.4,41.0,20.1,41.0,20.6,38.0,21.6
2021-05-03,06:00,44.0,16.8,39.0,18.0,40.0,20.3,41.0,19.9,41.0,20.4,38.0,21.5
2021-05-03,07:00,41.0,18.5,37.0,19.2,40.0,20.7,40.0,20.8,40.0,20.8,37.0,21.6
2021-05-03,08:00,38.0,21.1,35.0,20.8,39.0,21.2,38.0,22.2,40.0,21.4,37.0,21.8
2021-05-03,09:00,36.0,22.5,33.0,22.0,38.0,21.7,37.0,22.7,38.0,22.0,36.0,22.1
2021-05-03,10:00,34.0,23.5,32.0,23.0,37.0,22.2,36.0,23.0,38.0,22.5,35.0,22.5
2021-05-03,11:00,32.0,24.4,29.0,23.8,36.0,22.7,35.0,23.6,37.0,22.9,34.0,22.8
2021-05-03,12:00,30.0,24.9,28.0,24.3,35.0,23.1,33.0,23.9,35.0,23.0,33.0,23.0
2021-05-03,13:00,29.0,25.1,27.0,24.5,34.0,23.0,32.0,24.8,35.0,23.1,32.0,23.2
2021-05-03,14:00,28.0,25.6,26.0,25.0,34.0,23.2,32.0,25.4,34.0,23.2,32.0,23.4
2021-05-03,15:00,28.0,26.2,26.0,25.5,34.0,23.5,31.0,25.3,35.0,22.5,31.0,23.6
2021-05-03,16:00,27.0,26.5,25.0,25.9,33.0,23.7,31.0,24.9,34.0,22.2,31.0,23.9
2021-05-03,17:00,27.0,26.7,24.0,26.0,33.0,23.9,30.0,24.9,34.0,22.5,30.0,24.0
2021-05-03,18:00,26.0,26.8,24.0,26.1,32.0,24.0,30.0,24.8,34.0,22.3,30.0,24.1
2021-05-03,19:00,25.0,26.6,23.0,26.0,31.0,24.0,29.0,24.6,33.0,22.0,29.0,24.0
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: DzVents script to create CSV file for further processing

Post by waaren »

Mibeus wrote: Monday 03 May 2021 20:53 This is file
This is the csv file. What are the first and last record of the persistent data file? (see my previous post)
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
Mibeus
Posts: 8
Joined: Sunday 29 November 2020 20:29
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: DzVents script to create CSV file for further processing

Post by Mibeus »

this is first rows

Code: Select all

-- Persistent Data
local multiRefObjects = {

} -- multiRefObjects
local obj1 = {
	["rows"] = {
and last

Code: Select all

}
return obj1
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: DzVents script to create CSV file for further processing

Post by waaren »

Mibeus wrote: Monday 03 May 2021 22:05 this is first rows
and last
Between these first and last rows you will find the actual records. I am interested in the first and last record. They will look similar to

Code: Select all

               [1] = {
                        ["time"] = "2021-5-3 22:0:0.168";
                        ["data"] = {
                                [1] = "2021-05-04";
                                [2] = "00:00";
                                [3] = 52.0;
                                [4] = 12.0;
                        };
                };


btw it is probably a bad idea to use persistent historical data for a months worth of hourly data of these sensors. It will slow down dzVents quite significant.
Performance wise it would probably be (much) better to trigger an offline OS script to do the work for you.
What is your domoticz version ?
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: DzVents script to create CSV file for further processing

Post by waaren »

waaren wrote: Tuesday 04 May 2021 0:13 Performance wise it would probably be (much) better to trigger an offline OS command to do the work for you.
New version without persistent Historical data

Code: Select all

--[[
    Requires:
        dzVents 3.1.0
        Linux based system with awk

    script to create a CSV file with following layout.
    now,inside,target,outside,lounge,greenhouse,heating  [header row]
    22:10,21.2,7.0,12.7,21.3,11.8,0   [oldest data: set by user]
    22:20,21.1,7.0,12.7,21.3,11.8,0   [next data: depends on script execution frequency]
    ...
    22:00,21.1,7.0,9.4,22.0,8.0,0     [latest data]

        History:
        20200108: Start coding
        20200110: Start testing on life situation (thanks to MikeF)
        20200122: First public release
        20210504: Do not use historical persistent data as it slows down domoticz when having large amount of records

]]--

local datafileHousekeepingTimerule = 'at 05:37' -- choose a quiet time on your system

return
{
    on =
    {
        timer =
        {
            'every hour',
            datafileHousekeepingTimerule,
        },
    },

    logging =
    {
        level = domoticz.LOG_DEBUG, -- set to LOG_ERROR when tested and OK
        marker = 'createCSV',
    },

    execute = function(dz,item,info)

        local row = {}

        -----------------------------------------------------------------------------------------------------------
        -- Enter header line of your csv text file (or nil if you don't need one )
        -- Enter devicenames / methods in the required order. This will determine what attribute-values will be written to the csvFile
        -- Enter location/name of your csv text file
        -- Enter time-span to be written to csv file in daysInCSV
        -----------------------------------------------------------------------------------------------------------
        local header =  'Date,Time,SMT Humidity,SMT Temperature, THT Humidity,THT Temperature, Operators Humidity, Operators Temperature, CC1 Humidity, CC1 Temperature, CC2 Humidity, CC2 Temperature, Storage Humidity, Storage Temperature'

        row[1] = dz.time.rawDate
        row[#row+1]= dz.time.rawTime:sub(1, 4) .. '0' -- consistent times even if script is not triggered precisely every x0 minutes
        row[#row+1]= dz.utils.round(dz.devices('SMTsensor').humidity, 1)
        row[#row+1]= dz.utils.round(dz.devices('SMTsensor').temperature, 1)
        row[#row+1]= dz.utils.round(dz.devices('THTsensor').humidity, 1)
        row[#row+1]= dz.utils.round(dz.devices('THTsensor').temperature, 1)
        row[#row+1]= dz.utils.round(dz.devices('Operatorssensor').humidity, 1)
        row[#row+1]= dz.utils.round(dz.devices('Operatorssensor').temperature, 1)
        row[#row+1]= dz.utils.round(dz.devices('CC1sensor').humidity, 1)
        row[#row+1] = dz.utils.round(dz.devices('CC1sensor').temperature, 1)
        row[#row+1] = dz.utils.round(dz.devices('CC2sensor').humidity, 1)
        row[#row+1] = dz.utils.round(dz.devices('CC2sensor').temperature, 1)
        row[#row+1] = dz.utils.round(dz.devices('Storagesensor').humidity, 1)
        row[#row+1] = dz.utils.round(dz.devices('Storagesensor').temperature, 1)

        local csvFile = '/home/pi/Desktop/GPV_data/Humidity' .. dz.time.rawDate .. '.csv' -- location/name of your csv text file
        local daysInCSV = 31

        -----------------------------------------------------------------------------------------------------------
        -- No changes required below these lines
        -----------------------------------------------------------------------------------------------------------
        local dataFile = 'scripts/dzVents/data/__data_' ..  info.scriptName

        local function appendRow2Data(row)
            local out = assert(io.open(dataFile, "a"))
            out:write(table.concat(row, ',') ..'\n')
            out:close()
        end

        local function cleanDatafile(cutofDate)
            local tmp = os.tmpname()
            local cmd =  'awk -v d="' .. cutofDate .. '" \'$1 > d\' ' .. dataFile .. ' >> '  .. tmp .. '; mv -f ' .. tmp .. ' ' .. dataFile
            dz.executeShellCommand(cmd)
        end

        local function createCSVfile(fromDate)
            local out = assert(io.open(csvFile, "w"))
            local cmd =  'awk -v d="' .. fromDate .. '" \'$1 > d\' ' .. dataFile .. ' >> '  .. csvFile -- strips old dates from data
            if header then
                out:write(header ..'\n')
                out:close()
            end
            dz.executeShellCommand(cmd)
        end

        local function makeFromDate(d)
            return dz.time.addDays(-d).rawDate
        end

        -- main
        if dz.time.matchesRule(datafileHousekeepingTimerule) then
            cleanDatafile(makeFromDate(daysInCSV + 30 ) )
        else
            appendRow2Data(row)
            createCSVfile(makeFromDate(daysInCSV))
        end
    end
}
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
Mibeus
Posts: 8
Joined: Sunday 29 November 2020 20:29
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: DzVents script to create CSV file for further processing

Post by Mibeus »

Here is full script :)

Code: Select all

-- Persistent Data
local multiRefObjects = {

} -- multiRefObjects
local obj1 = {
	["rows"] = {
		[1] = {
			["data"] = {
				[1] = "2021-05-04";
				[2] = "15:00";
				[3] = 30.0;
				[4] = 26.5;
				[5] = 29.0;
				[6] = 26.0;
				[7] = 37.0;
				[8] = 25.2;
				[9] = 33.0;
				[10] = 25.9;
				[11] = 38.0;
				[12] = 23.6;
				[13] = 36.0;
				[14] = 24.9;
			};
			["time"] = "2021-5-4 13:0:0.50";
		};
		[2] = {
			["data"] = {
				[1] = "2021-05-04";
				[2] = "14:00";
				[3] = 31.0;
				[4] = 26.6;
				[5] = 29.0;
				[6] = 26.1;
				[7] = 39.0;
				[8] = 25.3;
				[9] = 34.0;
				[10] = 26.1;
				[11] = 38.0;
				[12] = 23.6;
				[13] = 38.0;
				[14] = 24.8;
			};
			["time"] = "2021-5-4 12:0:0.129";
		};
		[3] = {
			["data"] = {
				[1] = "2021-05-04";
				[2] = "13:00";
				[3] = 31.0;
				[4] = 27.0;
				[5] = 30.0;
				[6] = 26.4;
				[7] = 42.0;
				[8] = 24.9;
				[9] = 36.0;
				[10] = 25.7;
				[11] = 39.0;
				[12] = 24.0;
				[13] = 40.0;
				[14] = 24.6;
			};
			["time"] = "2021-5-4 11:0:0.140";
		};
		[4] = {
			["data"] = {
				[1] = "2021-05-04";
				[2] = "12:00";
				[3] = 33.0;
				[4] = 25.7;
				[5] = 34.0;
				[6] = 25.0;
				[7] = 64.0;
				[8] = 21.4;
				[9] = 39.0;
				[10] = 24.3;
				[11] = 41.0;
				[12] = 23.1;
				[13] = 45.0;
				[14] = 24.1;
			};
			["time"] = "2021-5-4 10:0:0.314";
		};
		[5] = {
			["data"] = {
				[1] = "2021-05-04";
				[2] = "11:00";
				[3] = 31.0;
				[4] = 25.7;
				[5] = 32.0;
				[6] = 24.9;
				[7] = 37.0;
				[8] = 24.0;
				[9] = 36.0;
				[10] = 24.2;
				[11] = 45.0;
				[12] = 23.1;
				[13] = 41.0;
				[14] = 23.8;
			};
			["time"] = "2021-5-4 9:0:0.195";
		};
		[6] = {
			["data"] = {
				[1] = "2021-05-04";
				[2] = "10:00";
				[3] = 29.0;
				[4] = 25.7;
				[5] = 28.0;
				[6] = 25.1;
				[7] = 33.0;
				[8] = 24.3;
				[9] = 32.0;
				[10] = 24.3;
				[11] = 35.0;
				[12] = 22.5;
				[13] = 33.0;
				[14] = 23.9;
			};
			["time"] = "2021-5-4 8:0:0.176";
		};
		[7] = {
			["data"] = {
				[1] = "2021-05-04";
				[2] = "09:00";
				[3] = 28.0;
				[4] = 25.0;
				[5] = 27.0;
				[6] = 24.4;
				[7] = 32.0;
				[8] = 23.9;
				[9] = 31.0;
				[10] = 23.7;
				[11] = 34.0;
				[12] = 22.3;
				[13] = 32.0;
				[14] = 23.7;
			};
			["time"] = "2021-5-4 7:0:0.202";
		};
		[8] = {
			["data"] = {
				[1] = "2021-05-04";
				[2] = "08:00";
				[3] = 28.0;
				[4] = 24.2;
				[5] = 26.0;
				[6] = 23.6;
				[7] = 31.0;
				[8] = 23.4;
				[9] = 31.0;
				[10] = 23.0;
				[11] = 33.0;
				[12] = 22.7;
				[13] = 31.0;
				[14] = 23.4;
			};
			["time"] = "2021-5-4 6:0:0.229";
		};
		[9] = {
			["data"] = {
				[1] = "2021-05-04";
				[2] = "07:00";
				[3] = 28.0;
				[4] = 23.5;
				[5] = 26.0;
				[6] = 22.9;
				[7] = 31.0;
				[8] = 23.0;
				[9] = 31.0;
				[10] = 22.8;
				[11] = 32.0;
				[12] = 23.1;
				[13] = 30.0;
				[14] = 23.3;
			};
			["time"] = "2021-5-4 5:0:0.267";
		};
		[10] = {
			["data"] = {
				[1] = "2021-05-04";
				[2] = "06:00";
				[3] = 28.0;
				[4] = 23.3;
				[5] = 26.0;
				[6] = 22.6;
				[7] = 31.0;
				[8] = 22.9;
				[9] = 31.0;
				[10] = 22.8;
				[11] = 32.0;
				[12] = 22.9;
				[13] = 30.0;
				[14] = 23.2;
			};
			["time"] = "2021-5-4 4:0:0.295";
		};
		[11] = {
			["data"] = {
				[1] = "2021-05-04";
				[2] = "05:00";
				[3] = 28.0;
				[4] = 23.2;
				[5] = 26.0;
				[6] = 22.7;
				[7] = 31.0;
				[8] = 22.9;
				[9] = 31.0;
				[10] = 22.9;
				[11] = 32.0;
				[12] = 23.0;
				[13] = 30.0;
				[14] = 23.3;
			};
			["time"] = "2021-5-4 3:0:0.337";
		};
		[12] = {
			["data"] = {
				[1] = "2021-05-04";
				[2] = "04:00";
				[3] = 28.0;
				[4] = 23.2;
				[5] = 26.0;
				[6] = 22.7;
				[7] = 32.0;
				[8] = 23.0;
				[9] = 31.0;
				[10] = 22.8;
				[11] = 32.0;
				[12] = 23.0;
				[13] = 30.0;
				[14] = 23.3;
			};
			["time"] = "2021-5-4 2:0:0.350";
		};
		[13] = {
			["data"] = {
				[1] = "2021-05-04";
				[2] = "03:00";
				[3] = 28.0;
				[4] = 22.9;
				[5] = 27.0;
				[6] = 22.4;
				[7] = 32.0;
				[8] = 23.0;
				[9] = 31.0;
				[10] = 22.9;
				[11] = 32.0;
				[12] = 23.9;
				[13] = 30.0;
				[14] = 23.3;
			};
			["time"] = "2021-5-4 1:0:0.428";
		};
		[14] = {
			["data"] = {
				[1] = "2021-05-04";
				[2] = "02:00";
				[3] = 28.0;
				[4] = 22.9;
				[5] = 27.0;
				[6] = 22.4;
				[7] = 32.0;
				[8] = 23.1;
				[9] = 31.0;
				[10] = 23.1;
				[11] = 33.0;
				[12] = 23.4;
				[13] = 31.0;
				[14] = 23.4;
			};
			["time"] = "2021-5-4 0:0:0.489";
		};
		[15] = {
			["data"] = {
				[1] = "2021-05-04";
				[2] = "01:00";
				[3] = 28.0;
				[4] = 23.2;
				[5] = 27.0;
				[6] = 22.6;
				[7] = 32.0;
				[8] = 23.2;
				[9] = 31.0;
				[10] = 23.6;
				[11] = 33.0;
				[12] = 23.0;
				[13] = 31.0;
				[14] = 23.5;
			};
			["time"] = "2021-5-3 23:0:0.461";
		};
		[16] = {
			["data"] = {
				[1] = "2021-05-04";
				[2] = "00:00";
				[3] = 28.0;
				[4] = 23.9;
				[5] = 26.0;
				[6] = 23.2;
				[7] = 32.0;
				[8] = 23.3;
				[9] = 31.0;
				[10] = 24.1;
				[11] = 33.0;
				[12] = 23.3;
				[13] = 31.0;
				[14] = 23.7;
			};
			["time"] = "2021-5-3 22:0:0.540";
		};
		[17] = {
			["data"] = {
				[1] = "2021-05-03";
				[2] = "23:00";
				[3] = 27.0;
				[4] = 24.5;
				[5] = 26.0;
				[6] = 23.7;
				[7] = 32.0;
				[8] = 23.5;
				[9] = 31.0;
				[10] = 23.7;
				[11] = 33.0;
				[12] = 23.7;
				[13] = 30.0;
				[14] = 23.7;
			};
			["time"] = "2021-5-3 21:0:0.34";
		};
		[18] = {
			["data"] = {
				[1] = "2021-05-03";
				[2] = "22:10";
				[3] = 27.0;
				[4] = 25.0;
				[5] = 25.0;
				[6] = 24.4;
				[7] = 32.0;
				[8] = 23.6;
				[9] = 31.0;
				[10] = 23.1;
				[11] = 33.0;
				[12] = 22.6;
				[13] = 30.0;
				[14] = 23.8;
			};
			["time"] = "2021-5-3 20:11:0.225";
		};
		[19] = {
			["data"] = {
				[1] = "2021-05-03";
				[2] = "22:10";
				[3] = 27.0;
				[4] = 25.0;
				[5] = 25.0;
				[6] = 24.4;
				[7] = 32.0;
				[8] = 23.6;
				[9] = 31.0;
				[10] = 23.0;
				[11] = 33.0;
				[12] = 22.6;
				[13] = 30.0;
				[14] = 23.8;
			};
			["time"] = "2021-5-3 20:10:0.294";
		};
		[20] = {
			["data"] = {
				[1] = "2021-05-03";
				[2] = "22:00";
				[3] = 27.0;
				[4] = 25.0;
				[5] = 25.0;
				[6] = 24.4;
				[7] = 32.0;
				[8] = 23.6;
				[9] = 31.0;
				[10] = 23.0;
				[11] = 34.0;
				[12] = 22.6;
				[13] = 30.0;
				[14] = 23.8;
			};
			["time"] = "2021-5-3 20:9:0.193";
		};
		[21] = {
			["data"] = {
				[1] = "2021-05-03";
				[2] = "22:00";
				[3] = 27.0;
				[4] = 25.0;
				[5] = 25.0;
				[6] = 24.4;
				[7] = 32.0;
				[8] = 23.6;
				[9] = 30.0;
				[10] = 23.0;
				[11] = 34.0;
				[12] = 22.6;
				[13] = 30.0;
				[14] = 23.8;
			};
			["time"] = "2021-5-3 20:8:0.167";
		};
	};
}
return obj1
Mibeus
Posts: 8
Joined: Sunday 29 November 2020 20:29
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: DzVents script to create CSV file for further processing

Post by Mibeus »

Hello this edit works, now saving continous over day.

Best solution will be save this data in internal database, but this is over me :(.
Now saveing every 5 minutes into sqllite. But I need save only one per hour but during 2 years :(.

waaren wrote: Monday 03 May 2021 20:42
Mibeus wrote: Monday 03 May 2021 19:22 this is excellent.

Please can you help me for modification? I need log one month. I dont know what I need change for log more as 24 hours. Now I have this modification for more data limits but still after 24 hours stop recording.
does it really stops recording? you can check by looking at first and last entry in the data file.
(<domoticz dir>/scripts/dzVents/data/__data_<scriptname>.lua)

Can you try changing line

Code: Select all

            local relevantRows = dz.data.rows.subsetSince(span)
to

Code: Select all

            local relevantRows = dz.data.rows.subset(1)
My version of Domoticz is
Version: 2020.2
Build Hash: b63341bc0
Compile Date: 2020-04-26 13:47:55
dzVents Version: 3.0.2
Python Version: 3.7.3
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: DzVents script to create CSV file for further processing

Post by waaren »

Mibeus wrote: Wednesday 05 May 2021 0:21 Best solution will be save this data in internal database, but this is over me :(.
Now saving every 5 minutes into sqlite. But I need save only one per hour but during 2 years :(.
You might want to look at influxDB. Seems to be an exact match for your requirements.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest