Spike in Enphase data

Topics (not sure which fora)
when not sure where to post, post here and mods will move it to right forum.

Moderators: leecollings, remb0

Post Reply
dbennik
Posts: 5
Joined: Saturday 04 April 2020 20:35
Target OS: Raspberry Pi / ODroid
Domoticz version: 2024.7
Location: Netherlands
Contact:

Spike in Enphase data

Post by dbennik »

Version: 2024.7
Platform: Raspberry Pi (Raspbian)
Plugin/Hardware: Enphase Envoy with LAN (HTTP) interface (Version: D7.6.175)
Description:

In the past i had some issues with the counter value (total) received from the Enphase Envoy was reset to 0 at some point. This would result in negative spikes (drops) in the collected data. Resolved this by deleting the spike values.

Now i have a new issue. A sudden jump in the counter value (total) received from the Enphase Envoy (so it seems). I could also remove the spike values to correct most graphs, but i still have a spike in the month compare graph which i cant seem to remove? Is this because it is in the current month and will it be corrected at the end of this month?

And does anyone have already encountered this behaviour with the Enphase Envoy. Sudden resets of the total counter to 0 as well as jumps up?
domoticz-pv-comparemonths.png
domoticz-pv-comparemonths.png (27.21 KiB) Viewed 1082 times
domoticz-pv-table1.png
domoticz-pv-table1.png (67.86 KiB) Viewed 1082 times
domoticz-pv-table2.png
domoticz-pv-table2.png (16.55 KiB) Viewed 1082 times
User avatar
gizmocuz
Posts: 2352
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi / ODroid
Domoticz version: beta
Location: Top of the world
Contact:

Re: Spike in Enphase data

Post by gizmocuz »

The reset sometimes happens when you have a power outage or a firmware update.
For this reason, Domoticz keeps internally a counter offset, so when this happens, it should continue to count

There is a uservariable with the name 'EnphaseOffset_Production_xxxxxxx' with the offset

Did something happened? Power outage, firmware upgrade?
Quality outlives Quantity!
User avatar
gizmocuz
Posts: 2352
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi / ODroid
Domoticz version: beta
Location: Top of the world
Contact:

Re: Spike in Enphase data

Post by gizmocuz »

As a matter of fact, I have the same issue at 8 august.
I remember I have read that Enphase was forcing an update to all systems because some Dutch researchers have found a vulnerability in the system

https://www.divd.nl/newsroom/articles/d ... to-vendor/
Quality outlives Quantity!
dbennik
Posts: 5
Joined: Saturday 04 April 2020 20:35
Target OS: Raspberry Pi / ODroid
Domoticz version: 2024.7
Location: Netherlands
Contact:

Re: Spike in Enphase data

Post by dbennik »

You might be right about the resets, I think I had the problem with my own scripts to update other meters I have (to calculate the direct used PV and real energy usage). These scripts did not handle the resets to 0.

And do you know about the month compare graph, were you able to fix that one? Other graphs I was able to fix by deleting the incorrect day values. But the month compare graph keeps showing the spike for the month total.

The report table shows the corrected value though, will the graph be fixed after the month ends? I don’t know how the value for the month total is calculated, by diff of the counter for start and end of month, or by summing all day values of the month.

The graph seems to be doing the first (at least for the current month) and the month report the last.
redhouse
Posts: 6
Joined: Tuesday 10 August 2021 18:02
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Spike in Enphase data

Post by redhouse »

I have the exact same issue and managed to resolve some of the erroneous values by manipulating the database using DB SQL Browser Lite. Mainly Meter, Meter_Calendar and Device_Status. I'm also interested on how the monthly value is being calculated. I have run some tests on a test system, and it looks like the counter counts "Current Value" - "Start of Month Value" and is updated every 5 minutes to show the actual production during the month. I have also tried changing values in between (both positive values and negative values to compensate for the sudden jump) but that doesn't change anything in the graph.
User avatar
FireWizard
Posts: 1747
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: Spike in Enphase data

Post by FireWizard »

Be aware that last year several vulnerabilities has been discovered, which has been patched rather recently by Enphase.

Since years I used firmware D5, which has been updated to D7 last Friday night.
This caused an interruption of the data stream, because of the authorization.
When this had been fixed, I noticed a spike as well.

Tonight I received another update from D7 to D8. I have received firmware version: D8.2.4264.
Also I noticed a spike again.

So, obviously, because the vulnerabilities has been fixed, they push now updates to the various systems.
This may cause the spikes.

Regards
microkid
Posts: 8
Joined: Tuesday 29 March 2016 18:30
Target OS: Raspberry Pi / ODroid
Domoticz version: 2024.7
Location: NL
Contact:

Re: Spike in Enphase data

Post by microkid »

Same here. I already manually modified the meter data (monthly), but the next day, the spike is back. Wondering how to solve this permanently.
User avatar
gizmocuz
Posts: 2352
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi / ODroid
Domoticz version: beta
Location: Top of the world
Contact:

Re: Spike in Enphase data

Post by gizmocuz »

Maybe by adjusting the uservariable
Quality outlives Quantity!
microkid
Posts: 8
Joined: Tuesday 29 March 2016 18:30
Target OS: Raspberry Pi / ODroid
Domoticz version: 2024.7
Location: NL
Contact:

Re: Spike in Enphase data

Post by microkid »

gizmocuz wrote: Thursday 22 August 2024 8:43 Maybe by adjusting the uservariable
Can you eleborate a little bit more please?
User avatar
gizmocuz
Posts: 2352
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi / ODroid
Domoticz version: beta
Location: Top of the world
Contact:

Re: Spike in Enphase data

Post by gizmocuz »

Well, I was going to write something, but I did some internal testing, and this is not going to work.
Sure, I could 'detect' the counter reset, but then again, I am having issues with the compare graph at the bottom

There is a user variable with the counter offset, this should be lowered to the correct value.
Then you need to patch the counter values in the database (meter_calendar) (and the huge usage spike at the specific day)

Would be nice if we could solve this internally somehow

The uservariable you can find by going to Setup->More Options->User Variables, and in the search, box enter 'enphase'
But I would suggest correcting that in your database backup as well, so when you restore your patched database, all should be ok

Now why this happend beats me at the moment, because I try to solve this by using a special internal counter (See EnphaseAPI.cpp, ~line 853)
When this counter is lower than the previous value, a reset has occurred, and the new counter offset (the user variable) should be the last corrected counter value

Seems like something is going wrong here:

CounterHelper::SendKwhMeter

As this is used on other places as well, hope someone can have a look at this.

Best to create a github issue with all this information as I don't browse the forum that much

Code: Select all

rowid	DeviceRowID	Value	Date	Counter	Price
107941	8023	26919	2024-08-06 00:00:00.000	3780803	5.2376
108002	8023	17667	2024-08-07 00:00:00.000	3798470	3.3184
108064	8023	7182913	2024-08-08 00:00:00.000	10981383	1454.4884
108126	8023	14854	2024-08-09 00:00:00.000	10996237	2.3638
*** Wait ***

Something else is going on... the 'counter' received from Enphase has a big jump!!! It was not reset...

My current value is

"whLifetime": 7781778

my uservariable still has a value of

3578.851000

This because I has some resets in the past when I turned of the power.

I wonder what happens if I turned off the power from the Envoy gateway, will it reset to 0 again? And will this correct the counter again the next day?
Still some patching up to do in the database

(No time to test this, maybe someone else could)

We could use a minus value for the uservariable (in my case probably subtract 7182913)
Quality outlives Quantity!
dbennik
Posts: 5
Joined: Saturday 04 April 2020 20:35
Target OS: Raspberry Pi / ODroid
Domoticz version: 2024.7
Location: Netherlands
Contact:

Re: Spike in Enphase data

Post by dbennik »

I had the same thought about the jump.

I had several resets to 0 in the past as well and thus a offset user variable > 0 filled in.

It now seems that the envoy has made up for the resets in the past and started reporting the actual usage counter again (maybe Enphase fixed this?). My current lifetime production is 5,7Mwh at this point. The offset had 2,4Mwh and the counter is now at 8,1Mwh.

So when I reset the offset variable to 0 it seems that my envoy is reporting the correct counter again (which seems like a jump because it made up for the resets).
User avatar
gizmocuz
Posts: 2352
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi / ODroid
Domoticz version: beta
Location: Top of the world
Contact:

Re: Spike in Enphase data

Post by gizmocuz »

Found a way to fix it, but it involves some steps:

Steps performed to fix the jump (A huge jump up that is!!!, very rare occasion!!):

- press 'Edit' on the Enphase kWh delivery sensor and notice the IDX
- shutdown Domoticz, cleanly
- copy the database domoticz.db (if there are domoticz.db-wal and domoticz.db-shm files, also copy these!)
- make also a backup/zip, so you can always go back!!!
- open the database with an editor like (for example) SQLite Expert Personal edition
- go a sql query editor and execute

DELETE FROM Meter WHERE DeviceRowID = (above found IDX), for example

DELETE FROM Meter WHERE DeviceRowID = 8023

- open the Meter_Calendar table, filter all records where DeviceRowID = (above found IDX)

- find the moment where the big jump is, you will see the 'value' column have a HUGE value, copy/note this value, this is your 'offset' that you need to subtract from the 'counter' column from that point

For example my view:

Code: Select all

rowid	DeviceRowID	Value	Date	Counter	Price
107941	8023	26919	2024-08-06 00:00:00.000	3780803	5.2376
108002	8023	17667	2024-08-07 00:00:00.000	3798470	3.3184
108064	8023	7182913	2024-08-08 00:00:00.000	10981383	1454.4884
108126	8023	14854	2024-08-09 00:00:00.000	10996237	2.3638
108187	8023	24588	2024-08-10 00:00:00.000	11020825	3.1892
108248	8023	28569	2024-08-11 00:00:00.000	11049394	3.8624
108309	8023	29174	2024-08-12 00:00:00.000	11078568	5.591
108370	8023	26394	2024-08-13 00:00:00.000	11104962	6.3272
...
...
7182913 is my 'offset' to subtract from that day onwards in the counter field, till today/yesterday, so all records from that day!
(I let excel help me with this)

The moment where this is happening and you have a huge 'value', set this value to 0, and also set the 'price' column to zero, when you have done this, it will look like:

Code: Select all

rowid	DeviceRowID	Value	Date	Counter	Price
107941	8023	26919	2024-08-06 00:00:00.000	3780803	5.2376
108002	8023	17667	2024-08-07 00:00:00.000	3798470	3.3184
108064	8023	0	2024-08-08 00:00:00.000	3798470	0
108126	8023	14854	2024-08-09 00:00:00.000	3813324	2.3638
108187	8023	24588	2024-08-10 00:00:00.000	3837912	3.1892
108248	8023	28569	2024-08-11 00:00:00.000	3866481	3.8624
108309	8023	29174	2024-08-12 00:00:00.000	3895655	5.591
108370	8023	26394	2024-08-13 00:00:00.000	3922049	6.3272
108431	8023	9123	2024-08-14 00:00:00.000	3931172	2.4712
...
...
109285	8023	27285	2024-08-28 00:00:00.000	4204920	6.0352
(If you are unable to do this, or it is too complicate, too much work, you can also 'delete' all records from the moment where you have that 'huge' value), but do this at least for the last value, you will need this corrected counter value in the next step


- take a notice of the last corrected counter value, in my case '4204920'

- next, open the 'DeviceStatus' table, and make a filter/find your device (ID = (above id)

- change the 'sValue' so after the semicolon ( ; ) you change that value with the above value (4204920.000)

- next, open the 'UserVariables' table, and find your enphase offset (name should be something like 'EnphaseOffset_Production_xxxxxx')

- If your offset value above was 7182913, divide this by 1000, so you will get 7182.913

- subtract the uservariable with this, 3578.851000 - 7182.913 = -3604.062

- set the uservariable to this value (-3604.062)

- now close the editor, and copy the database back to your Domoticz folder (in the correct folder if you use a docker instance)

- start Domoticz again

I think we should be able to make a python script for this that detects the spike, and performs the above steps
But my python skills are limited

And I also should be able to detect the jump in the c++ code in the CounterHelper class.

Probably this addition will work, but maybe someone could test this offline

Code: Select all

void CounterHelper::SendKwhMeter(const int NodeID, const int ChildID, const int BatteryLevel, const double musage, double mtotal, const std::string& defaultname, int RssiLevel)
{
	if (mtotal != 0)
	{
		double rTotal = m_CounterOffset + mtotal;

		bool bHaveCounterIssue = false;

		if (
			(rTotal < m_nLastCounterValue)
			&& (m_nLastCounterValue != 0)
			)
		{
			//Possible counter reset (newly received counter value is less then previous recieved!)
			m_CounterOffset = m_nLastCounterValue;
			bHaveCounterIssue = true;
		}
		else if (m_nLastCounterValue > 10 && rTotal > 10)
		{
			constexpr double m_allowed_cntr_jump_perc = 500; //We consider a 500% jump 'not' normal... what's a good value?
			if (abs((100.0 / m_nLastCounterValue) * rTotal) > m_allowed_cntr_jump_perc)
			{
				//Possible counter jump up!?
				m_CounterOffset = (m_nLastCounterValue - mtotal);
				bHaveCounterIssue = true;
			}
		}
		if (bHaveCounterIssue)
		{
			m_sql.safe_query("UPDATE UserVariables SET Value='%q', LastUpdate='%s' WHERE (Name=='%q')", std::to_string(m_CounterOffset).c_str(), TimeToString(nullptr, TF_DateTime).c_str(), m_szUservariableName.c_str());
			rTotal = m_CounterOffset + mtotal;
		}
		m_pHardwareBase->SendKwhMeter(NodeID, ChildID, BatteryLevel, musage, static_cast<double>(rTotal), defaultname, RssiLevel);
		m_nLastCounterValue = rTotal;
	}
}
Quality outlives Quantity!
dbennik
Posts: 5
Joined: Saturday 04 April 2020 20:35
Target OS: Raspberry Pi / ODroid
Domoticz version: 2024.7
Location: Netherlands
Contact:

Re: Spike in Enphase data

Post by dbennik »

Thanks for the information :)

Following your steps to correct the spike i managed to do so. I just have one strange thing i can not really explain.

The month total in the graph is still showing 24kwh above what it should be. The DB values all add up to the correct amount and when i click to the report, the table and the graph there also show the correct amount.

How is the month overview graph calculating the month values and how can it be that it is still too high?

Month graph too high:
month_graph_too_high.png
month_graph_too_high.png (47.11 KiB) Viewed 708 times
Report correct:
report_correct.png
report_correct.png (132.73 KiB) Viewed 708 times
User avatar
gizmocuz
Posts: 2352
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi / ODroid
Domoticz version: beta
Location: Top of the world
Contact:

Re: Spike in Enphase data

Post by gizmocuz »

It's calculated with the counter value.
Quality outlives Quantity!
dbennik
Posts: 5
Joined: Saturday 04 April 2020 20:35
Target OS: Raspberry Pi / ODroid
Domoticz version: 2024.7
Location: Netherlands
Contact:

Re: Spike in Enphase data

Post by dbennik »

That is what i thought it would be, but when i calculate it myself with the counter value, it still should be 424.32 and the year overview per month on the main meter page still shows 448.31

Below my Meter_Calendar data for the Enphase meter.

It should be 3319314 (2024-08-31) - 2894942 (2024-07-31) = 424372 (= 424,327). This is the correct value and is shown on the report page.

So i am a bit in the dark to why the main page month graph still shows 448.31 for august.

Edit: Even when i add up all the Value column values for august it results in 424372.

Code: Select all

rowid	DeviceRowID	Value	Counter	Date	Price
11415	172	13707	3349757	2024-09-02 00:00:00.000	0
11398	172	16736	3336050	2024-09-01 00:00:00.000	0
11381	172	16974	3319314	2024-08-31 00:00:00.000	0
11364	172	16114	3302340	2024-08-30 00:00:00.000	0
11347	172	13243	3286226	2024-08-29 00:00:00.000	0
11330	172	18113	3272983	2024-08-28 00:00:00.000	0
11313	172	18766	3254870	2024-08-27 00:00:00.000	0
11296	172	12407	3236104	2024-08-26 00:00:00.000	0
11279	172	13345	3223697	2024-08-25 00:00:00.000	0
11262	172	6226	3210352	2024-08-24 00:00:00.000	0
11245	172	9954	3204126	2024-08-23 00:00:00.000	0
11228	172	11117	3194172	2024-08-22 00:00:00.000	0
11211	172	14538	3183055	2024-08-21 00:00:00.000	0
11194	172	6120	3168517	2024-08-20 00:00:00.000	0
11177	172	16407	3162397	2024-08-19 00:00:00.000	0
11160	172	13379	3145990	2024-08-18 00:00:00.000	0
11431	172	19300	3132611	2024-08-17 00:00:00.000	0
11126	172	4642	3113311	2024-08-16 00:00:00.000	0
11109	172	14240	3132611	2024-08-15 00:00:00.000	0
11092	172	5791	3094429	2024-08-14 00:00:00.000	0
11075	172	11926	3088638	2024-08-13 00:00:00.000	0
11058	172	19116	3076712	2024-08-12 00:00:00.000	0
11041	172	19504	3057596	2024-08-11 00:00:00.000	0
11024	172	16944	3038092	2024-08-10 00:00:00.000	0
11007	172	11178	3021148	2024-08-09 00:00:00.000	0
10990	172	18398	3009970	2024-08-08 00:00:00.000	0
10973	172	15623	2991572	2024-08-07 00:00:00.000	0
10956	172	17981	2975949	2024-08-06 00:00:00.000	0
10939	172	19259	2957968	2024-08-05 00:00:00.000	0
10922	172	10924	2938709	2024-08-04 00:00:00.000	0
10905	172	10550	2927785	2024-08-03 00:00:00.000	0
10888	172	15550	2917235	2024-08-02 00:00:00.000	0
10871	172	6743	2901685	2024-08-01 00:00:00.000	0
10854	172	12173	2894942	2024-07-31 00:00:00.000	0
10837	172	19071	2882769	2024-07-30 00:00:00.000	0
Hansbit
Posts: 36
Joined: Monday 17 February 2014 15:10
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Spike in Enphase data

Post by Hansbit »

Thanks! That procedure solved it for me. My firmware was updated to D8, that caused the counter-jump.

(I'm happy with D8 because it solves the Enphase problem https://support.enphase.com/s/question/ ... 5712522212 when switching production off and on again, the panels won't start producing until the next morning. I have to test that next week).

Thanks, Hans
User avatar
gizmocuz
Posts: 2352
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi / ODroid
Domoticz version: beta
Location: Top of the world
Contact:

Re: Spike in Enphase data

Post by gizmocuz »

Strange, as I do this via Domoticz (Domoticz has a switch for this, so there is no need to perform this action in the web interface), and this works like a charm.
Made a script that does this automatically... (a few times a year)
Quality outlives Quantity!
Hansbit
Posts: 36
Joined: Monday 17 February 2014 15:10
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Spike in Enphase data

Post by Hansbit »

gizmocuz wrote: Monday 09 September 2024 7:22 Strange, as I do this via Domoticz (Domoticz has a switch for this, so there is no need to perform this action in the web interface), and this works like a charm.
Made a script that does this automatically... (a few times a year)
Problem existed for more than a year. Didn't matter if I switched via Domoticz or via web-interface, the panels won't startup again until next morning. Only restarted when I temporarily disconnected the panel (DC) from the micro inverter, but too much work on the roof :D . (Temporarily disconnect is same as dark at night; no DC). Support from NL and VS tested remote on my system. So hopefully solved in D8 (Sorry for off-topic reply).
Hansbit
Posts: 36
Joined: Monday 17 February 2014 15:10
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Spike in Enphase data

Post by Hansbit »

That's weird! This morning (edited all counters yesterday) the spike is back!

What to do now?
Scherm­afbeelding 2024-09-09 om 11.16.59.jpg
Scherm­afbeelding 2024-09-09 om 11.16.59.jpg (103.01 KiB) Viewed 625 times
freekdk
Posts: 13
Joined: Monday 01 February 2021 16:33
Target OS: Linux
Domoticz version: V2024.7
Location: Netherlands
Contact:

Re: Spike in Enphase data

Post by freekdk »

I did have the same type of problem. Because I started collecting data from Enphase right after the installation, the begin value of the Counter in Meter_Calendar starts at 1 (made it so). I made the data in Value and Counter consistent, so the counter is value+previous-counter. The last counter (of today) is consistent with the reported value of the total produced kWh on the webpage.

In another script I changed the values in table Meter items Value to be equal at midnight with the last counter in table Meter_Calendar and keep the difference of these values the same as before. I also changed item sValue in table DeviceStatus to the last value in table Meter_calendar. This was done between stopping Domoticz and starting it.

After that I saw new values in Meter with the right value in Value, in fact the total Wh produced. However at 05:20 suddenly a much higher value appeared, producing again a spike in the graphics. Reading in this subject I got the idea it is caused by the value in UserVariables, but when I calculate the difference they do not match. The totalWh should be 5256837, but is 7666212, difference 2409375. The value in UserVariables of EnphaseOffset_Production_12 is 2386.086000.

Question: what to put in UserVariables as offset and does that solve my problem?
Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest