New Brultech Energy Monitoring Model (ECM-1240) due in 2009

[Btech]

tenholde, I'd like to get a copy of your app. I've been running ECMServer for a couple of days sampling at 1 second and I've accumulated 17MB of data. I started to look at the information in Excel and it chokes on that much data. (The graphs can only show 30,000 points = ~8 hours and you can't zoom in to look at the detail). In looking for a solution I remembered experimenting with Splunk at work (www.splunk.com). It can read/parse/graph/analyze data from a multitude of inputs. One of them is UDP packets being sent to it. My hope is that with tenholde's software I can send the data and use splunk to keep the data and to view it.

How are other people dealing with viewing/analyzing this much data?

If you want to ease on the data size, you can reduce the packet send frequency to something like every 15 or 30 seconds and setup the power trigger so that only a change in power of x number of Watt will cause an immediate packet send to occur. After all, if it's 4am and everyone is sleeping and not much going on, there is no use in accumulating 1 sec interval data. The amount of power change which will send a packet immediately may be set to a desired power value (usually 30 to 100W depending on circumstance/application). These values may be set using the configuration software.

Paul

 

[WayneW]

Paul, any ETA for Pkg A (no radio)? Are there plans for a CT discount price like on the previous model?

What are the pros/cons of using a pair of CTs on Ch 1 & 2 versus using a Y-SPLIT pair on Ch 1 and "something else" (another Y-SPLIT?) on Ch 2? I assume having the two legs monitored separately makes it easier to spot load changes on one leg and "know" what device it was?

 

[mdiehl]

A little O.T., but what shipment method is Brueltech using, and can we get a tracking #?

 

[hucker]

A little O.T., but what shipment method is Brueltech using, and can we get a tracking #?

Mine was sent via the mail from Canada, this is pretty slow, it took 6 business days to get here.

 

[Brightan]

How are other people dealing with viewing/analyzing this much data?

I wrote a simple program to save to an Access DB and choose the following compromise in amount of data vs DB size...
There are 4 tables in the DB, seconds, minutes, hours and days
seconds stores every second but only keeps the last 600 (10 minutes)
minutes stores the total for each minute but only keeps the last 720 (12 hours)
hours stores the total for each hour but only keeps the last 720 (30 days)
days stores the total for each day but only keeps the last 365 (1 year)

This way there is only one file to access and it will never get too large.

if could be argued that there should be enough seconds to fill an hour (3600) and enough minutes to fill a day (1440) but the record size becomes a little large at that point and I am not sure that graphing 3600 points is useful.

 

[bpwwer]

How are other people dealing with viewing/analyzing this much data?

I wrote a simple program to save to an Access DB and choose the following compromise in amount of data vs DB size...
There are 4 tables in the DB, seconds, minutes, hours and days
seconds stores every second but only keeps the last 600 (10 minutes)
minutes stores the total for each minute but only keeps the last 720 (12 hours)
hours stores the total for each hour but only keeps the last 720 (30 days)
days stores the total for each day but only keeps the last 365 (1 year)

This way there is only one file to access and it will never get too large.

if could be argued that there should be enough seconds to fill an hour (3600) and enough minutes to fill a day (1440) but the record size becomes a little large at that point and I am not sure that graphing 3600 points is useful.

That's pretty much the same thing I do with the MainLobby plug-in. However, I'm keeping all minutes, all hours, all days, all months, and all years. I've been collecting since about Jan 25th and it's at 13Mb. I'm doing something similar for graphing but that only keeps 1 days worth of minute data and the graphing software can only handle about 7 hours worth. I don't have anything tracking by second anymore. I didn't find by second data too useful. The trends and usage are easily visible with 1 minute resolution. I think I posted this above, you can look at the by minute chart at http://www.bobshome.net/graph/energychart.php I'm planing to have the EMCServer software do something like this too.

 

[3tones]

I suppose it isn't very useful to keep per second data for very long. I think I might keep it for a week, though at least in my "discovery" phase. I've seen some weird power blips that consume twice as much as normal for a few seconds... not sure what that is yet.

My latest idea for logging is to decode the serial stream and output that via syslog to Splunk. The (no longer available on the website) Kiwi Logger provided a clean syslog API for windows... luckily I still have the install. Now I just need to reverse engineer the serial protocol. If anyone wants to donate some code, I'm listening.

 

[moridin1138]

If anyone wants to donate some code, I'm listening.

Can I ask why you would want it in a flat file format like syslog and not in a db? The advantage to a db is that you can do on-the-fly analysis. I am curious what youre thinking is.
Cacti or Munin may serve you better (or some sort of RRD or SQL based system).

Either way, here is what I have so far in Python. It's rough and basic, but it works for voltage and CH1 so far. It just outputs the most recent stat on the command line like this:

Volts: 118.7v
Ch1 Watts: 920w

Ill post updates here as well: http://fivejacksons.com/brian/?p=240

It's also untested in Windows, and since pySerial is a requirement I believe there is a library necessary for Python to access the serial port (in addition to, and specific for Windows). More info here: http://pyserial.wiki.sourceforge.net/pySerial

[codebox]#!/usr/bin/env python
#
# ecmread.py
# Many thanks to:
# Amit Snyderman <[email protected]>
# bpwwer & tenholde from the cocoontech.com forums
#

import serial, time, datetime, sys, array

SERIALPORT = "/dev/cu.serial1" # the com/serial port the ecm is connected to (COM4, /dev/ttyS01, etc)
BAUDRATE = 19200 # the baud rate we talk to the ecm

START_HEADER1 = '0xfe'
START_HEADER2 = '0xff'
START_HEADER3 = '0x03'

ser = serial.Serial(SERIALPORT, BAUDRATE)
ser.open()

pseconds = 0

#we only want one readout, and need two packets first
for i in range(0, 2):
#look for 0xfe
if hex(ord(ser.read())) == START_HEADER1:

#if 0xfe is found, read the packet
packet = ser.read(64)

#convert the packet to decimal from hex
packet = [ord© for c in packet]

#channel 1 total watt-seconds
ch1ws = packet[8] * 256 * 256 * 256 * 256 + packet[7] * 256 * 256 * 256 + packet[6] * 256 * 256 + packet[5] * 256 + packet[4]
#total seconds
seconds = packet[38] * 256 * 256 + packet[37] * 256 + packet[36]

#if we have a previous reading, display the totals
if pseconds != 0:

#total voltage
volts1 = (packet[2] * 256 + packet[3]) * 0.1
print "Volts: %sv" % (volts1)

#watts
ch1w = (ch1ws - pch1ws) / (seconds - pseconds)
print "Ch1 Watts: %sw" % (ch1w)

#make the current reading, the previous reading (in case we don't yet have a reading)
pch1ws = ch1ws
pseconds = seconds
ser.close()

[/codebox]

 

[Btech]

A little O.T., but what shipment method is Brueltech using, and can we get a tracking #?

Mine was sent via the mail from Canada, this is pretty slow, it took 6 business days to get here.

Getting product out has been a nightmare lately. My apologies. We have been using ExpressPost because it avoids customs brokerage charges that couriers would charge when delivering and this method also makes a tracking number available.

 

[tenholde]

A little O.T., but what shipment method is Brueltech using, and can we get a tracking #?

Mine was sent via the mail from Canada, this is pretty slow, it took 6 business days to get here.

Getting product out has been a nightmare lately. My apologies. We have been using ExpressPost because it avoids customs brokerage charges that couriers would charge when delivering and this method also makes a tracking number available.

I received a tracking number that tracked it to the US, but no further. Arrived in about 6 days. Great device.

tenholde

 

[3tones]

Can I ask why you would want it in a flat file format like syslog and not in a db? The advantage to a db is that you can do on-the-fly analysis. I am curious what youre thinking is.

Valid question. Sometimes I wonder myself.. Syslog is just a means to get it into splunk. Splunk does a good job at indexing and searching data and I'm looking for the ability to zoom around the data to look at it. The RRD based solutions I've seen are a little more static than what I'm looking for. My whole thought is a little different then others. At this point I am just looking to find my power hungry loads and just learn about my power consumption. Other solutions seem focused on long term data logging, which I agree - is suited better towards a database.

 

[3tones]

It's also untested in Windows, and since pySerial is a requirement I believe there is a library necessary for Python to access the serial port (in addition to, and specific for Windows).

The download link doesn't work on your website. However, I copied the code from your post and I was able to get it working in windows with minimal fuss. I installed python, pyserial, and the win32 extensions. The only change I made is:

SERIALPORT = "COM1"

 

[3tones]

Last post for the night

Is it acceptable to Brultech and the forum to discuss specifics of the ECM-1240 protocol, perhaps in another thread? I've got most of it figured out, but I'm stumped on a few bytes.

 

[moridin1138]

It's also untested in Windows, and since pySerial is a requirement I believe there is a library necessary for Python to access the serial port (in addition to, and specific for Windows).

The download link doesn't work on your website. However, I copied the code from your post and I was able to get it working in windows with minimal fuss. I installed python, pyserial, and the win32 extensions. The only change I made is:

SERIALPORT = "COM1"

Thanks for the heads up. Fixed the link. Also, great news that it works in Windows! I hope to continue updating it soon. I believe creating a separate thread is in order.

Is it acceptable to Brultech and the forum to discuss specifics of the ECM-1240 protocol, perhaps in another thread? I've got most of it figured out, but I'm stumped on a few bytes.

Great questions. In fact, in response to 3tones comment above I think this thread is getting a little long. The scope is pretty broad at the moment. ^_^

 

[tenholde]

If you want to avoid developing and maintaining the code to interface with the ECM-1240 and decode the datastream, I am developing a free interface program that broadcasts the datastream in ascii characters.

http://www.tenholder.net/tenWare2/tenHsEcm1240

The code appears to be very reliable now.

tenholde