dataMine Overview dataMine Overview dataMine is dataMine is a data capture and visualisation plugin for Vera. Vera. It includes a Vera Vera plugin to record and process the data, and a web GUI for data visualisation and configuration. The web GUI is shown in Figure 1. 1. The plugin that runs in Lua records all changes for variables that are being logged. The raw data is then written into log files on Vera (preferabl on a U!" stic#$, and the GUI then re%uests this data fro& the plugin. "oth sides are therefore re%uired to operate ' the Vera, Vera, providing the data capture cap ture and server, and the GUI, providing the data &anipulation and visualisation. This version of the guide was updated on ) *a +1.
dataMine wor#s dataMine wor#s b &onitoring all changes in device variables in Vera. Vera. -henever a variable changes, dataMine gets dataMine gets called, and it logs this data into a file for later use. The GUI then interacts with this data to provide the user a visualisation interface.
Installation dataMine is dataMine is installed through the UI &ar#etplace. This installs instal ls all re%uired files onto our Vera, Vera, however so&e configuration is re%uired before dataMine will dataMine will run. / U!" stic# is highl reco&&ended to store dataMine data. dataMine data. 0epending on the nu&ber of variables ou are logging, dataMine can dataMine can generate a reasonable a&ount of data, additionall, as this data is written regularl, this &a cause the internal in ternal Vera Vera &e&or proble&s if it is used for an etended duration. 2age 1
The U!" stic# can be for&atted on -indows and plugged into a free U!" slot on the Vera. Vera. 3ou 3ou should avoid for&atting the drive with the F/T14 F/T14 file sste& as this has li&itations. 5n an &odern version of -indows, it should co&pile as F/T+ F/T+ be default so this shouldn6t be a proble&. Linu et+7et file sste&s are fine. /fter for&atting in -indows, ou should set the dis# label in the dis# properties ' this allows ou to identif it once it is plugged into Vera. Vera. 3ou should then run the GUI and ou will be presented pres ented with the dialogue shown in Figure 1. 1. This dialogue as#s ou to select the UUI0 (the Universall Uni%ue Identifier$ of the U!" to be used for dataMine. dataMine. If ou have &ore than 1 U!" stic# installed in our Vera, Vera, and ou6re unsure which one is the one for dataMine, dataMine, then it is suggested to add a label to the drive ou wish to use for dataMine, dataMine, and refresh the GUI page so that onl one UUI0 is shown. 5nce ou6ve selected the UUI0, press !ave, and dataMine should dataMine should be configured. 8estart Luup (press 8eload in UI$, and dataMine should dataMine should be up and running.
Figure 2: dataMine dataMine initialisation dialogue. The configuration dialogue will set the variables shown in Table 1. 1. It is also possible to set these values &anuall if ou please, however the dialogue will be shown anti&e that the U!" stic# fails to &ount correctl c orrectl.. Configuration Variable
Suggested Value
!et0ata0irector
7data*ine
!et*ount2oint
"lan#
!et*anual*ount
!et*ountUUI0
U!" UUI0
Table 1: Suggested configuration for use with Vera USB logging.
2age +
-hen dataMine starts, it will create two further variables 9 :&ountLocation: and :&ountTpe:. It should be noted that these ;ust report the current &ount status, and are not &eant to be used as data entr points b the user. If ou have &anuall set a &ount point with the
Figure 3: dataMine configuration variables.
Using dataMine with the Vera USB logging For use with the Vera U!" logging, additional configuration &a be re%uired. This is because Vera will &ount the U!" stic#, and if dataMine is also allowed to &ount the sa&e stic# to a different &ount point, corruption can occur. 3ou &ust either use a separate U!" stic# for dataMine and use the configuration described above, or point the dataMine director to a location below the log director. >ote that the preferred option is to use a separate stic# if possible to avoid conflicts. !uggested configuration for use with U!" logging is shown in Table +. Configuration Variable
Suggested Value
!et0ata0irector
7t&p7log7c&h7data*ine
!et*ount2oint
>ot re%uired to be set
!et*anual*ount
1
!et*ountUUI0
>/
Table 2: Suggested configuration for use with Vera USB logging.
2age
Use Plugin Properties The data*ine plugin presents a si&ple device in the UI dashboard ' as shown in Figure ?. This ;ust shows how &an channels are being &onitored, and how &an channels have been configured in all.
Figure ": #evice overview. Figure shows the &ain properties tab. /gain, this present the sa&e infor&ation as the device overview, however there is also a lin# to the GUI webpage which is outside of the UI interface.
Figure : dataMine !ro!erties !age.
2age ?
Graphical User Interface The GUI is a web interface that allows ou to visualise the data fro& Vera. It uses a tabbed interface to select different data views or tpes as shown in Figure 4. /t the right side of the tab bar is an <5nline !tatus= indicator ' this is green if the web interface is co&&unicating with Vera, and gre if Vera is not co&&unicating with the browser. It displas red if co&&unication is o#, however there are errors within dataMine (eg. Initialisation error, or error writing data to the U!"$.
Figure $: dataMine User %nterface.
Graph Tab The graph tab provides an interface to graph the data being recorded b the dataMine plugin.
2age
Figure &: dataMine gra!h window. The graph displa is split into two &ain areas ' the configuration area to the left, and the graph itself. In between the two is a slider bar that allows ad;ust&ent of the position between the two. >ote that there6s also a button in the &iddle of this slider with an arrow pointing to the left. @lic#ing on this will hide the configuration area and ou6ll have a screen that loo#s li#e Figure ). @lic#ing on the bar on the left will te&poraril displa the configuration area, while clic#ing on the arrows at the top of the bar will restore the configuration area. This allows the full window area to be &ade available for graphs. If running data*ine on a &obile browser, the bar will default to the left to provide the &ost space possible for s&all screens.
2age 4
Figure ): 'ra!h tab with configuration area hidden.
The configuration area includes areas for selecting the channels to graph, and the graph options. In the Graph @hannels area, clic#ing on a variable na&e, the variable will select or deselect. -hen selected, the variables can be graphed b clic#ing on the U!date 'ra!h button (the graph will use the sa&e ti&e period as is on the current graph$. @lic#ing on the (eset 'ra!h button will clear the current selection and reset the ti&e period bac# to default ' it will not affect the current graph, however. @lic#ing on the Save 'ra!h button will allow ou to save the graph for later %uic# recall. !aved graphs will be listed in the Saved 'ra!hs list.
2age A
Figure ,: 'ra!h channel selection
Bach channel has an icon on the left side 9 this indicates the tpe of data. The *ast U!dated colu&n also shows if data has been received recentl. / tic# is shown if the variable data has changed recentl, a ellow ecla&ation is shown if the data is %uite old, and a red ecla&ation is shown if the data is ver old. The definition of these flags is configurable per channel since different channel tpes epect different change rates. @lic#ing on the 'ra!h +!tions navigation bar brings up a nu&ber of para&eters that can be changed in the graph. !ee Figure 1 and Table for details of the options.
Figure 1-: 'ra!h o!tion window.
2age )
Option
Description
0ispla data &ar#ers
0isplas a point on the graph where each
0ispla shadows
0isplas a border and shadow around the chart,
0ispla legend
0isplas the graph legend showing what data is displaed with each colour.
0ispla crosshairs
Bnables crosshairs.
0ispla tooltip
!hows the tooltips where the cursor or crosshair is near a data point. The tooltip shows the actual data displaed on the graph.
0ispla local ti&es
!witches between local ti&e and Universal ti&e.
Line width
@hanges the line width of graphs.
/ni&ation
Bnable or disable the chart ani&ation when data is drawn. Table 3: 'ra!h o!tions.
The toolbar across the top of the graph screen provides a nu&ber of buttons for interacting with the graph. Bach of the buttons has a tooltip, so if ou put the &ouse over the button, ou will get a hint as to what the button does.
2age C
Icon
Description
Doo& in Doo& out 0ispla last das data 0ispla last wee#s data 0ispla last &onths data 0ispla last ears data !croll left !croll right 0ispla real9ti&e graph Turn off displaing Vera >otifications 0ispla notifications relevant to the channels being graphed (fro& the sa&e device, or its parent device$ 0ispla all notifications ' note that this can be overwhel&ing when long ti&e9scales are displaed. 0isplas gre bands on the chart during night9ti&e. 0ispla graph infor&ation Table ": 'ra!h toolbar buttons. It is possible to Eoo& using the &ouse as shown in Figure 11. !i&pl hold the &ouse button down at the start of the Eoo& period, &ove to the end, and release the &ouse. >ew graph data will be loaded.
2age 1
Figure 11: 'ra!h window oo/
The real9ti&e graph displas data for the past 1 &inutes, and updates ever second. 0ue to the nature of the displa, it displas data in
ote 8eal9ti&e graphing is not i&ple&ented in the current version ' it will return soon$.
Notifications dataMine can displa Vera notifications on the graph as shown in Figure 1+. The notifications are updated ever &inute, so ver new notifications &a not be displaed. If there are &ore than notifications in the graph, dataMine will onl show a selection of b tri&&ing out ever nth notification. In realit, this is far too &an to be able to view anwa 2utting our &ouse over the notification will displa &ore infor&ation on the notification.
2age 11
Figure 12: 0otification dis!la
Day-Night banding dataMine records the sunrise7sunset ti&es calculated b Vera, and then allows the nights to be displaed as gre bands in the graphs. This is shown in Figure 1. To activate7deactivate this option, clic# on the &oon icon in the toolbar. >ote that as dataMine does not calculate the sunrise7sunset ti&es, this feature will onl wor# after the date that this was added to dataMine (/pril +1$. /lso, if our Vera is turned off for a period, this will also stop the ti&es being logged.
2age 1+
Figure 13: #a0ight banding.
Saing Graphs Fro& the graph selection window, ou can clic# the !ave button to save a graph configuration for later recall. Figure 1? shows the window that is then displaed. This allows selection of which ais each channel will be displaed on, as well as an icon that will be displaed alongside the graph in the selection list. The ais selection &ust be a nu&ber between 1 and ?. The Huic#View reference is a reference for displaing the graph in an e&bedded webpage. For &ore infor&ation, see the section on e&bedding below. >ote that this should >5T contain spaces .
2age 1
Figure 1": 'ra!h Save 4dit window.
Figure 1: Saved gra!h selection list.
Figure 1 !hows the saved graph selection list. The and the icon allows ou to edit the graph data.
icon allows ou to delete the saved graph,
>ote that when editing a graph, if ou change the na&e, dataMine will save a new cop of the graph and ou will need to delete the one ou don6t want.
Dashboard Tab Figure 14 shows the dashboard. @urrentl this is (ver &uch$ wor# in progress and it shows the energ currentl being used. There is a gauge showing the current whole house energ, and a donut chart showing the split b device and categor. / trend chart shows the whole9house use over the past +? hours. 2age 1?
(This will change$.
Figure 1$: #ashboard tab
The whole9house gauge provides the current whole house energ use. It is a su& of all the whole house energ categories, less the energ generator categories.
The energ consu&ption donut shows energ consu&ers b categor and device. The Unaccounted categor is the difference between the total of all consu&ers, and the current whole9house consu&ers (note that this doesn6t include the energ generators$.
2age 1
Configuration Tab The configuration tab shown in Figure 1A allows configuration of dataMine. @urrentl, this pri&aril includes the configuration of variable logging, but further configuration will be added in future.
Figure 1&: 5onfiguration Tab
Variable !onfiguration Two lists are presented for variable configuration. The active devices are listed under #evice Variables, and inactive devices are listed under 'host Variables. Ghost variables are variables that still eist in the dataMine configuration, but the devices have been deleted fro& Vera. 3ou can delete the ghost variable, or #eep it for reference. 0eleting the ghost variable will not delete the historical data fro& the archive. 5nce configuration changes have been &ade, ou &ust press the effect.
Save button for the& to ta#e
Option
Description
0evice
The Vera device na&e and nu&ber (can not be changed$.
!ervice
The Vera !ervice na&e (can not be changed$.
Variable
The Vera Variable na&e (can not be changed$.
Vera @ategor
The Vera 0evice @ategor (can not be changed$.
2age 14
Option
Description
0ispla >a&e
The displa na&e that will be used in dataMine graphs.
Units
/ tet string to set the units.
Logging Bnabled
Indicates if dataMine is logging data for this variable. !et to true to enable logging.
Bnerg @ategor
The dataMine energ categor. This is #ept separate fro& the Vera device categor. The following categories can be used 9 1 -hole9house + Lighting eating ? Bntertain&ent /ppliances 4 @o&puting A Utilities +1 !olar 2V generation ++ -ind generation (>ote that this will be &ade &ore user friendl in future. It is currentl li&ited b the libraries being used$
Graph Tpe
!ets the graph tpe to be used with this channel 0efault (spline$ 1 Line + !catter /rea ? /rea7Line co&bined (for status graphs$ (>ote that this will be &ade &ore user friendl in future. It is currentl li&ited b the libraries being used$
0ata 5ffset
/dds a data offset when graphing out the channel. The value will be added to each value to allow calibration of a channel. This does not affect the logged data which will alwas be the value.
0rows -arning
/llows &odification of the drows warning ti&e. This is set in &inutes. The drows warning is the ti&e after which the icon is shown. !etting this to will use the sste& default.
0rows Brror
/llows &odification of the drows error ti&e. This is set in &inutes. The drows error is the ti&e after which the icon is shown. !etting this to will use the sste& default.
Bnable out9of9li&its filter
Bnables or disables filtering of values. Filtered data outside of the specified li&its will not be displaed in graphs. This can be used to filter out large spi#es.
Filter &ai&u&
*ai&u& filter value. 0ata above this nu&ber will not be displaed.
Filter &ini&u&
*ini&u& filter value. 0ata below this nu&ber will not be displaed. Table : Variable configuration o!tions.
2age 1A
Figure 1) shows the 9ais loo#up. This is pri&aril useful for alphanu&erical data (wind direction, weather condition etc$. dataMine actuall generates this loo#up auto&aticall for such data, however this configuration window allows ou to change the wa the data is graphed. " changing the nu&bers against the labels, ou will change the order the data is graphed in the vertical ais. It is also possible to use this configuration for nu&erical data that ou would prefer to be displaed using labels. >ote however that this &a produce strange results for data with continuall varing data, and is best used for variables with a li&ited nu&ber of possible points (as per the data &entioned above$.
Figure 1): 6a7is loo8u! configuration
Embedding graphs in a web page -hen saving a graph, there is the option to specif a
the full dataMine GUI. 0a7night boundaries can be displaed, but notifications are not supported.
Advanced Topics Under the hood dataMine wor#s b loo#ing for changes in device variables ' it is not periodicall logging data based on ti&e, it si&pl logs changes. This is i&portant to note when thin#ing about how a graph is rendered ' for status tpe data (eg. on7off$, the graph needs to account for this to avoid diagonal lines. dataMine records data in @!V files (@o&&a !eparated Value$. (@urrentl$ three different files ar e generated for each variable being recorded 9 a raw, hourl and dail file. Files are then split to avoid the& getting too large (which is a potential proble& for constantl changing sensor data such as te&perature$. Bach ti&e Vera observes a change in the data for a &onitored variable, dataMine will log the current value into a file. Bach da, the long ter& histor files are generated fro& the raw data. Bach line of data in the @!V logfiles includes two nu&bers 9 the ti&e of the data9point (as a uni ti&esta&p$, and the data value. For the raw file, the actual changes are logged. For the hourl and dail files, the &ini&u& and &ai&u& value and ti&e are logged for each ti&e period.
USB and "inu# Linu has a different philosoph to -indows when it co&es t o attaching U!" &e&or stic#s, or dis# drives. !uch devices are <&ounted= into a director in the root file9sste&, rather than being seen as a separate drive. / director is first created in the root file9sste&, and the U!" stic# is then &ounted into this director. This has the following i&plications 9 •
•
"efore the U!" stic# is &ounted, if ou were to loo# in the director, ou would see files that are resident in the &ain file9sste&. /fter the U!" stic# is &ounted, ou will see files that are in the root director of the U!" stic#. If there were an files in the &ount director on the &ain file9sste&, these will no longer be visible as this director is effectivel replaced b the U!" stic#.
Setting syste$ defaults / UI will be added to set the sste& defaults, but for now, this can be done in the following wa... Tpe the following U8L into our browser <>/port_3480/data_request?id=lr_dmCtrl&control=saveConi!&
then add the following para&eter to the end of the above line to set variables (note ' configuration na&es are case sensitive$.
2age 1C
energ-holeouse*inMnu&
!ets the &ini&u& value of the whole house gauge @an not be set above .
energ-holeouse*aMnu&
!ets the &ai&u& value of the whole house gauge. @an not be set less than .
general0rows-arningMnu&
!ets the drows warning sste& default. Values in &illiseconds ' defaults to 1 hour (4$
general0rowsBrrorMnu&
!ets the drows error sste& default. Values in &illiseconds ' defaults to 1 da ()4?$
graph0efault2eriodMnu&
!ets the default graph period. Values in das ' defaults to 1 da. *ust be between 1 to .
3ou should get the <5O= response if everthing went o#. >ote that no error chec#ing is perfor&ed when ou &a#e changes ' these settings are used in the GUI, and when the GUI reads the& bac# fro& Vera, it does perfor& a sanit chec# and &a reset to a default if the are dee&ed to be outside of nor&al bounds.
Bac%ing up dataMine data The following is a ðod fro& Pvespa&an on how to auto&aticall #eep a bac#up of our data*ine data director. This runs on linu. >ote that ou need to change the ip address and directories.
Fro& Pvespa&an original post Qanuar 4, +1 (http77foru&.&icasaverde.co&7inde.php7topic,1A).&sgCA+4.ht&lR&sgCA+4$
5O, so I have now whipped soðing up, which Ssea&sS to do a bout what I need for now. I thin#, that if the logs get ver big over ti&e, &abe this isn6t the correct ðod. "ut at least it can run at ti&es where house is at sleep. "asicall three steps 1. @reate public ssh #es for unattended login to our Vera fro& our linu server +. Using scp script, cop the whole data*ine log direcor to our linu server. . 2ut a line in ou crontab so it is done once ever night. It would have been nice to use rsnc instead of scp, but it sea&s that it is not included in the vera dist, so scp will have to do. >ot sure how &uch data we will see in the end, but ti&e will tell. !o in detail for anone that wants to use this tpe of bac#up 1. @reate our public #es according to e.g. this guide http77wi#i.openwrt.org7do#u.php idMoldwi#idropbearpublic#eauthenticationhowto
2age +
+. *odif the script for our path6s and vera ip (if ou don6t have 6vera6 in our dns7hosts file$. /long with our veras I2, ou will want to change both destdir and the source path on our vera. #!/bin/bash date="`date +%Y-%m-%d`" olddate="`date --date '2 months ago' +%Y-%m`" destdir=~/backu/datamine # get the iles era does not ha$e rsnc& so e use regular sc sc -(r root)$era*/mnt/dataminelogs ,destdir/ # create the backu tar ball tar -c ,destdir/datamine,datetarg ,destdir/dataminelogs # i. rights chmod 00 ,destdir/datamine,datetarg # remo$e temiles rm -r ,destdir/dataminelogs # 1emo$e all backus created 2 months ago rm - ,destdir/datamine,olddate-targ
@reate the destdir ( in this case 7bac#up7data&ine $ . 5nce ou have tested the above configuration, and ou are sure it wor#s ;ust add line to our own crontab (crontab 9e in & case$ and add a line li#e this (running o6cloc# in the &orning$. 0 3
~/bin/backu$eradatamine
If you hae a proble$ If ou have a proble&, I6& happ to help. owever, please send the Vera logfiles, provide infor&ation on what Vera &odel ou are using, if ou have other U!" stic#s fitted. There is now a %uic# wa to get so&e of this with the following U8L 44516 7899/ort:;<0/datare(uestid=lrdm>trl?control=debug
This will du&p our configuration, U!" &ount data, and an recent log entries. owever, this will onl wor# once the basic plugin is loaded and running. 2rior to that, please provide as &uch infor&ation as ou can.
U!" 8ead9onl file sste& @ogging error logging $ariable A**urn*un-org*ser$ice7d*BemeratureCensorD**>urrentBemerature Enable to oen ile or rite /dataFine/@ot Bemerature - >urrentBemerature G122;HIt.t 5rror* '/dataFine/@ot Bemerature - >urrentBemerature G122;HIt.t* 1ead-onl ile sstem'
@aused b errors in the file sste&. @op all files off and refor&at.
2age +1
