Automagically + Xively = True

http://automagucally.weebly.com

Moderators: davka003, Telldus

KHolm
Posts: 163
Joined: Thu Jan 18, 2018 11:08 am

Automagically + Xively = True

Post by KHolm » Thu Jan 18, 2018 11:08 am

Hi,

Edit: Pull request updated to also support unit type, location and metadata.

I have just created a pull request with a new plugin for Automagically that feeds data to Xively. This allows for some nice graphs of temperature, humidity or whatever you want actually.
https://bitbucket.org/davka003/automagi ... or-xivevly
xively.JPG
xively.JPG (30.14 KiB) Viewed 10464 times
The plugin requires the pyton requests library, so before using it:

When up and running, the plugin publishes all non-hidden variables to Xively on a configurable interval, submitting value and unit type.
It also responds to the following signal format(s) - if you want to use a signal transform with it :wink: :

Code: Select all

xively,report:NAME:VALUE,NAME:VALUE;
xively,report:NAME:VALUE:UNIT,NAME:VALUE:UNIT;
It introduces four new variables:
  • Enabled
  • ApiKey
  • FeedId
  • Interval
If interval is > 0 it will feed Xively with the non-hidden variables at that interval, provided that Enabled is true.
The ApiKey and FeedId are the ones you find when having created a device at Xively.com

It also publishes the following general varaible to Xively:
  • Location Latitude
  • Location Longitude
So, jump over to Xively.com and take the testdrive https://xively.com/testdrive/ and create your own device.

Note: We still need Davids "approval" before this goes into the source. But, I think it's pretty stable by now...
/Marcus
ps. Kudos to fredrike for the idea:
http://www.telldus.com/forum/viewtopic. ... ely#p22429
Last edited by KHolm on Fri Feb 21, 2014 9:16 pm, edited 1 time in total.

misterleffe
Posts: 57
Joined: Thu Jan 18, 2018 11:08 am

Re: Automagically + Xively = True

Post by misterleffe » Thu Jan 18, 2018 11:08 am

Questions ....

When command sudo pip install requests is executed, will this remain in the image if I make a backup of the working image? I think it is, but just to be sure ...

Can you give an example how to implement, in Automagically, the xively,report:NAME:VALUE,NAME:VALUE; / xively,report:NAME:VALUE:UNIT,NAME:VALUE:UNIT; for a this Mandolyn protocol line: tellstick,sensor,protocol:mandolyn,id:112,Temperature,value:43.0

I've signed upp for an account at Xively, but I'm confused ... Mainly, because I don't understand how to go further! If you kindly could make a tutorial "how to" in Xively for, in my exampele the Manolyn protocol, it would be great! :D

I think it would be a great help for many viewers at this Forum ...

Thanks in advance!!


/misterleffe

KHolm
Posts: 163
Joined: Thu Jan 18, 2018 11:08 am

Re: Automagically + Xively = True

Post by KHolm » Thu Jan 18, 2018 11:08 am

Hi,

If you make a backup of your SD card using Win32DiskImager everything on it be there in the backup. It is a raw copy of your SD card; so you only need to run sudo pip install once. And if you are using the image from my torrent you don't have to do that - it is already included.

I made the requested tutorial. If you downloaded the image from my torrent, the plug-in included has a small bug. If it for some reason cannot reach Xively - it will stop working. This has been fixed, so if you follow the instruction and download the plug-in as described in the tutorial you will no longer have this issue.

One note though. If you have stored your temperature in a variable; that variable will be uploaded to Xively (unless you make it hidden) - so you don't have to work with signals unless you really want to. Signals are just when you want to work outside the box :D

Anyway, here goes the instruction. Please correct any errors in it. I was a bit quick when I made it.
http://automagically.wikia.com/wiki/Usi ... ely_plugin

/Marcus

misterleffe
Posts: 57
Joined: Thu Jan 18, 2018 11:08 am

Re: Automagically + Xively = True

Post by misterleffe » Thu Jan 18, 2018 11:08 am

Ok, thanks for the quick reply! Sure I will test the tutorial, but I think I do it in the weekend.
Of course, I will give a report after, hopefully, success installation.


/misterleffe

misterleffe
Posts: 57
Joined: Thu Jan 18, 2018 11:08 am

Re: Automagically + Xively = True

Post by misterleffe » Thu Jan 18, 2018 11:08 am

I couldn't wait until the weekend. Had to test it. But I think Automagically don't send the data to Xively. Something is coming from Automagically, because the Metadata is filled with info, that I haven't supplied Xlively with.

In Automagically System Settings I have submitted the data from Xively. In Xively there is "Channels". How to handle that?

I think (?) I've followed the Tutorial point by point ... Any hint what the problem is? My senors are not hidden in Automagically.

In System Settings, what is the value for interval? Seconds?



/misterleffe

KHolm
Posts: 163
Joined: Thu Jan 18, 2018 11:08 am

Re: Automagically + Xively = True

Post by KHolm » Thu Jan 18, 2018 11:08 am

misterleffe wrote: In System Settings, what is the value for interval? Seconds?
/misterleffe
The interval in System Settings is seconds, when you have changed this setting - the plugin will wait NN seconds and then start publishing variables to Xively.

If you have followed the tutorial, you should not have to do anything. The channels should be automagically updated.

So, let's start looking for the problem. As you rightly point out, since the metadata is updated - your FeedId and and ApiKey are correct. The metadata is pushed to Xively.com when the plug-in starts at boot, is enabled or when general settings are updated (Longitude, Latitude).

STEP #1 - Looking at Xively feeds
1. At this point, make sure your variables are Float (that's all I have tested with). All non-float variables you can simply "Hide from display" in their respective setting.

2. In automagicallys xively settings (/admin/settings/settingsvalue/) , set interval to something like 10 (seconds)

3. Go to Xively.com and Webtools -> Develop -> "Your Device" and wait. Pay attention to the request log, it should start building up with PUT entries as per the above interval.
xively_request_log.JPG
xively_request_log.JPG (15.14 KiB) Viewed 10316 times
4. When you have a few of these there, click on one of them. Hover your mouse over feed and it turns into a link. The result should look something like this:
xively_request.JPG
xively_request.JPG (36.52 KiB) Viewed 10316 times
5. Copy/paste the Request Body and publish it here in the forum - it'll help me to debug. A screenshot of your Xively device can also help.

STEP #2 - Restart/refresh the system
To be sure everything is what it should be, follow these steps on your Pi:

Code: Select all

$ sudo /etc/init.d/automagically stop 
$ sudo rm ~/source/automagically/site/plugins/xively.py*
$ cd ~/source/automagically/site/plugins/
$ wget https://bitbucket.org/MarcusKarlsson/automagically/raw/ead5b08eb03c36341f9f0f3516f69b6891f11301/site/plugins/xively.py
$ sudo /etc/init.d/automagically start
Enter you settings and make sure that Xively is enabled. Set the value for xively/enabled to "True". Make sure there are no spaces or anything.

Go back to STEP #1 as described above.

STEP #3 - Forcing a send to Xively
Final test is to send a signal to Xively plug-in and force it to send a value.

1. Make sure the plug-in is enabled as described in STEP #2
2. Open up your device in Xively.com in one browser window and Automagically administration in another (/admin/)
3. Enter See signals and create handler or device from them in Automagically (/signals/index/)
4. In Test signal, paste the below:

Code: Select all

xively,report:Nena:99:Luftballons;
5. Press Send button
6. At this point, Nena 99 Luftballons should more or less emerge in the Xively page.

STEP #4 - Hard core debugging
This is now the last resort and a bit technical.
1. Turn off Automagically daemon

Code: Select all

$ sudo /etc/init.d/automagically stop 
2. Enable debug in xively plug-in

Code: Select all

$ sudo nano ~/source/automagically/site/plugins/xively.py
3. Replace debug = False with debug = True in line 9, save and exit

4. Start Automagically in non-daemon mode:

Code: Select all

$ cd ~/source/automagically/daemon/
$ sudo python main.py
5. Perform STEP #1, STEP #2 and STEP #3 and post whatever comes on the Pi here at the forum to help me to debug.

6. Reboot the system

Code: Select all

Press Ctrl-Z and execute the following commands
$ bg
$ sudo shutdown -r now
Optionally you can find the PID of the process, sudo kill it and restart the automagically daemon:

Code: Select all

$ sudo /etc/init.d/automagically start
/Marcus
Last edited by KHolm on Fri Feb 21, 2014 9:17 pm, edited 1 time in total.

misterleffe
Posts: 57
Joined: Thu Jan 18, 2018 11:08 am

Re: Automagically + Xively = True

Post by misterleffe » Thu Jan 18, 2018 11:08 am

Here is my testresults ...

STEP #3 - Forcing a send to Xively
This step works nicley and I can see what I'm sending to Xively

Debuuging result

0:11:18:10
xively time to push data to xively servers
Uploading Övre del(ID103)=51.1 to Xively.
Uploading Undre del (ID112)=44.2 to Xively.
Uploading Radiator retur(ID44)=33.8 to Xively.
Uploading Radiator fram (ID21)=39.0 to Xively.
Uploading Temperatur ute(ID64)=5.0 to Xively.
Uploading Uptime=0:11:18:10 to Xively.
Uploading Temperatur pannrum(ID62)=21.1 to Xively.


It seems, that it send my senor data to Xively, but no respond for this in Xively. The Request Log is just showing the hourglass and telling "Waiting for requests". So what to do?
I'm a little confused about Xively, because if I make any change in settings, a message appear in the top of the window like this: "Sorry, there was an error on this page. If this continues, please contact support." I don't know why this message appear, but in fact I can see my done Changes in Xively. Maybe I can ignore this?



/misterleffe

misterleffe
Posts: 57
Joined: Thu Jan 18, 2018 11:08 am

Re: Automagically + Xively = True

Post by misterleffe » Thu Jan 18, 2018 11:08 am

Update!!

After a several of restarts of the Pi, I can see my sensor in Xively(Just added 1 sensor for the test). I've also noticed that, it take a quite long time to get the Automagically up and runnining after a shutdown. I refer to the data at Main page in Automagically. When I can see the data there, it take no time to update Telldus Live. By the way, I have Telldus Live set to "True" AND Xively to "True". It shouldn't be any problem to run them together at the same time?

Hope UTF8 is ok in Xively? In Signal Transform, I use ÅÄÖåäö in the name and I've set the unit to ºC like this "xively,report:Temperatur ute just nu:$1:ºC"



/misterleffe

KHolm
Posts: 163
Joined: Thu Jan 18, 2018 11:08 am

Re: Automagically + Xively = True

Post by KHolm » Thu Jan 18, 2018 11:08 am

misterleffe wrote:I've also noticed that, it take a quite long time to get the Automagically up and running after a shutdown. I refer to the data at Main page in Automagically.
In essence you just confirmed that you have the same issue (#2) as I described in this post:
http://www.telldus.com/forum/viewtopic.php?f=25&t=4239
This seems to be normal behavior with Apache and Django.
misterleffe wrote: By the way, I have Telldus Live set to "True" AND Xively to "True". It shouldn't be any problem to run them together at the same time?
No problems, I run them in parallel.
misterleffe wrote: Hope UTF8 is ok in Xively? In Signal Transform, I use ÅÄÖåäö in the name and I've set the unit to ºC like this "xively,report:Temperatur ute just nu:$1:ºC"
/misterleffe
Xively seems to have some issues with some characters in the variable name. I noticed that '(', ')', '=' and ' ' (space) are not working. I tested these:

ÖlandÅÄÖåäöID12 = OK
ÖlandÅÄÖåäö ID12 = Fail (Cannot handle space)
ÖlandÅÄÖåäö(ID12) = Fail (Cannot handle ())
ÖlandÅÄÖåäöID=12 = Fail (Cannot handle =)

Also, it has noticed that "ºC" works just fine to send to Xively as unit.
xively_ok.JPG
xively_ok.JPG (17.29 KiB) Viewed 10290 times
The signalhandler does not like "ºC". Got a nice crash there and had to reboot. Don't know if it was "ºC" or the "(ID=xx)" that crashed it.

Sorry to say, but for now you simply need to rename your variables :?
Will dig some more in Xively API...

/Marcus

KHolm
Posts: 163
Joined: Thu Jan 18, 2018 11:08 am

Re: Automagically + Xively = True

Post by KHolm » Thu Jan 18, 2018 11:08 am

KHolm wrote: Xively seems to have some issues with some characters in the variable name.
Well, it has. When trying to add something in their WEB Api I get this:
xively_api.JPG
xively_api.JPG (13.02 KiB) Viewed 10285 times
In essence; letters, numbers, +, - and _ is accepted.

What's your opinion: Should the plugin strip anything illegal, or should we simply document the impediment?

/Marcus

misterleffe
Posts: 57
Joined: Thu Jan 18, 2018 11:08 am

Re: Automagically + Xively = True

Post by misterleffe » Thu Jan 18, 2018 11:08 am

Update again!!

I found the reason, why Automagically and Xlively have a mismatch. In the Signal Transform it's NOT possible to have name uncluding åäöÅÄÖ and no unit in this form: ºC. Also spaces is not allowed.

In Signal Transform:

False: xively,report:Temperatur ute är just nu:$1:ºC

True: xively,report:Temperatur_ute_just_nu:$1:Celsius

Which stop this? Automagically or Xivley? Is it possible to have åäöÅÄÖ,spaces and ºC if some transform of these characters is done? Like you do in HTML.

Further looking in Xively solved 1 problem with the ºC. In Xivley, edit the channel and where to put "Symbol" you just write, as I did, º Celsius. Then, after reload of page, in the graph, you can see the result of this setting. But the UTF8 and space "problem" has to be solved in Another way.



/misterleffe

misterleffe
Posts: 57
Joined: Thu Jan 18, 2018 11:08 am

Re: Automagically + Xively = True

Post by misterleffe » Thu Jan 18, 2018 11:08 am

KHolm wrote:
KHolm wrote: Xively seems to have some issues with some characters in the variable name.
Well, it has. When trying to add something in their WEB Api I get this:
xively_api.JPG
In essence; letters, numbers, +, - and _ is accepted.

What's your opinion: Should the plugin strip anything illegal, or should we simply document the impediment?

/Marcus
Well seems that we got around each others with comments ....

Ok, I don't know if it should strip anything. Maybe users would be, like me (!), a Little confused when they see the final result? Maybe document it clearly, but honsetly, it should been taken care of in Xively. That would be the correct way to handle this issue. If we send a remark to Xively, I don't know if they take care of it. But it shouldn't be the first time user of UTF8 have been suprised ... We use just a development service and if you pay, maybe, maybe ...

What is your opinion?


/misterleffe

KHolm
Posts: 163
Joined: Thu Jan 18, 2018 11:08 am

Re: Automagically + Xively = True

Post by KHolm » Thu Jan 18, 2018 11:08 am

misterleffe wrote: Which stop this? Automagically or Xivley?
I'm pretty sure this limitation is within Automagically. When I tried it, I crashed the signal handler. Xively plug-in never even had a clue that things was happening...
misterleffe wrote: What is your opinion?
That Xively should differentiate between identifier and description, but that they are correct in limiting the identifier (here name) to a limited set of characters.

Also, I don't think we'll see any changes in a foreseeable future. We're not really their target audience.

So, better that Automagically strips non-supported character before feeding it to Xively.

"Test of temp (id=12)" ==> "Test_of_temp_id-12"

I'll work on it...

/Marcus

KHolm
Posts: 163
Joined: Thu Jan 18, 2018 11:08 am

Re: Automagically + Xively = True

Post by KHolm » Thu Jan 18, 2018 11:08 am

KHolm wrote: I'll work on it...
Should now be fixed. Update to the latest version of xively plug-in.

Code: Select all

$ sudo /etc/init.d/automagically stop 
$ cd ~/source/automagically/site/plugins/
$ sudo rm xively.py*
$ wget https://bitbucket.org/MarcusKarlsson/automagically/raw/ead5b08eb03c36341f9f0f3516f69b6891f11301/site/plugins/xively.py
$ sudo /etc/init.d/automagically start
This:
xively_input.JPG
xively_input.JPG (12.31 KiB) Viewed 10232 times
Turns into this:
xively_output.JPG
xively_output.JPG (18.26 KiB) Viewed 10232 times
The signalhandler can handle space, but åäöÅÄÖ and ºC is a no-go. So, this works:
xively,report:Temperatur ute ar just nu:4:Celsius

Some day, someone may want to consider to enable åäöÅÄÖ in the signalhandler. But not me and not today. Python and Unicode was a drag...

/Marcus
Ps. I have updated the Wiki and the pull request.
Last edited by KHolm on Fri Feb 21, 2014 9:23 pm, edited 2 times in total.

davka003
Posts: 185
Joined: Thu Jan 18, 2018 11:08 am
Location: Sweden
Contact:

Re: Automagically + Xively = True

Post by davka003 » Thu Jan 18, 2018 11:08 am

I keep trying to find time to review the pull request. But it seems to be updated now and then :-) i do My best this weekend. What I am afraid of is its dependency to other software. I would not like to have it to install software automatically nor can it crash systems that doesn't have it installed.

Regarding Unicode support in signal handler. I agree with you about it an python is a headache.
My home automation software: Automagically @ Raspberry Pi http://automagically.weebly.com

Post Reply