Developer discussion - Hanging Threads

http://automagucally.weebly.com

Moderators: davka003, Telldus

Post Reply
KHolm
Posts: 163
Joined: Thu Jan 18, 2018 11:08 am

Developer discussion - Hanging Threads

Post by KHolm » Thu Jan 18, 2018 11:08 am

Hi (developers),

[My instance of] Automagically seems to enter some hanging state every now and then. It seems to be rather random and the effect is a unresponsive system.
When this happens, nothing can be seen on the signal trace and the automation stops. Starting and stopping the service brings back everything to normal.

When looking at the plugin system there seems to be a built-in weakness that we should be able to overcome. The typical design pattern for plugins is to have a execution thread that is fed by the signalhandler. The way to feed the execution thread is to use a Queue.

Item #1 - Improvements to design pattern

If the execution thread for some reason bails out, eventually the Queue will be full and since all plugins uses a blocking Queue.put(), the signal handler will block. Typically when feeding things like sensors and such into the plugins.
This block goes all the way to the SignalProcessingThread() in signals\models.py, traversing through the processSignal() site\plugins\plugins.py.

So, now to the discussion. Would it make sense to use a non blocking Queue.put() in our plugins?
Example from plugins\tellstick.py:

Code: Select all

def signalHandler(signal):
    if signal.content == 'terminate':
        workQueue.put(None)        

    elif signal.content.startswith('tellstick,do:'):
        for i in signal.content[13:].split(';'):
            if i != '':
                workQueue.put(i)
Can we replace workQueue.put(i) with workQueue.put(i, False) as a design pattern? And capture that in the calling function.

Code: Select all

# Update the plugins signalhandler to raise exceptions
try:
   workQueue.put(i, False)
execept Queue.Full:
  raise PluginQueueFull("The plugins queue is full")


##
## Update signals\models.py to catch it.
##
try:
   plugins.processSignal(s)
except PluginQueueFull:
  # Kill the plugins thread and re-start it?
Item #2 - Misuse of design pattern?

I notice that some of the plugins, such as the tellduslive plugin, executes quite a lot of code in the signalHandler. I wonder if that is working well in this design pattern; e.g. when waiting on a socket, the signalling system is "hanging".
I think at least the tellduslive plugin needs some refactoring to reduce the risk of a hanging system.

I think the above mentioned items only becomes "critical" when Automagically starts interacting with the outside world.
A quick look tells me:
OK: 1-wire, email, httpreq, tellstick, datafetcher, datafetcher, timedeventes
Questionable: hdmi-cec, tellduslive

What are your thoughts on the above?

/Marcus

davka003
Posts: 185
Joined: Thu Jan 18, 2018 11:08 am
Location: Sweden
Contact:

Re: Developer discussion - Hanging Threads

Post by davka003 » Thu Jan 18, 2018 11:08 am

That seems like very good thinking. I am impressed.

Even thou I am the author of pretty much all of it I am for sure not an expert in several fields here. This whole project started as a learning experience for some technologies and areas that I don't use on my day job. So this feedback is for sure welcome.
My home automation software: Automagically @ Raspberry Pi http://automagically.weebly.com

KHolm
Posts: 163
Joined: Thu Jan 18, 2018 11:08 am

Re: Developer discussion - Hanging Threads

Post by KHolm » Thu Jan 18, 2018 11:08 am

Well then we're two. I've never actually touched python before I came across your program, but have been exposed to coding before. But that was a decade ago. I really like the modularity of your design and think the plugin system is a very nice piece of work/design pattern.

I am thinking that the mySql may be a bit heavy on the Pi, thinking that possibly a sqlite would be as good; possibly with a lighter httpd (such as lighthttpd). But I have no experience with Django so I don't even know it the above would work. And, likely it's too much work to replace...

Seems at least to be possible, http://joncraton.org/blog/39/lighttpd-django-sqlite/ on a quick google search

/Marcus
ps. How do you update django, looked like quite a work to update...

KHolm
Posts: 163
Joined: Thu Jan 18, 2018 11:08 am

Re: Developer discussion - Hanging Threads

Post by KHolm » Thu Jan 18, 2018 11:08 am

KHolm wrote: ps. How do you update django, looked like quite a work to update...
I actually meant to ask: How do you update jQuery, looked like quite a work to update every time jQuery is updated.
/Marcus

davka003
Posts: 185
Joined: Thu Jan 18, 2018 11:08 am
Location: Sweden
Contact:

Re: Developer discussion - Hanging Threads

Post by davka003 » Thu Jan 18, 2018 11:08 am

I have been thinking of switching to something less resource hungry than mysql and Apache. The later is probably easier to change as it would still make it possible for users to import their configurations.

Reason for MySQL is that it is an area I am comfortable with from work, but it is probably overkill for this application.

When it comes to update, I don't get your question. Swapping out jquery that is used on remotes is just placing the new files in correct folder for static files and update references.

Updating django on the other hand is not trivial (I think)
My home automation software: Automagically @ Raspberry Pi http://automagically.weebly.com

KHolm
Posts: 163
Joined: Thu Jan 18, 2018 11:08 am

Re: Developer discussion - Hanging Threads

Post by KHolm » Thu Jan 18, 2018 11:08 am

davka003 wrote: When it comes to update, I don't get your question. Swapping out jquery that is used on remotes is just placing the new files in correct folder for static files and update references.
I think that is more a competency issue on my side. I noticed that there did exist a new version of jQuery and thought of making an attempt to update. Having downloaded the files I could not understand how to replace current version. Most likely I downloaded the wrong files or something.

Bottom line is that I don't know/understand to use the powers of jQuery with Automagically. Say that I wanted to use Black on white instead of White on black, or change some other visual - how do I go about it.

So once again, a competency gap on my side and a lack of instructions :-)

/Marcus

KHolm
Posts: 163
Joined: Thu Jan 18, 2018 11:08 am

Re: Developer discussion - Hanging Threads

Post by KHolm » Thu Jan 18, 2018 11:08 am

KHolm wrote: [My instance of] Automagically seems to enter some hanging state every now and then. It seems to be rather random and the effect is a unresponsive system.
I notice that some of the plugins, such as the tellduslive plugin, executes quite a lot of code in the signalHandler. I wonder if that is working well in this design pattern; e.g. when waiting on a socket, the signalling system is "hanging".
I think at least the tellduslive plugin needs some refactoring to reduce the risk of a hanging system.
I have now had the system up for more than a week which is a new record for me. A week ago I disabled the Telldus-live plugin (I don't need it anyway) and since then it is stable as a rock. It feeds Xively ca. every minute, both using Wireless Energy Meter Plugin as well as with temperatures from internal variables.

Anyone else experience problems when Telldus-live plugin is enabled? As said, I think it needs to be re-factored but it'd be good to quantify the problem before someone spends time chasing ghosts.

/Marcus

davka003
Posts: 185
Joined: Thu Jan 18, 2018 11:08 am
Location: Sweden
Contact:

Re: Developer discussion - Hanging Threads

Post by davka003 » Thu Jan 18, 2018 11:08 am

I have had it running several months now (since Christmas) without any problems and I got Telldus live activated.
My home automation software: Automagically @ Raspberry Pi http://automagically.weebly.com

jimmy927
Posts: 27
Joined: Thu Jan 18, 2018 11:08 am
Location: Stockholm
Contact:

Re: Developer discussion - Hanging Threads

Post by jimmy927 » Thu Jan 18, 2018 11:08 am

KHolm wrote:I am thinking that the mySql may be a bit heavy on the Pi, thinking that possibly a sqlite would be as good; possibly with a lighter httpd (such as lighthttpd). But I have no experience with Django so I don't even know it the above would work. And, likely it's too much work to replace...
If i look at my Pi CPU resources are not spent in SQL-database but in Apache and/or Django, but i honestly don't understand where all time is spent here.
It typically uses a lot of CPU and very little I/O.

Switching to sqlite is propably easy, but sqlite locks the entire DB even for simple operations, i think we might get plenty of blocking http requests if we do.
I think there is something fishy about performance, but i doubt MySQL is to blame - i think wee should look elsewhere, starting with Apache. Maybe have a look at nginx ?

jimmy927
Posts: 27
Joined: Thu Jan 18, 2018 11:08 am
Location: Stockholm
Contact:

Re: Developer discussion - Hanging Threads

Post by jimmy927 » Thu Jan 18, 2018 11:08 am

jimmy927 wrote:i think wee should look elsewhere, starting with Apache. Maybe have a look at nginx ?
Actually, if i turn off Apache and just start the django devserver instead perforamnce increases.
Anyone else seeing the same thing ?

Post Reply