Quantcast

DuoVero-Y COM freezes under WiFi load

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

DuoVero-Y COM freezes under WiFi load

Daniel Madill

I have been having trouble with the DuoVero-Y COM module freezing under heavy WiFi load. I am able to replicate the issue repeatedly using the open source iperf3 application (https://iperf.fr/iperf-download.php#windows), for which I have attached a version I compiled from source for the DuoVero-Y. During testing the DuoVero-Y is communicating with a Windows machine through a wireless router. Communication works fine for many minutes and then all of a sudden the DuoVero stops responding. The HDMI display goes blank and the blue flashing LED beside the HDMI connector stops flashing and goes out. The green LED at the corner of the WiFi module on the gumstix board also goes out. The processor, which runs fairly hot, goes cold. I thought it might be thermal, but using a fan that keeps it running around 41 C does not change the behaviour (measured using a thermal camera). The serial console becomes unresponsive, yet there are no kernel messages dumped to the serial console. I am running a very recent (Friday, Nov. 18 12:15:34) version of the master branch which I compiled as a gumstix-console-image using Yocto. I did not make any changes other than to select duovero as the MACHINE in local.conf. The image runs fine under normal use.

 

To replicate the freezing behaviour:

 

1.       Copy iperf3 and libiperf.so.0 to the DuoVero-Y COM. Use chmod to make them executable.

2.       Copy libiperf.so.0 to /usr/lib.

3.       Create a symbolic link:

ln –s /lib/ld-linux-armhf.so.3 /lib/ld-linux.so.3

4.       Download the 64-bit Windows version of iperf3 (latest version: 3.1.3) from the web link above onto a Windows desktop and extract the zip file.

5.       In a Command Prompt on the Window machine, go to the directory where iperf3 was unarchived and run:

iperf3 –s

                This runs iperf3 as a server.

6.       On the DuoVero-Y COM, go to the directory where iperf3 was copied and run the following command:

iperf3 –c <IP address of Windows machine> -l 1000k –P 100 –t 4000

                I usually get failures within about 360 seconds. Use ./iperf3 --help to get help on the iperf3 command-line options.

 

The test creates 100 connections and sends data repeatedly from the DuoVero-Y to the Windows machine, displaying statistics as it does so. The WiFi performance is about 18-20 Mbps in my tests based on the Windows Task Manager Performance tab in Windows 10. The iperf3 application reports lower performance. Our application does not behave the same way as iperf3 (100 parallel connections is a bit ridiculous), but it is the only way I could find to replicate the freezing behaviour reliably using iperf3 (we stream live video). The nice thing about iperf3 is you have the full source if you need it for debugging and it gives you some performance statistics at the same time.

 

I can understand if the WiFi cannot keep up with the amount of data, but having the entire O/S freeze is not a graceful failure. My concern is that there may be a race condition in the WiFi kernel driver that is being hit. If that’s the case, then we may get less frequent failures under lighter load. We’re using the gumstix in a UAV, so catastrophic failures like that would be problematic.

 

Any help resolving this issue would be greatly appreciated.

 

Sincerely,

 

Daniel Madlil


------------------------------------------------------------------------------

_______________________________________________
gumstix-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gumstix-users

iperf3 (35K) Download Attachment
libiperf.so.0 (473K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: DuoVero-Y COM freezes under WiFi load

allsey87
My first thoughts are that the system + wifi load is causing too much current to be drawn and you are browning out. I see similar behaviour when the batteries in my system start getting low, and the extra pull of current from the wifi is often enough to crash things like you are describing. What is your power setup? Have you measured current draw and the supply/battery voltage with a scope?

On 23 November 2016 at 10:40, Daniel Madill <[hidden email]> wrote:

I have been having trouble with the DuoVero-Y COM module freezing under heavy WiFi load. I am able to replicate the issue repeatedly using the open source iperf3 application (https://iperf.fr/iperf-download.php#windows), for which I have attached a version I compiled from source for the DuoVero-Y. During testing the DuoVero-Y is communicating with a Windows machine through a wireless router. Communication works fine for many minutes and then all of a sudden the DuoVero stops responding. The HDMI display goes blank and the blue flashing LED beside the HDMI connector stops flashing and goes out. The green LED at the corner of the WiFi module on the gumstix board also goes out. The processor, which runs fairly hot, goes cold. I thought it might be thermal, but using a fan that keeps it running around 41 C does not change the behaviour (measured using a thermal camera). The serial console becomes unresponsive, yet there are no kernel messages dumped to the serial console. I am running a very recent (Friday, Nov. 18 12:15:34) version of the master branch which I compiled as a gumstix-console-image using Yocto. I did not make any changes other than to select duovero as the MACHINE in local.conf. The image runs fine under normal use.

 

To replicate the freezing behaviour:

 

1.       Copy iperf3 and libiperf.so.0 to the DuoVero-Y COM. Use chmod to make them executable.

2.       Copy libiperf.so.0 to /usr/lib.

3.       Create a symbolic link:

ln –s /lib/ld-linux-armhf.so.3 /lib/ld-linux.so.3

4.       Download the 64-bit Windows version of iperf3 (latest version: 3.1.3) from the web link above onto a Windows desktop and extract the zip file.

5.       In a Command Prompt on the Window machine, go to the directory where iperf3 was unarchived and run:

iperf3 –s

                This runs iperf3 as a server.

6.       On the DuoVero-Y COM, go to the directory where iperf3 was copied and run the following command:

iperf3 –c <IP address of Windows machine> -l 1000k –P 100 –t 4000

                I usually get failures within about 360 seconds. Use ./iperf3 --help to get help on the iperf3 command-line options.

 

The test creates 100 connections and sends data repeatedly from the DuoVero-Y to the Windows machine, displaying statistics as it does so. The WiFi performance is about 18-20 Mbps in my tests based on the Windows Task Manager Performance tab in Windows 10. The iperf3 application reports lower performance. Our application does not behave the same way as iperf3 (100 parallel connections is a bit ridiculous), but it is the only way I could find to replicate the freezing behaviour reliably using iperf3 (we stream live video). The nice thing about iperf3 is you have the full source if you need it for debugging and it gives you some performance statistics at the same time.

 

I can understand if the WiFi cannot keep up with the amount of data, but having the entire O/S freeze is not a graceful failure. My concern is that there may be a race condition in the WiFi kernel driver that is being hit. If that’s the case, then we may get less frequent failures under lighter load. We’re using the gumstix in a UAV, so catastrophic failures like that would be problematic.

 

Any help resolving this issue would be greatly appreciated.

 

Sincerely,

 

Daniel Madlil


------------------------------------------------------------------------------

_______________________________________________
gumstix-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gumstix-users



------------------------------------------------------------------------------

_______________________________________________
gumstix-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gumstix-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: DuoVero-Y COM freezes under WiFi load

Daniel Madill
I'm using the 5V 1A wall adapter, not a battery at the moment (although I will be using battery power once I've confirmed I can get everything I need working). I did not suspect a brown out because it can run for more than 5 minutes before shutting down, but I will investigate that as a possibility.

Thank you for the suggestion.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: DuoVero-Y COM freezes under WiFi load

Daniel Madill
I replaced the wall adapter with a laboratory power supply. The average current under load is about 0.5A, which is half the rating of the wall adapter, but the average current reported by the power supply can get up to 0.74A very briefly and only rarely (only occurred once so far during the test). I suspect therefore that there are current spikes that exceed the 1A very briefly, but long enough to cause a brownout as suggested. I haven't hooked up a scope yet to confirm. Using the laboratory power supply I've been able to run the test for much longer (it's still running), so power supply issues do appear to be the culprit. Ironically, if I had started with the battery I would probably not have seen the issue without warning as we monitor the battery voltage and the batteries have higher current capacity than the wall adapter.

Thank you for the help.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: DuoVero-Y COM freezes under WiFi load

allsey87
Hi Daniel,

Yes, those numbers sound about right. Different things can cause the current draw to spike and it may be a combination of things happening at the same time. For instance I noticed that when I run lsusb on my board, it can cause a jump to around 0.7-0.8A, so adding a heavy wifi load on to that would certainly exceed your 1A input.

On 24 November 2016 at 03:59, Daniel Madill <[hidden email]> wrote:
I replaced the wall adapter with a laboratory power supply. The average
current under load is about 0.5A, which is half the rating of the wall
adapter, but the average current reported by the power supply can get up to
0.74A very briefly and only rarely (only occurred once so far during the
test). I suspect therefore that there are current spikes that exceed the 1A
very briefly, but long enough to cause a brownout as suggested. I haven't
hooked up a scope yet to confirm. Using the laboratory power supply I've
been able to run the test for much longer (it's still running), so power
supply issues do appear to be the culprit. Ironically, if I had started with
the battery I would probably not have seen the issue without warning as we
monitor the battery voltage and the batteries have higher current capacity
than the wall adapter.

Thank you for the help.





--
View this message in context: http://gumstix.8.x6.nabble.com/DuoVero-Y-COM-freezes-under-WiFi-load-tp4971165p4971168.html
Sent from the Gumstix mailing list archive at Nabble.com.

------------------------------------------------------------------------------
_______________________________________________
gumstix-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gumstix-users


------------------------------------------------------------------------------

_______________________________________________
gumstix-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gumstix-users
Loading...