CPU lumbered by kworker in pre-built image

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

CPU lumbered by kworker in pre-built image

cookednoodle
Hello,

I have installed the pre-built Yocto 1.8 image from here and have run it on a FireSTORM and an IronSTORM-P connected to a Tobi board. Both are slowed to a crawl by a very busy kworker (according to "top"). I do not see this problem with the Ubuntu image nor the images on the NAND of the COMs. I am currently compiling a Yocto 1.7 image to see if the problem extends there.

Anyone else seeing this behavior? I don't yet see how I could be causing this myself. I copy the files to a card. I insert it. I apply power to the Tobi and watch over the serial terminal.

Thank you,

Tim
Reply | Threaded
Open this post in threaded view
|

Re: CPU lumbered by kworker in pre-built image

Scott Ellis
There is nothing inherently wrong with Yocto 1.8.1 built Overo images
using a 3.18 kernel.

I'm sitting in front of one now.

root@riid:~# cat /etc/issue
Poky (Yocto Project Reference Distro) 1.8.1 \n \l

root@riid:~# uname -a
Linux riid 3.18.24-riid #1 SMP Mon Nov 9 15:18:36 EST 2015 armv7l GNU/Linux

root@riid:~# uptime
 15:28pm  up   0:06,  1 user,  load average: 0.00, 0.04, 0.04

root@riid:~# free
              total        used        free      shared  buff/cache   available
Mem:         499940        6172      462524         372       31244      476060
Swap:             0           0           0


This is a custom built image, not one from the Gumstix downloads.
But it uses most of the same Gumstix patches to the linux-stable kernel.
I took them from meta-gumstix.

"cookednoodle" <[hidden email]> said:

> Hello,
>
> I have installed the pre-built Yocto 1.8 image from  here
> <https://www.gumstix.com/software/software-downloads/>   and have run it on
> a FireSTORM and an IronSTORM-P connected to a Tobi board. Both are slowed to
> a crawl by a very busy kworker (according to "top"). I do not see this
> problem with the Ubuntu image nor the images on the NAND of the COMs. I am
> currently compiling a Yocto 1.7 image to see if the problem extends there.
>
> Anyone else seeing this behavior? I don't yet see how I could be causing
> this myself. I copy the files to a card. I insert it. I apply power to the
> Tobi and watch over the serial terminal.
>
> Thank you,
>
> Tim
>
>
>
> --
> View this message in context:
> http://gumstix.8.x6.nabble.com/CPU-lumbered-by-kworker-in-pre-built-image-tp4970435.html
> Sent from the Gumstix mailing list archive at Nabble.com.
>
> ------------------------------------------------------------------------------
> Presto, an open source distributed SQL query engine for big data, initially
> developed by Facebook, enables you to easily query your data on Hadoop in a
> more interactive manner. Teradata is also now providing full enterprise
> support for Presto. Download a free open source copy now.
> http://pubads.g.doubleclick.net/gampad/clk?id=250295911&iu=/4140
> _______________________________________________
> gumstix-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gumstix-users
>



------------------------------------------------------------------------------
Presto, an open source distributed SQL query engine for big data, initially
developed by Facebook, enables you to easily query your data on Hadoop in a
more interactive manner. Teradata is also now providing full enterprise
support for Presto. Download a free open source copy now.
http://pubads.g.doubleclick.net/gampad/clk?id=250295911&iu=/4140
_______________________________________________
gumstix-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gumstix-users
Reply | Threaded
Open this post in threaded view
|

Re: CPU lumbered by kworker in pre-built image

cookednoodle
Thank you, Scott

---

I went through the process again:

1) Deleted my previous "yocto" folder
2) Followed the instructions in yocto-manifest/README.md
3) Got a few certificate and license errors after bitbaking
4) Created an image with wic and wrote it to an SD card
5) Booted an Overo IronSTORM-P connected to a Tobi connected via serial
6) Looked over boot messages
   -  "Bluetooth: module_cfg_cmd(f1) failed"
7) Logged in as root and ran a few commands
        root@overo:~# cat /etc/issue
        Poky (Yocto Project Reference Distro) 1.8 \n \l
       
        root@overo:~# uname -a
        Linux overo 3.18.18-custom #1 SMP Mon Nov 9 15:45:33 CST 2015 armv7l GNU/Linux

        root@overo:~# uptime
         23:39:10 up 4 min,  1 user,  load average: 1.13, 0.74, 0.33

        root@overo:~# free
                      total        used        free      shared  buff/cache   available
        Mem:         499472       11920      416888        8680       70664      457576
        Swap:        524284           0      524284

8) Waited a while and reran uptime
        root@overo:~# uptime
         23:55:40 up 21 min,  1 user,  load average: 0.83, 0.86, 0.69

---

I will investigate ways of finding out what the kernel is up to.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
Reply | Threaded
Open this post in threaded view
|

Re: CPU lumbered by kworker in pre-built image

cookednoodle
I get the same results on an Overo FireSTORM.

I have run these commands to find out what the kworker is doing:

     root@overo:~# echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
     root@overo:~# cat /sys/kernel/debug/tracing/trace_pipe > out.txt

The resulting file contains this entry every 0.5 ms:

     kworker/0:4-995   [000] d...   387.186768: workqueue_queue_work: work struct=de1e2d48 function=pm_runtime_work workqueue=de00c600 req_cpu=2 cpu=0

So, it would appear my problem has something to do with runtime power management. However, I still have no idea why I would be having this problem while others are not.
Reply | Threaded
Open this post in threaded view
|

Re: CPU lumbered by kworker in pre-built image

Ash Charles
Hi,

It looks like enabling SDIO interrupts [1] on the wireless interface
causes spurious activity that keeps a kworker thread busy (Florian
alludes to this in this [2] discussion).  This affects Overo series
COMS using a Wi2Wi-based wireless chips---i.e. the '-P' series COMs
using W2CBW0015 and original COMs using W2CBW003, not '-Y' series COMs
using Wilink8.  Simply commenting out the 'interrupts-extended' line
in the omap3-overo-base.dtsi file stops the kworker from hogging the
CPU for me; wireless throughput drops though.
By reworking the line to directly reference the padmux, it seems to
restore SDIO interrupts without a busy kworker on the '-P' COMs for
me:

interrupts-extended = <&intc 86 &omap3_pmx_core 0x12e>; /* padconf
offset for mmc2.dat1 */

Unfortunately, the libertas driver for the W2CBW003 chip doesn't seem
to like this change on the 3.18 kernel.

[1] https://github.com/gumstix/linux/commit/ae434177b9ef7784b8e874419babc8d23dd9b56a
[2] http://www.spinics.net/lists/linux-mmc/msg26858.html

--Ash

On Wed, Dec 9, 2015 at 2:28 PM, cookednoodle <[hidden email]> wrote:

> I get the same results on an Overo FireSTORM.
>
> I have run these commands to find out what the kworker is doing:
>
>      root@overo:~# echo workqueue:workqueue_queue_work >
> /sys/kernel/debug/tracing/set_event
>      root@overo:~# cat /sys/kernel/debug/tracing/trace_pipe > out.txt
>
> The resulting file contains this entry every 0.5 ms:
>
>      kworker/0:4-995   [000] d...   387.186768: workqueue_queue_work: work
> struct=de1e2d48 function=pm_runtime_work workqueue=de00c600 req_cpu=2 cpu=0
>
> So, it would appear my problem has something to do with *runtime power
> management*. However, I still have no idea why I would be having this
> problem while others are not.
>
>
>
> --
> View this message in context: http://gumstix.8.x6.nabble.com/CPU-lumbered-by-kworker-in-pre-built-image-tp4970435p4970499.html
> Sent from the Gumstix mailing list archive at Nabble.com.
>
> ------------------------------------------------------------------------------
> _______________________________________________
> gumstix-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gumstix-users

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
gumstix-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gumstix-users
Reply | Threaded
Open this post in threaded view
|

Re: CPU lumbered by kworker in pre-built image

a71128
Hi Ash,

I am facing the same SDIO overhead problem caused by W2CBW003.
I have tried turning the wifi chip off with rfkill and it does relieve the problem.
Is there any update on how to fix the problem on 3.18?
Also, I would like to ask if you happen to know how to modify device tree using Yocto.

Thank you.

Best,
Jeffery Hsu