Need help to measure and tune the latency in Linux RT

[Ashoka K]

Hi,

I have an embedded system running 2.6.33 cross compiled for ARM OMAP
with 1 GHZ uni-processor system.

The application has many threads and processes (around 15 total). In
that 1 thread is time critical and must run every interval confiurable
as 2, 4 or 8 milli sec etc. There is another thread which transfers
Images to a FTP or to HMI etc, whoever requested the image.
With this image transfer enabled my critical process misses the
deadline and timesout. This critical thread is at RR priority 50. I
used a HR timer thread with RR pri 55 to wakeup the critical thread
every 1 milli sec to do the job.
If image transfer is enabled i see that real time thread misses deadlines.

All traffic is flowing on common ethernet connection. Does it affect
in any way ?

How to find out where the kernel is waiting or delaying to schedule
the critical thread even though it is at higher RR priority. How to
measure the latency ?

I am trying to use Oprofile tool, but got some error in cross
compiling. Doesn't these tools add their own latency to the original
problem ?

Any tools ? which one is better. Please provide your input on this.


Regards
Ashoka. K
--
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

[Jason Cooper]

Ashoka,

linux-embedded is pretty quiet. I'm not even sure why I'm still
subscribed to it. You'll have better luck with your question by asking
on linux-arm-kernel (added to the cc). I believe there is also a

Post by Ashoka K
Hi,
I have an embedded system running 2.6.33 cross compiled for ARM OMAP
with 1 GHZ uni-processor system.

Wow, this is ancient. Is this a vanilla kernel? If it's a vendor
kernel, you'll need to ask the vendor.

Post by Ashoka K
The application has many threads and processes (around 15 total). In
that 1 thread is time critical and must run every interval confiurable
as 2, 4 or 8 milli sec etc. There is another thread which transfers
Images to a FTP or to HMI etc, whoever requested the image.
With this image transfer enabled my critical process misses the
deadline and timesout. This critical thread is at RR priority 50. I
used a HR timer thread with RR pri 55 to wakeup the critical thread
every 1 milli sec to do the job.
If image transfer is enabled i see that real time thread misses deadlines.
All traffic is flowing on common ethernet connection. Does it affect
in any way ?
How to find out where the kernel is waiting or delaying to schedule
the critical thread even though it is at higher RR priority. How to
measure the latency ?
I am trying to use Oprofile tool, but got some error in cross
compiling. Doesn't these tools add their own latency to the original
problem ?
Any tools ? which one is better. Please provide your input on this.

I'm not the guy to answer this ;-) I'll just forward to the appropriate
lists.

hth,

Jason.

[1] http://vger.kernel.org/vger-lists.html#linux-rt-users
--
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

[Stanislav Meduna]

Post by Ashoka K
This critical thread is at RR priority 50.

Try to put it higher than the irq thread handling your HD hardware.
I've even encountered hardware where I had to put the ATA thread
in the SHED_OTHER domain - see [1] for the thread.

Post by Ashoka K
I used a HR timer thread with RR pri 55 to wakeup the critical thread
every 1 milli sec to do the job.

Which prio is the ksoftirqd running at? The HRTIMER interrupts are
routed through it so it also has to be high enough.

I got burned with this myself [2]

Regards
Stano

[1] http://www.spinics.net/lists/linux-rt-users/msg08711.html
[2] http://www.spinics.net/lists/linux-rt-users/msg08745.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Ashoka K]

Thanks for all the replies.
I am stuck with this issue for more tan a month. Explained this in
more detail at:

http://stackoverflow.com/questions/15805231/need-to-improve-the-linux-performance-for-embedded-system

But with no response from anyone.

I use PREEMPT patch for 2.6.33 vanilla kernel.

I have pasted all the processes running on my system:

PID USER VSZ STAT COMMAND
1 root 3192 S init
2 root 0 SW [kthreadd]
3 root 0 SW [ksoftirqd/0]
4 root 0 SW [events/0]
5 root 0 SW [khelper]
8 root 0 SW [async/mgr]
84 root 0 SW [sync_supers]
86 root 0 SW [bdi-default]
88 root 0 SW [kblockd/0]
91 root 0 SW [omap2_mcspi]
116 root 0 SW [rpciod/0]
123 root 0 SW [kswapd0]
124 root 0 SW [aio/0]
125 root 0 SW [nfsiod]
126 root 0 SW< [kslowd001]
127 root 0 SW< [kslowd000]
128 root 0 SW [crypto/0]
267 root 0 SW [mtdblockd]
351 root 0 SWN [jffs2_gcd_mtd8]
359 root 3192 S /sbin/syslogd
361 root 3192 S /usr/sbin/telnetd
364 root 1996 S vsftpd
410 root 43844 S my_appl
416 root 56412 S < my_appl
417 root 84804 S my_appl
418 root 43844 S my_appl
419 root 84824 S my_appl
430 root 46552 S < my_appl

The process my_appl is my application processes.

the ksoftirq is running with SCHED_OTHER.

$ chrt -p 3
pid 3's current scheduling policy: SCHED_OTHER
pid 3's current scheduling priority: 0

I will test by changing ksoftirq priority to see how it works.

Regards
Ashoka. K

Post by Stanislav Meduna

Post by Ashoka K
This critical thread is at RR priority 50.

Try to put it higher than the irq thread handling your HD hardware.
I've even encountered hardware where I had to put the ATA thread
in the SHED_OTHER domain - see [1] for the thread.

Post by Ashoka K
I used a HR timer thread with RR pri 55 to wakeup the critical thread
every 1 milli sec to do the job.

Which prio is the ksoftirqd running at? The HRTIMER interrupts are
routed through it so it also has to be high enough.
I got burned with this myself [2]
Regards
Stano
[1] http://www.spinics.net/lists/linux-rt-users/msg08711.html
[2] http://www.spinics.net/lists/linux-rt-users/msg08745.html

--
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

[ddegraff]

Post by Ashoka K
Hi,
I have an embedded system running 2.6.33 cross compiled for ARM OMAP
with 1 GHZ uni-processor system.

A project I am working on is using the same vintage kernel with RT
patches applied, though on DaVinci (omap l138).

I, myself, however, have not dealt with any of that work, but I will ask
the other developer if he has any suggestions.

Post by Ashoka K
The application has many threads and processes (around 15 total). In
that 1 thread is time critical and must run every interval confiurable
as 2, 4 or 8 milli sec etc. There is another thread which transfers
Images to a FTP or to HMI etc, whoever requested the image.
With this image transfer enabled my critical process misses the
deadline and timesout. This critical thread is at RR priority 50. I
used a HR timer thread with RR pri 55 to wakeup the critical thread
every 1 milli sec to do the job.
If image transfer is enabled i see that real time thread misses deadlines.
All traffic is flowing on common ethernet connection. Does it affect
in any way ?
How to find out where the kernel is waiting or delaying to schedule
the critical thread even though it is at higher RR priority. How to
measure the latency ?
I am trying to use Oprofile tool, but got some error in cross
compiling. Doesn't these tools add their own latency to the original
problem ?
Any tools ? which one is better. Please provide your input on this.
Regards
Ashoka. K
--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

[Ashoka K]

Hello Android Techies and others,

I am still debugging this issue to understand the traces generated. I
am debugging at this level for first time, and now i am bit familiar
with traces.

To summarize again:

I have ARM OMAP embedded system, running vanilla Linux 2.6.33-29 cross
compiled with PREEMPT patch. There is a application, say Thread_1 (RR,
with highest priority) which need to send and receive data every 2ms,
or 4ms or 8 ms etc. (configurable). This data is small and atmost it
will be 1 ethernet frame (1500 bytes). So i expect this application to
execute for very short time but regularly without missing deadlines
(There is deviation allowed with watchdog timeout but in worst cases).
There is another process running (RR, lower pri than Thread_1) on same
system which transfers images on request from outside users/HMI. When
this is enabled and i see a frequesnt timeout for the time critical RT
thread_1. Even if this image transfer is not enabled then also i see
Kernel Oops after long time, say 1 day.
All data is flowing on ethernet.

I had increaed the KSOFTIRQD policy and priority to be more than my
time critical application , but it didn't help much. Now i am trying
to understand the debug traces generated to know where the latency is
coming from.

Any suggestions is greatly appreciated.

Regards
Ashoka. K


On Mon, May 20, 2013 at 12:32 AM, Android Techies

Hello Ashoka,
You can try to measure the hrtimer latency and see whether there is a delay
in hrtimer itself. I will suggest to try and make Oprofile working. If you
have already figured out a way please let me know as well as I need similar
stuff.

Post by Ashoka K
Hi,
I have an embedded system running 2.6.33 cross compiled for ARM OMAP
with 1 GHZ uni-processor system.
The application has many threads and processes (around 15 total). In
that 1 thread is time critical and must run every interval confiurable
as 2, 4 or 8 milli sec etc. There is another thread which transfers
Images to a FTP or to HMI etc, whoever requested the image.
With this image transfer enabled my critical process misses the
deadline and timesout. This critical thread is at RR priority 50. I
used a HR timer thread with RR pri 55 to wakeup the critical thread
every 1 milli sec to do the job.
If image transfer is enabled i see that real time thread misses deadlines.
All traffic is flowing on common ethernet connection. Does it affect
in any way ?
How to find out where the kernel is waiting or delaying to schedule
the critical thread even though it is at higher RR priority. How to
measure the latency ?
I am trying to use Oprofile tool, but got some error in cross
compiling. Doesn't these tools add their own latency to the original
problem ?
Any tools ? which one is better. Please provide your input on this.
Regards
Ashoka. K
--
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

--
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs