Page tree

Question: "What steps can you do to get the best performance for your Linux system (running CODESYS runtime)?"
Answer:    "It is not that simple...! But step by step:..."

What can I do to improve scheduling of my system / runtime / IEC application?

If your IEC application has certain requirements regarding realtime, it is clear, that the underlying operating system and runtime system must help to achieve these requirements.

So let's check the different levels step by step from bottom to top:

  • System setup (Linux)
  • Runtime setup
  • IEC Application
  • Fieldbus specific (e.g. EtherCAT / ProfiNet)


System setup / basic configuration:

There are some measures you should take to setup and configure your base system, that are very important.

  • Use a realtime kernel

We recommend using the RT-Preempt Kernel (https://rt.wiki.kernel.org) for your Linux system.
On debian / ubuntu distributions, you can find a rt-kernel as package, that can easily be installed via "apt". See your distribution manual for details about that.

    • On debian systems:

      sudo apt-get install linux-image-rt-amd64
    • On Ubuntu systems:

      sudo apt-get install linux-lowlatency
    • you can check what kernel you are using, e.g. with the command

      uname -a
  • Don't use any kind of window-manager / GUI / X-server etc. on your system

    → this might harm the realtime capabilities of your system (resulting in high jitter on the IEC application)

  • Disable energy-saving modes in kernel / scheduling governors:

    e.g.: use "performance" scheduling governor (see https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt)
    There are also tools like "cpu-freq-utils" that can be used.
    This might interfere with intel pstate drivers (they normally require a different approach to pin the CPU frequency).

    You can check what scheduling/scaling_governor is used with:

    cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor


    You can set it (as root/admin) to "performance":

    echo "performance" > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
    # set it for all available cores:
    echo "performance" > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
    echo "performance" > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
    echo "performance" > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor

    Be aware that the scaling_governor needs to be set after each startup of the system if you do it like this. You might also set it via kernel configuration.

    "intel_pstate" kernel driver might make it hard to set it properly → double-check your configuration with "cpufreq-info"!

  • Disable frequency scaling / switching as far as possible:

    e.g.: set min and max CPU freq. to a fixed value (switching CPU freq might cause jitter in the system)

  • Isolating the relevant CPUs (from other kernel worker / other threads)

    Identify the relevant CPU that you want your high-prio job to work on, e.g. the IEC Task with the EtherCAT Master.
    Use the CODESYS MultiCore feature and pin the IEC Task to this core.

    Then also isolate this CPU for the kernel, with the kernel command line argument "isolcpus" → https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html?highlight=isolcpu

    Your kernel must be built with CONFIG "ISOL" enabled!

    For example, one can add the following as kernel parameter to isolate CPU 1 and 2:

    isolcpus=1,2

    If you use GRUB (Bootloader), you can change the kernel parameters e.g. with

    gksudo gedit /etc/default/grub

    Then find the command line for your kernel and add the parameter "isolcpus".
    You can find further information about the GRUB bootloader at: https://www.gnu.org/software/grub/


  • For PC systems, we further recommend:
    • disable HyperV in BIOS (if available)

    • disable all energy-saving modes in BIOS (if available), e.g. check the "C/P" States of your CPU

  • For advanced users, we recommend diving into these aspects of Kernel configuration:

    We recommend that you dive into these topics/keywords and check if you can use these aspects to further improve your system regarding realtime capabilities.

    • PREEMPT_FULL
    • isolcpu
    • rcu_nocbs
    • rcu_nocb_poll
    • nosoftlockup
    • irqbalance disable
    • kernel.sched_rt_runtime_us


You can verify the realtime capabilities of your system with the official tool "cyclictest" → https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/cyclictest/start

Cyclictest is part of this debian package:

sudo apt install rt-tests

In case you see problems, we recommend using mechanisms/tools like "kernel trace / kernelshark": https://www.kernel.org/doc/html/latest/trace/index.html
With these tools, you can identify the possible culprits of realtime violations.

If you want to learn more, we recommend the official kernel and rt-preempt documentation:



Linux kernel network driver:

If an Ethernet based fieldbus is involved, of course the performance of the network driver is also important. No message should be "late" or even lost. 

Known issues:

We know that in certain combinations of kernel and network drivers, different problems occurred that lead to loss of Ethernet packages and or too late message sending/receiving.

We observed this behavior with kernel versions 5.10 (more often) and 4.19 (less often), and depending on what network driver used (e.g. intel e1000e or igb kernel drivers).

→ we are currently working on some testcode that you can run to check if you see this effect on your system.



Runtime setup

If you need to, you can set some configuration in the CODESYS runtime system to influence realtime capabilities:

  • Setting "disable_dma_latency":
    This prevents sleep modes / energy saving modes of the CPU and reduces jitter.

    You can enable this in the CODESYS runtimes with the setting (cfg file):

    [SysCpuHandling]
    Linux.DisableCpuDmaLatency=1
  • PLC shell commands "irq-list" and "irq-setprio"
    Especially for Ethernet based field-busses it can make sense to set the prio of the interrupts used by the network driver to the same prio (or one higher) than the IEC task using this.



IEC Application configuration

Of course, it is also important to configure your IEC application in a way to optimize for realtime requirements.

Focus of priorities

Don't expect the system to focus on too many "highest priority" things at once.

Ask yourself:

  • What really needs to have highest prio?
  • What is the second most important aspect etc...

E.g.: Only configure a single aspect with the highest priority.
E.g. If the EtherCAT fieldbus is the most important player in your IEC application, you should not add other aspects inside your IEC application "higher prio" than that.

You might harm the realtime capabilities of your system with too many high prio tasks in parallel!


Choose the priorities wisely

Set the priorities of the different Tasks / aspects in a reasonable way (relative to each other but also absolute!)

Configure the highest priority task as high as required, but as low as possible (relative to the next priority tasks). 


Mapping of IEC Task priorities and Linux thread-priorities:

IEC Task priority

Linux priority (default)

88 (SCHED_FIFO)

57 (SCHED_FIFO)

0 (highest realtime Prio)

56 (SCHED_FIFO)

15 (lowest realtime Prio)

41 (SCHED_FIFO)

16 (non realtime Prio)

0 (SCHED_OTHER)

31 (non realtime Prio)

0 (SCHED_OTHER)

0 (SCHED_OTHER)

Hints

  • Default linux interrupt priorities are on priority "50" (that means IEC priority "6")
    This typically includes network interrupts (for sending and receiving network packages, e.g. for EtherCAT) → see chapter below for details

  • IEC Task priorities from 16-31 are non-realtime (using SCHED_OTHER) and are mapped "only" with nice levels.

EtherCAT Fieldbus specific configuration

If you are using our EtherCAT Master, you might want to optimize your system for EtherCAT use, and there are some settings and hints how to achieve this.
But there are also some diagnostic mechanisms to help you detect what goes wrong, e.g. if you see lost packages, long sending times etc.

To enable the enhanced debug mechanisms of the EtherCAT Master, you have to set some "IEC compiler defines":

  • right-click on your Application → Properties → Build and enter "EtherCATPerformance, EtherCATSyncDebug"

To see additional information, you should also enable the generic options editor:

  • Tools → Options → Device Editor → Show generic device configuration views


Inside a watch window, add these variables and observe them:

Variable nameDescription...

EtherCAT_Master.m_dwBeforeReadInputs

Time in us to read all messages (contains also GetEthFrame)


EtherCAT_Master.m_dwBeforeWriteOutputs

Time in us to do the state machines


EtherCAT_Master.m_dwAfterWriteOutputs

Time in us to send all data (contains also SendEthFrame)


EtherCAT_Master.m_Master.m_dwSendEthFrame

Time in us for runtime Sendethframe call


EtherCAT_Master.m_Master.m_dwGetEthFrame 

Time in us for runtime getethframe call.


EtherCAT_Master.m_Master.m_diMaxSyncTime

EtherCAT_Master.m_Master.m_diMinSyncTime

EtherCAT_Master.m_Master.m_diSyncBuffer