How to reduce jitter on LinucCNC?

linuxcnc, jitter, hardware  · 

You know, jitter is a key metric for CNC controller. The lower the jitter, the smoother the machine will operate. However, this is not always obvious how to lower it to achieve something less than 50k. Especially under a full load. Here are some tips with rough estimation of its impact.

Important: jitter/latency and the system performance are opposite things. Less latency you have - slower system will be. Kernel compilation is a good example: on regular kernel with jitter ~20M+ it takes 1.5 hours, on PREEMPT_FULL with isolcpus and other tweaks - more than 3 hours.

1. Use PREEMPT-RT kernel

This is the most obvious thing. Main point: do not mix up built in PREEMPT kernel modes like PREEMPT_VOLUNTARY/PREEMPT/PREEMPT_RT with PREEMPT_FULL which is a part of PREEMPT-RT patch. Use only PREEMPT_FULL if you want to achieve the lowest jitter.

Some jitter comparisons (for my particular hardware):

As you see, even out of the box RT kernel is ok for indirect machine control. For example, together with Mesa board usage. With more tweaks it might be suitable even for a direct control of stepper motors.

Jitter vs network latency

Important: for Mesa-like installation jitter is not the main metric. Network latency is more important. Usually, jitter related with the latency, but not always. Sometimes need to tweak network driver and related parts. See r8168-dkms package for details.

To measure network latency do the following check

sudo chrt 99 ping -i .001 -q 10.10.10.10 -c 50000

Look at maximum latency value. It should be measurable less than 1ms for a correct work of Mesa board. For me it’s ~300us.

2. Isolate CPU cores

This is a well known technique. The idea is to isolate some CPU cores from the general scheduler. Then LinuxCNC will use only those isolated cores. Other processes will not be scheduled on them.

To do it need to edit /etc/default/grub and add isolcpus= parameter to GRUB_CMDLINE_LINUX_DEFAULT. For example, to isolate cores 2 and 3:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=2,3"

Then update grub configuration:

sudo update-grub

In some cases rcu_nocbs=2,3 and mitigations=off parameters might help too.

As was mentioned earlier, isolating CPU cores might reduce jitter to 30-50k, i.e. twice.

3. Disable unused hardware

This is a good practice in general. The less hardware you have - the less interrupts you will get. For my installation I disabled Bluetooth and audio/sound.

Bluetooth disabling is easy:

sudo systemctl stop bluetooth.target
sudo systemctl disable bluetooth.target

Sound is more tricky because need to blacklist sound modules. First of all, select all sound modules:

lsmod | grep snd_

Then create sudo vim /etc/modprobe.d/disable-sound.conf file and add there all found modules. In my case these are following:

blacklist snd
blacklist snd_hda_intel
blacklist snd_hda_codec_hdmi
blacklist snd_hda_codec_realtek
blacklist snd_hda_codec_generic
blacklist snd_hda_core
blacklist snd_hwdep
blacklist snd_hda_codec_hdmi
blacklist snd_sof_pci
blacklist snd_sof_xtensa_dsp
blacklist snd_sof
blacklist snd_sof_utils
blacklist snd_soc_hdac_hda
blacklist snd_hda_ext_core
blacklist snd_soc_acpi_intel_match
blacklist snd_soc_acpi
blacklist snd_intel_dspcfg
blacklist snd_intel_sdw_acpi
blacklist snd_hda_codec
blacklist snd_hda_core
blacklist snd_hwdep
blacklist snd_pcm

Feel free to disable anything which you don’t expect to use.

For me, the devices disabling didn’t impact jitter a lot. Maybe 5-10k reduction. But it measurable reduces dispersion of jitter. The histogram becomes more narrow.

4. BIOS tweaks

Power management is a good thing for laptops. But for CNC controller it’s better to have everything running at full speed all the time. So, disable CPU frequency scaling and other power management features.

In my case I disabled anything in Power Management section of BIOS. All fancy Intel technologies like SpeedStep, C-States, TurboBoost, P-states, etc.

Also, virtualization features is impacting the jitter and should be disabled too.

Finally, feel free to disable any devices in BIOS which you don’t expect to use.

I did that changes as a last step and didn’t notice any major jitter changes. But I believe this helps within longer runs.