Veeam Agent for Linux crash with null pointer dereference on 5.9+ kernels

KB ID: 4404
Version: 5.0.2.4567
Published: 2023-01-23
Last Modified: 2023-01-23
Get weekly article updates
By subscribing, you are agreeing to have your personal information managed in accordance with the terms of Veeam's Privacy Notice.

Cheers for trusting us with the spot in your mailbox!

Now you’re less likely to miss what’s been brewing in our knowledge base with this weekly digest

error icon

Oops! Something went wrong.

Please try again later.

Challenge

Symptom

Shortly after Veeam Agent for Linux creates a veeamsnap snapshot, the system hangs or crashes with a null pointer dereference:

Example of dmesg:

BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 8000000003a47067 P4D 8000000003a47067 PUD 3a48067 PMD 0
Oops: 0000 [#1] SMP PTI
CPU: 0 PID: 396645 Comm: VeeamService se Kdump: loaded Tainted: G           OE     5.10.0-17-amd64 #1 Debian 5.10.136-1

 

Associated Environment Specifications

This issue occurs when all 3 of the following are true:

  1. Linux kernel 5.9 or higher is in use.
  2. Veeam Agent for Linux version 5.0.2.4567 is installed and uses the veeamsnap kernel module.
  3. /proc/kallsyms shows all symbol addresses as zeroes:
    $ sudo tail /proc/kallsyms
    0000000000000000 t fuse_open_common [fuse]
    0000000000000000 t fuse_flush_writepages [fuse]
    0000000000000000 t fuse_fill_super_common [fuse]
    0000000000000000 t fuse_simple_background [fuse]
    0000000000000000 t fuse_getxattr [fuse]
    0000000000000000 r fuse_dentry_operations [fuse]
    0000000000000000 t fuse_free_conn [fuse]
    0000000000000000 t fuse_write_update_size [fuse]
    0000000000000000 t fuse_sync_release [fuse]
    0000000000000000 t fuse_direct_io [fuse]
    

 

Cause

On kernels newer than 5.9, Veeam Agent for Linux 5.0.2.4567 has to retrieve symbols usually exposed through /proc/kallsyms. However, several configuration options may restrict this file, causing zero values to be returned instead of valid addresses. A null pointer dereference occurs when Veeam Agent for Linux attempts to use those zero addresses.

Solution

To resolve this, ensure that /proc/kallsyms returns non-zero addresses.

One of the most common ways those values become restricted is through the use of the following sysctl parameters:

  • kernel.perf_event_paranoid
  • kernel.kptr_restrict
sudo sysctl kernel.perf_event_paranoid
sudo sysctl kernel.kptr_restrict 

To avoid this issue, both parameters must be set to either:

  • 0 — effectively exposing symbol information to all users

    or 
  • 1 — exposing addresses to users with CAP_SYSLOG capability (i.e., root, which usually executes Veeam Agent for Linux service process)

If the values are higher than 1, they must be set to either 0 or 1.

sudo sysctl -w kernel.perf_event_paranoid=1 
sudo sysctl -w kernel.kptr_restrict=1 

After setting those values, recheck the /proc/kallsyms contents and verify that non-zero addresses are returned:

$ sudo tail /proc/kallsyms
ffffffffc01391a0 t fuse_open_common [fuse]
ffffffffc013bbd0 t fuse_flush_writepages [fuse]
ffffffffc013e930 t fuse_fill_super_common [fuse]
ffffffffc012f9f0 t fuse_simple_background [fuse]
ffffffffc013fab0 t fuse_getxattr [fuse]
ffffffffc01428c0 r fuse_dentry_operations [fuse]
ffffffffc013cfb0 t fuse_free_conn [fuse]
ffffffffc013a360 t fuse_write_update_size [fuse]
ffffffffc0137280 t fuse_sync_release [fuse]
ffffffffc01397f0 t fuse_direct_io [fuse]
To make these changes persist after a reboot, you must adjust these values in the sysctl configuration files. For more information, refer to the sysctl manpage.
To submit feedback regarding this article, please click this link: Send Article Feedback
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.

Spelling error in text

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Thank you!

Thank you!

Your feedback has been received and will be reviewed.

Oops! Something went wrong.

Please try again later.

KB Feedback/Suggestion

This form is only for KB Feedback/Suggestions, if you need help with the software open a support case

By submitting, you are agreeing to have your personal information managed in accordance with the terms of Veeam's Privacy Notice.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Verify your email to continue your product download
We've sent a verification code to:
  • Incorrect verification code. Please try again.
An email with a verification code was just sent to
Didn't receive the code? Click to resend in sec
Didn't receive the code? Click to resend
Thank you!

Thank you!

Your feedback has been received and will be reviewed.

error icon

Oops! Something went wrong.

Please try again later.