[PR #6214] [MERGED] [client] Prevent corruption from competing log rotation and improve debug bundle #29174

Closed
opened 2026-08-05 08:07:34 -04:00 by saavagebueno · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/netbirdio/netbird/pull/6214
Author: @theodorsm
Created: 5/19/2026
Status: Merged
Merged: 6/4/2026
Merged by: @theodorsm

Base: mainHead: fix/bundle-logrotate


📝 Commits (10+)

  • 16d1a4d Add daemon version to bundle
  • 1e66db8 Add logrotate conflict detection
  • 8c50979 Ensure that log directory exists
  • 5a3301b Fix comment
  • 702552e Add env flag and refactor
  • 97b8c53 Add uncompressed logrotate files to bundle
  • 471e2f9 Refactor log disable check
  • b9a7375 Remove directory creation
  • bcda5ed Fix log file open and bundle uncompressed files
  • a5a0bf6 Fix daemon and cli version in bundle and status

📊 Changes

18 files changed (+513 additions, -32 deletions)

View changed files

📝 client/cmd/debug.go (+4 -0)
📝 client/cmd/service_controller.go (+14 -8)
📝 client/internal/debug/debug.go (+18 -4)
client/internal/debug/debug_logfiles_test.go (+103 -0)
📝 client/internal/engine.go (+2 -0)
📝 client/proto/daemon.pb.go (+13 -2)
📝 client/proto/daemon.proto (+1 -0)
📝 client/proto/generate.sh (+1 -1)
📝 client/server/debug.go (+3 -0)
📝 client/status/status.go (+12 -2)
📝 client/ui/debug.go (+3 -0)
📝 go.mod (+1 -1)
📝 go.sum (+2 -2)
📝 util/log.go (+42 -12)
util/log_test.go (+96 -0)
util/logrotate_linux.go (+93 -0)
util/logrotate_linux_test.go (+95 -0)
util/logrotate_nonlinux.go (+10 -0)

📄 Description

Describe your changes

We have received corrupted log files in debug bundles recently. This PR contains the following fixes:

  • Adds heuristic to detect an edge case on Linux where a system has configured logrotate as a separate service to rotate log files which would mangle our client log files. If we detect logrotate being configured for netbird, we disable our rotation.
  • Adds new env var to disable log rotation: NB_LOG_DISABLE_ROTATION
  • Adds compressed and plain logrotate files to debug bundle.
  • Replaces lumberjack with timberjack (maintained fork with bug fixes and extra features).
  • Clarifies which daemon version is running in the bundle stats.
  • Change logging for client service status to console (currently lumberjack silently fails to open log file as it is with restrictive permissions).

Running logrotate before

Logrotate is configured for the system:

root@kubuntu:/var/log/netbird# cat /etc/logrotate.d/tmp
/var/log/netbird/*.log {
daily
rotate 30
compress
delaycompress
missingok
notifempty
copytruncate
}

After running logrotate manually, client.log log is corrupted:

root@kubuntu:/var/log/netbird# file client.log                                
client.log: data                                                              
root@kubuntu:/var/log/netbird# file client.log.1                              
client.log.1: ASCII text                                                      
root@kubuntu:/var/log/netbird# ls                                    
client.log    client.log.1  netbird.err   netbird.out                         
root@kubuntu:/var/log/netbird# hexdump -C client.log                          
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*                                                                             
00000780  00 00 00 00 00 32 30 32  36 2d 30 35 2d 32 30 54  |.....2026-05-20T|
00000790  31 34 3a 31 39 3a 30 33  2e 35 36 33 2b 30 32 3a  |14:19:03.563+02:|
000007a0  30 30 20 49 4e 46 4f 20  63 6c 69 65 6e 74 2f 73  |00 INFO client/s|
000007b0  65 72 76 65 72 2f 73 65  72 76 65 72 2e 67 6f 3a  |erver/server.go:|
000007c0  34 38 32 3a 20 61 63 74  69 76 65 20 70 72 6f 66  |482: active prof|
000007d0  69 6c 65 3a 20 64 65 66  61 75 6c 74 20 66 6f 72  |ile: default for|  

Running logrotate after this PR

user@kubuntu:~$ sudo ./netbird service start
WARN[0000] log rotation conflict detected in: "/etc/logrotate.d/tmp", rotation is disabled 
NetBird service has been started

After running logrotate manually, client.log log is not corrupted:

root@kubuntu:/var/log/netbird# ls
client.log  client.log.1  netbird.err  netbird.out
root@kubuntu:/var/log/netbird# file client.log
client.log: ASCII text
root@kubuntu:/var/log/netbird# file client.log.1
client.log.1: ASCII text
root@kubuntu:/var/log/netbird#  hexdump -C client.log
00000000  32 30 32 36 2d 30 35 2d  32 30 54 31 35 3a 31 30  |2026-05-20T15:10|
00000010  3a 33 36 2e 31 39 37 2b  30 32 3a 30 30 20 49 4e  |:36.197+02:00 IN|
00000020  46 4f 20 63 6c 69 65 6e  74 2f 69 6e 74 65 72 6e  |FO client/intern|
00000030  61 6c 2f 65 6e 67 69 6e  65 2e 67 6f 3a 33 31 37  |al/engine.go:317|
00000040  3a 20 4e 65 74 77 6f 72  6b 20 6d 6f 6e 69 74 6f  |: Network monito|

Logs that are rotated by logrotate is also included in the bundle:

root@kubuntu:/tmp# unzip -v netbird.debug.601212628.zip
Archive:  netbird.debug.601212628.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
<...>
   17895  Defl:N     2978  83% 2026-05-20 15:21 93f3c6e7  client.log
    5362  Defl:N     1643  69% 2026-05-20 15:10 0128eb6f  client.log.1
     135  Defl:N      119  12% 2026-05-20 15:20 c6974df3  netbird.err
       0  Defl:N        5   0% 2026-05-20 15:09 00000000  netbird.out
--------          -------  ---                            -------
  181750            61991  66%                            27 files

Stack

Checklist

  • Is it a bug fix
  • Is a typo/documentation fix
  • Is a feature enhancement
  • It is a refactor
  • Created tests that fail without the change (if possible)

By submitting this pull request, you confirm that you have read and agree to the terms of the Contributor License Agreement.

Documentation

Select exactly one:

  • I added/updated documentation for this change
  • Documentation is not needed for this change (explain why)

Docs PR URL (required if "docs added" is checked)

logrotate config conflicts and troubleshooting are mentioned in the docs:
https://github.com/netbirdio/docs/pull/763

Summary by CodeRabbit

  • New Features

    • Debug bundles now include daemon and CLI version metadata for clearer diagnostics.
    • CLI debug commands and UI flows attach client version when generating bundles.
  • Bug Fixes

    • Rotated-log collection now detects and includes both gzipped and plain rotated logs.
    • Log rotation can be disabled when a system logrotate conflict is detected to prevent failures.
  • Tests

    • Added tests validating rotated-log bundling and log rotation behavior and conflict detection.

Review Change Stack


🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/netbirdio/netbird/pull/6214 **Author:** [@theodorsm](https://github.com/theodorsm) **Created:** 5/19/2026 **Status:** ✅ Merged **Merged:** 6/4/2026 **Merged by:** [@theodorsm](https://github.com/theodorsm) **Base:** `main` ← **Head:** `fix/bundle-logrotate` --- ### 📝 Commits (10+) - [`16d1a4d`](https://github.com/netbirdio/netbird/commit/16d1a4d5505c97af34507441ff717c4585f4d141) Add daemon version to bundle - [`1e66db8`](https://github.com/netbirdio/netbird/commit/1e66db8ddb6a449dd7fbc5a0b849ada5c62c2957) Add logrotate conflict detection - [`8c50979`](https://github.com/netbirdio/netbird/commit/8c509794688173149e4c353b19f1707d206df62b) Ensure that log directory exists - [`5a3301b`](https://github.com/netbirdio/netbird/commit/5a3301b3c715709b7b19f3033a4a08a89311703e) Fix comment - [`702552e`](https://github.com/netbirdio/netbird/commit/702552e9ddbbf0f0b4693f50f12d2bf3bf2fa71f) Add env flag and refactor - [`97b8c53`](https://github.com/netbirdio/netbird/commit/97b8c53dff93d3ef322017ab76755d2513c46c95) Add uncompressed logrotate files to bundle - [`471e2f9`](https://github.com/netbirdio/netbird/commit/471e2f98d764da3da7f23fc9ed7e61bddcea045f) Refactor log disable check - [`b9a7375`](https://github.com/netbirdio/netbird/commit/b9a7375f646d0e3ea0f10ee90f02a551425c6c07) Remove directory creation - [`bcda5ed`](https://github.com/netbirdio/netbird/commit/bcda5eddbbe164a15daf34f5f8bc79a6b0a84f0a) Fix log file open and bundle uncompressed files - [`a5a0bf6`](https://github.com/netbirdio/netbird/commit/a5a0bf6ff4733c0c6de4ef3032229dbde0d83703) Fix daemon and cli version in bundle and status ### 📊 Changes **18 files changed** (+513 additions, -32 deletions) <details> <summary>View changed files</summary> 📝 `client/cmd/debug.go` (+4 -0) 📝 `client/cmd/service_controller.go` (+14 -8) 📝 `client/internal/debug/debug.go` (+18 -4) ➕ `client/internal/debug/debug_logfiles_test.go` (+103 -0) 📝 `client/internal/engine.go` (+2 -0) 📝 `client/proto/daemon.pb.go` (+13 -2) 📝 `client/proto/daemon.proto` (+1 -0) 📝 `client/proto/generate.sh` (+1 -1) 📝 `client/server/debug.go` (+3 -0) 📝 `client/status/status.go` (+12 -2) 📝 `client/ui/debug.go` (+3 -0) 📝 `go.mod` (+1 -1) 📝 `go.sum` (+2 -2) 📝 `util/log.go` (+42 -12) ➕ `util/log_test.go` (+96 -0) ➕ `util/logrotate_linux.go` (+93 -0) ➕ `util/logrotate_linux_test.go` (+95 -0) ➕ `util/logrotate_nonlinux.go` (+10 -0) </details> ### 📄 Description ## Describe your changes We have received corrupted log files in debug bundles recently. This PR contains the following fixes: - Adds heuristic to detect an edge case on Linux where a system has configured logrotate as a separate service to rotate log files which would mangle our client log files. If we detect logrotate being configured for netbird, we disable our rotation. - Adds new env var to disable log rotation: `NB_LOG_DISABLE_ROTATION` - Adds compressed and plain logrotate files to debug bundle. - Replaces lumberjack with [timberjack](https://github.com/DeRuina/timberjack) (maintained fork with bug fixes and extra features). - Clarifies which daemon version is running in the bundle stats. - Change logging for client service status to console (currently lumberjack silently fails to open log file as it is with restrictive permissions). ### Running logrotate before Logrotate is configured for the system: ``` root@kubuntu:/var/log/netbird# cat /etc/logrotate.d/tmp /var/log/netbird/*.log { daily rotate 30 compress delaycompress missingok notifempty copytruncate } ``` After running logrotate manually, `client.log` log is corrupted: ``` root@kubuntu:/var/log/netbird# file client.log client.log: data root@kubuntu:/var/log/netbird# file client.log.1 client.log.1: ASCII text root@kubuntu:/var/log/netbird# ls client.log client.log.1 netbird.err netbird.out root@kubuntu:/var/log/netbird# hexdump -C client.log 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000780 00 00 00 00 00 32 30 32 36 2d 30 35 2d 32 30 54 |.....2026-05-20T| 00000790 31 34 3a 31 39 3a 30 33 2e 35 36 33 2b 30 32 3a |14:19:03.563+02:| 000007a0 30 30 20 49 4e 46 4f 20 63 6c 69 65 6e 74 2f 73 |00 INFO client/s| 000007b0 65 72 76 65 72 2f 73 65 72 76 65 72 2e 67 6f 3a |erver/server.go:| 000007c0 34 38 32 3a 20 61 63 74 69 76 65 20 70 72 6f 66 |482: active prof| 000007d0 69 6c 65 3a 20 64 65 66 61 75 6c 74 20 66 6f 72 |ile: default for| ``` ### Running logrotate after this PR ``` user@kubuntu:~$ sudo ./netbird service start WARN[0000] log rotation conflict detected in: "/etc/logrotate.d/tmp", rotation is disabled NetBird service has been started ``` After running logrotate manually, `client.log` log is not corrupted: ``` root@kubuntu:/var/log/netbird# ls client.log client.log.1 netbird.err netbird.out root@kubuntu:/var/log/netbird# file client.log client.log: ASCII text root@kubuntu:/var/log/netbird# file client.log.1 client.log.1: ASCII text root@kubuntu:/var/log/netbird# hexdump -C client.log 00000000 32 30 32 36 2d 30 35 2d 32 30 54 31 35 3a 31 30 |2026-05-20T15:10| 00000010 3a 33 36 2e 31 39 37 2b 30 32 3a 30 30 20 49 4e |:36.197+02:00 IN| 00000020 46 4f 20 63 6c 69 65 6e 74 2f 69 6e 74 65 72 6e |FO client/intern| 00000030 61 6c 2f 65 6e 67 69 6e 65 2e 67 6f 3a 33 31 37 |al/engine.go:317| 00000040 3a 20 4e 65 74 77 6f 72 6b 20 6d 6f 6e 69 74 6f |: Network monito| ``` Logs that are rotated by logrotate is also included in the bundle: ``` root@kubuntu:/tmp# unzip -v netbird.debug.601212628.zip Archive: netbird.debug.601212628.zip Length Method Size Cmpr Date Time CRC-32 Name -------- ------ ------- ---- ---------- ----- -------- ---- <...> 17895 Defl:N 2978 83% 2026-05-20 15:21 93f3c6e7 client.log 5362 Defl:N 1643 69% 2026-05-20 15:10 0128eb6f client.log.1 135 Defl:N 119 12% 2026-05-20 15:20 c6974df3 netbird.err 0 Defl:N 5 0% 2026-05-20 15:09 00000000 netbird.out -------- ------- --- ------- 181750 61991 66% 27 files ``` ## Issue ticket number and link ## Stack <!-- branch-stack --> ### Checklist - [x] Is it a bug fix - [ ] Is a typo/documentation fix - [ ] Is a feature enhancement - [ ] It is a refactor - [ ] Created tests that fail without the change (if possible) > By submitting this pull request, you confirm that you have read and agree to the terms of the [Contributor License Agreement](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT.md). ## Documentation Select exactly one: - [x] I added/updated documentation for this change - [ ] Documentation is **not needed** for this change (explain why) ### Docs PR URL (required if "docs added" is checked) `logrotate` config conflicts and troubleshooting are mentioned in the docs: https://github.com/netbirdio/docs/pull/763 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Debug bundles now include daemon and CLI version metadata for clearer diagnostics. * CLI debug commands and UI flows attach client version when generating bundles. * **Bug Fixes** * Rotated-log collection now detects and includes both gzipped and plain rotated logs. * Log rotation can be disabled when a system logrotate conflict is detected to prevent failures. * **Tests** * Added tests validating rotated-log bundling and log rotation behavior and conflict detection. <!-- review_stack_entry_start --> [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/netbirdio/netbird/pull/6214?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
saavagebueno added the pull-request label 2026-08-05 08:07:34 -04:00
Sign in to join this conversation.
No Label pull-request
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#29174