Debug an agent
Enable debug logging on the Comm, Remote, and Health agents and on the User Process, reproduce a problem, collect the log files, and turn debugging off again.
Debug an agent
This guide turns on debug logging for the NetLock RMM agent services on a device, walks through reproducing a problem while logs are being written, and shows where to collect the log files from. By default the agents write no log files at all, not even errors, so an empty or missing Logs folder on a device is normal, not a fault. Debug logging is the on-device counterpart to the server-side Logging page described in A.9.
Each of the three agent services (Comm, Remote, and Health) checks once at process startup whether a file named debug.txt exists in its data directory. The file's content is ignored; an empty file is enough. Because the check runs only at startup, creating or deleting debug.txt takes effect only after the service restarts.
Before you start
- You have administrator or root access on the target device. The agent data directories are protected on every supported OS, so both creating
debug.txtand restarting the services require elevation. - You know which agent you need to debug.
Comm Agenthandles the server connection, policies, jobs, and sensors;Remote Agenthandles remote control sessions;Health Agentsupervises the other two. When in doubt, enable debugging on all three. - Plan to turn debugging off again. Debug logs grow without rotation or a size limit, so do not leave
debug.txtin place on a production device after you have collected what you need.
The debug.txt file goes into the agent's data directory. The directory names contain spaces on every OS, so always quote the paths in a shell:
- Windows:
C:\ProgramData\0x101 Cyber Security\NetLock RMM\<Agent>\debug.txt - Linux:
/var/0x101 Cyber Security/NetLock RMM/<Agent>/debug.txt - macOS:
/Library/Application Support/0x101 Cyber Security/NetLock RMM/<Agent>/debug.txt
where <Agent> is Comm Agent, Remote Agent, or Health Agent.
Note: The Health agent checks the other services every 60 seconds and automatically starts a stopped Comm or Remote service. Always use a single
restartcommand rather than stopping a service, waiting, and starting it again. Otherwise the Health agent may start the service before you do.
Steps
- Create an empty
debug.txtin the data directory of each agent you want to debug.
Windows (PowerShell, elevated):
New-Item -ItemType File -Path "C:\ProgramData\0x101 Cyber Security\NetLock RMM\Comm Agent\debug.txt"
New-Item -ItemType File -Path "C:\ProgramData\0x101 Cyber Security\NetLock RMM\Remote Agent\debug.txt"
New-Item -ItemType File -Path "C:\ProgramData\0x101 Cyber Security\NetLock RMM\Health Agent\debug.txt"Linux (Bash, with sudo):
sudo touch "/var/0x101 Cyber Security/NetLock RMM/Comm Agent/debug.txt"
sudo touch "/var/0x101 Cyber Security/NetLock RMM/Remote Agent/debug.txt"
sudo touch "/var/0x101 Cyber Security/NetLock RMM/Health Agent/debug.txt"macOS (Terminal, with sudo):
sudo touch "/Library/Application Support/0x101 Cyber Security/NetLock RMM/Comm Agent/debug.txt"
sudo touch "/Library/Application Support/0x101 Cyber Security/NetLock RMM/Remote Agent/debug.txt"
sudo touch "/Library/Application Support/0x101 Cyber Security/NetLock RMM/Health Agent/debug.txt"- Restart the agent services so they pick up the file. When you enabled debugging on more than one agent, restart all three.
Windows (PowerShell, elevated):
Restart-Service NetLock_RMM_Agent_Comm
Restart-Service NetLock_RMM_Agent_Remote
Restart-Service NetLock_RMM_Agent_HealthLinux (Bash, with sudo):
sudo systemctl restart netlock-rmm-agent-comm
sudo systemctl restart netlock-rmm-agent-remote
sudo systemctl restart netlock-rmm-agent-healthmacOS (Terminal, with sudo):
sudo launchctl kickstart -k system/com.netlock.rmm.agent.comm
sudo launchctl kickstart -k system/com.netlock.rmm.agent.remote
sudo launchctl kickstart -k system/com.netlock.rmm.agent.health-
Reproduce the problem. Trigger the failing action again (run the job, start the remote session, wait for the sync cycle) so the agents write log lines while the problem happens.
-
Collect the logs. Each agent writes JSON-lines
.txtfiles into aLogssubfolder next to itsdebug.txt(for exampleC:\ProgramData\0x101 Cyber Security\NetLock RMM\Comm Agent\Logs\). The key files areDebug.txtandError.txt, plus topic files such asJobs.txt,Sensors.txt, andPowerShell.txton the Comm agent orRemote_Control.txton the Remote agent. Copy or zip the entireLogsfolder per agent rather than picking individual files.
Folder permissions: the working folders of the agents (
Scripts,Jobs,Sensors,Temp,Toolsbelow each agent, andInstaller,Updates,TempbelowNetLock RMM) are accessible to the system account and administrators only. Open them from an elevated console (dir,icacls) rather than through the Explorer prompt "You don't currently have permission to access this folder — Continue": that prompt adds your account to the folder permissions, and the agent resets them on its next start. TheLogsfolders anddebug.txtstay readable for every user.The scripts that ship with the agent live under
C:\Program Files\0x101 Cyber Security\NetLock RMM\Comm Agent\Scripts(system account and administrators only). The agent refuses a changed script with an Endpoint protection event (type 14, reasonscript_manifest_mismatch) and a missing one withscript_missing. Do not edit these files; run the installer withfixto restore them.
Tip: Independent of debug mode, the services' standard output is available too. On Linux, use journald:
journalctl -u netlock-rmm-agent-comm -n 300 --no-pager(likewise for theremoteandhealthunits). Errors that stop a service right after its start, before it writes any log file, appear only there. On macOS, the LaunchDaemons log to/var/log/netlock_rmm_agent.comm.logand/var/log/netlock_rmm_agent.comm_error.log(likewise.remote/.health).
- Disable debugging again. Delete
debug.txtfrom each data directory and restart the services with the same commands as in step 2. Debug logs grow unbounded, so do not skip this on a production device. You can delete the collectedLogsfolders afterwards to reclaim disk space.
Verify it worked
- After step 2, the
Logssubfolder appears in each agent's data directory andDebug.txtinside it grows as the agent works. The Comm agent writes lines on every sync cycle, so within a minute or two the file should have content. - After step 5, the services run without writing new lines, and the timestamps on the files in
Logsstop advancing.
Debug the User Process
Screen capture, keyboard and mouse input and the clipboard of a remote control session are not handled by one of the three services but by the User Process, a program of the agent that runs inside the desktop session of the signed-in user. When a remote control session connects but shows no picture or takes no input, its log is the one to read, next to the Remote agent's. This section covers Windows, Linux and macOS.
Debugging is switched on with a debug.txt in the data directory of the User Process:
- Windows:
C:\ProgramData\0x101 Cyber Security\NetLock RMM\User Process\debug.txt - Linux:
/var/0x101 Cyber Security/NetLock RMM/User Process/debug.txt - macOS:
/Library/Application Support/0x101 Cyber Security/NetLock RMM/User Process/debug.txt
Nothing has to be restarted: the User Process looks for the file every time it is about to write a log line, so logging starts as soon as the file exists and stops as soon as it is deleted.
Where the log is written depends on the platform:
- Windows: the
Logsfolder next todebug.txt(Debug.txtandError.txt). - Linux (agent 3.3.0.11 or later): below the home directory of the desktop user, because the User Process runs as that user:
~/.config/netlock/logs/(Debug.txtandError.txt), next to the stored screen-sharing permissionpipewire_restore_token. From a root shell that is/home/<user>/.config/netlock/logs/. The folder and its files are readable for that user and root only. - macOS (agent 3.3.0.11 or later): below the home directory of the signed-in user, because the User Process runs as that user:
~/Library/Logs/NetLock RMM/User Process/(Debug.txtandError.txt). From a root shell that is/Users/<user>/Library/Logs/NetLock RMM/User Process/. The folder and its files are readable for that user and root only. The log records the permission state at every capture start, the start and the end of each ScreenCaptureKit stream with its reason, the fall-back to single pictures and the kind of each input error; never what was typed or clicked.
Windows (PowerShell, elevated):
New-Item -ItemType File -Path "C:\ProgramData\0x101 Cyber Security\NetLock RMM\User Process\debug.txt"Linux (Bash, with sudo):
sudo touch "/var/0x101 Cyber Security/NetLock RMM/User Process/debug.txt"macOS (Terminal, with sudo):
sudo touch "/Library/Application Support/0x101 Cyber Security/NetLock RMM/User Process/debug.txt"On macOS the log of a signed-in user is in /Users/<user>/Library/Logs/NetLock RMM/User Process/; remove the marker with sudo rm on the same path when you are done.
Reproduce the problem (on Linux, start a remote control connection from the console) and collect the log. On Linux, replace <user> with the name of the desktop user:
sudo ls -la /home/<user>/.config/netlock/logs/Switch debugging off again when you are done; the log grows without a limit. On Linux:
sudo rm "/var/0x101 Cyber Security/NetLock RMM/User Process/debug.txt"On macOS the User Process also has a diagnosis mode that reports the permission state, the signature of the program, the displays and the result of a short capture without a session from the console; it is described under Remote Screen Control on macOS. The signature and notarization of the installed programs can be checked on the Mac with codesign --verify --deep --strict and spctl -a -t exec on the application bundles under /usr/local/bin/0x101_Cyber_Security/NetLock_RMM/, and the LaunchAgents of a signed-in user with launchctl print gui/$(id -u)/com.netlock.rmm.user.process.
The log of a Linux device covers the whole start of the screen capture on Wayland: the availability check with the name of what failed (for example gstreamer_element_missing:jpegenc, python_modules_missing, pipewire_not_running or portal_not_available), each request to the desktop portal with its answer and the time it took, whether a stored screen-sharing permission was used, the GStreamer pipeline with its error output, the first picture, and the reason whenever the capture is stopped. It holds neither what was typed or clicked during a session nor the stored permission token. The Remote agent's Debug.txt records the capture results the User Process reports to it. The checks that go with it are listed under Troubleshooting Remote Screen Control on Linux.
Troubleshooting
- No log output appears even though
debug.txtexists. The services were not restarted after the file was created, and the check runs only at process startup. Restart all three services as in step 2. On Windows, also check the file name: with hidden file extensions enabled, Explorer'sNew → Text Documentcreatesdebug.txt.txtwithout showing the extension, and the agent ignores that file. Create the file from PowerShell as shown above to be safe. - The Comm agent service on Linux restarts every few seconds and the device never comes online. Read the journal first:
sudo journalctl -u netlock-rmm-agent-comm -n 300 --no-pager. The agent checks at startup that its SQLite library loads; if it does not, the journal andError.txtcontain one line starting withNetLock RMM Comm Agent: the SQLite library could not be loaded, followed by the loader error, the operating system and the glibc version. Agents before 3.3.0.11 do not run on systems with a glibc older than 2.34 (Ubuntu 20.04, Debian 11, RHEL/Oracle/Rocky/AlmaLinux 8). Those devices do not update themselves; install the agent again with an installer from a server on 3.3.0.11 or later. See X.8 — Platform Support for the supported distributions. - A Linux device with SELinux in enforcing mode (Fedora, RHEL and its rebuilds) stays on an agent before 3.3.0.11 and no longer syncs.
sudo systemctl status netlock-rmm-agent-installershowsstatus=203/EXEC.sudo ausearch -m avc -ts todayshows a deniedexecuteonNetLock_RMM_Agent_Installer. Until the reinstall, the device does not sync its policy or report patch results. These devices do not update themselves. Install the agent again with an installer from a server on 3.3.0.11 or later, the same way as a first installation (see Deploy your first agent): the installer with its embedded configuration or the--serverparameter mode. This runs acleaninstallation, which removes the old agent and its local data first, and the device then waits for approval underUnauthorized Devices(see Approve and label a new device). A failed start of the installer is written to the agent'sError.txt. - The agent is offline or cannot reach the server. Debug logging still helps. The logs are written locally on the device, independent of the server connection, so connection problems are exactly what debug logging is for. Collect the
Logsfolders directly from the device (step 4) instead of expecting anything to show up in the Console. - The device is behind an HTTP proxy: which proxy did the agent pick? With debug logging on, the Comm, Remote and Health agents write the decision to
Debug.txtunderHelper.Proxy_Resolver.Create_Proxyon every start: the mode (auto,manual,none), the proxy address, the authentication method and, if one is used, the user name. The password is never written. No such line means no proxy applies and the agent connects directly, which inautomode is also what you see on a device that reaches the backend without one. The reachability check itself logs underHelper.Connectivity_Probe, including a proxy that answered407. The settings the agent reads come fromproxy.jsonnext toserver_config.json; see Devices behind an HTTP proxy. - A device turns up under
Unauthorized Devicesalthough nobody deauthorized it. The on-device logs cannot show the reason, because that decision is made on the server. Read the device's authentication log in the Console first, as described in the next section. A second entry with the name of a device that is still authorized is a reinstalled operating system; the entry carries the chip "Name already exists" and is continued with "Adopt as reinstall" rather than approved as a duplicate.
When a device becomes unauthorized: read the Authentication tab first
A device that suddenly waits under Devices → Unauthorized Devices although nobody deauthorized it is the one agent problem the on-device logs cannot explain, because the decision is made on the server. The server writes every agent check-in that it did not accept as a plain success to a per-device authentication log, and the Console shows it in two places:
Devices→ select the device → tab Authentication, for a device that is (again) authorized.Devices → Unauthorized Devices→ row action Authentication log, for a device that just lost its authorization.
Both require the permission "Authentication log" in the device information tabs group of the user settings.
Each row names the endpoint the request hit, the remote address, the agent version, the hardware ID the agent sent next to the one that was stored for the device, and a reason. Repeats of the same state within ten minutes are folded into one row whose counter grows, so a device that polls every 30 seconds while it waits for approval produces one row per ten minutes rather than twenty. The moment authentication succeeds again is recorded as "Authentication succeeded again". Successful check-ins of an authorized device are not listed, so an empty tab means the device has authenticated without trouble.
| Reason | What happened | What to do |
|---|---|---|
| Hardware ID changed: new ID stored, device stays authorized | The access key matched, but the hardware ID the agent sent differed from the stored one. The server stored the new ID, kept the previous one with the device and left the authorization as it was. The stored ID in the row is the previous value, the received ID the new one. The server sensor trigger "Hardware ID changed" fires for this row. | Nothing, unless you did not expect it. A new ID means the operating system was reinstalled or the device is a copy of an image that was not generalized (see Hardware ID). Two devices sharing one access key (a cloned image) show alternating IDs from two remote addresses; give each clone its own agent installation. |
| Hardware ID changed again while waiting for approval | The same for a device that was not authorized at that moment: the new ID is stored and the device keeps waiting. The auto-authorization window of its agent package is checked afterwards. | Approve the device; it stays stable. |
| Hardware ID migrated to the new format | The agent update moved the device to the current hardware ID format. The authorization is unchanged. The stored ID in the row is the previous value, the received ID the new one. | Nothing. A device that was already waiting for approval still needs that approval. |
| Reinstall detected: new access key, waiting for approval | The agent was reinstalled on a known device whose previous installation no longer reports. The device waits for approval under its new access key; the details name the prefix of the previous key. If the previous installation still reports, a new entry is created instead, with fallback_skipped=online in its details. | Expected after a reinstall with a fresh installer. Approve the device. |
| New device created, waiting for approval | Nothing matched, so a new device row was created. Details tenant_id=0 mean the tenant or location GUID of the installer did not resolve; the device is then in no tenant view. Details duplicate_of=<id> name an authorized device of the same name (duplicate_same_tenant=1 when it is in the same tenant): the new row is most likely that device after a reinstall of its operating system. Details fallback_skipped=online mean a device of the same name and hardware ID exists but its agent still reports, so it was not re-keyed. | Approve the device. With tenant_id=0, check the installer package. With duplicate_of, the Unauthorized devices page shows the chip "Name already exists" on the entry; use Adopt as reinstall there instead of approving a duplicate. |
| Adopted as reinstall of an existing device | An administrator used "Adopt as reinstall" on the Unauthorized devices page: the existing device took over the access key and hardware ID of the unauthorized entry, the entry was removed and its rows of this log were moved to the device. The stored ID in the row is the device's previous hardware ID, the received ID the new one; the details name the removed entry and the prefix of the device's previous key. | Nothing. The device fetches its policy at its next check-in. |
| Auto-authorized by its agent package / New device auto-authorized by its agent package | The agent package has an auto-authorization window that was still open. | Nothing. |
| Auto-authorization failed | The window was open, but the authorization could not be written. | Check the server error log; approve the device manually. |
| Not authorized, waiting for approval | The device polls while it waits for approval. | Approve the device. |
| Identity incomplete: hardware ID or device name missing / Identity rejected | The request lacked a hardware ID or a device name, or the stored authorization state was unreadable. | Usually a hand-made or truncated request. A file browser download with an incomplete identity ends here as well. |
| Server error while checking the identity | The check failed with an exception; the details carry the message. The agent treats this answer as a no-op and retries. | Check the server error log and the database connection. |
| Hub connection rejected: … | The Remote agent's connection to the command hub was refused: unknown agent package, device not authorized, identity incomplete (no hardware ID), unknown access key or server error. | An incomplete identity here while the HTTP check-ins pass means the Remote agent carries an older identity file than the Comm agent; restart the Remote agent service. |
| Deauthorized by the license limit | The number of authorized devices exceeded the license, and the newest devices were deauthorized. The details name the limit and the count at that moment. | Raise the license or reduce the number of authorized devices, then re-authorize. |
The log is written by the server, so agents need no update for it. Rows are kept for 90 days by default (Settings → Retention & cleanup). Update the Web Console before the server.
Hardware ID
Next to the access key, every device identity carries a hardware ID. The server stores it with the device, shows it in the authentication log and in the Authentication tab, and uses it to recognize a reinstalled agent; it is also the value behind {{device.hwid}} in scripts. The access key remains the identifier to use when you refer to a device from outside. A changed hardware ID does not take the authorization away: the server records the new value and the device stays authorized (see the table above).
The hardware ID identifies the operating system installation, not the hardware:
- A reinstalled operating system (same host name, fresh agent) produces a new hardware ID and, with a fresh installer, a new access key. The agent appears as a new entry under
Unauthorized Devices. The entry carries the chip Name already exists when an authorized device of the same name exists in your tenants; use the row action Adopt as reinstall to continue that device under the new agent (it keeps tenant, location, group, policy, label and history; the new entry is removed) instead of approving a duplicate. See Chapter 3.2. - An agent that is uninstalled and installed again on the same operating system installation keeps its hardware ID, and the server continues the existing device ("Reinstall detected") once the previous installation no longer reports. A reinstall while it still reports appears as a new entry with the chip, to be adopted.
- A replaced mainboard, a VM moved to another host or restored from a backup do not change the ID.
- Clones of an image that was not generalized share the hardware ID. Clones that also carry the agent's
server_config.jsonshare the access key and merge into one device row; clones with their own agent installation but the same ID can take over each other's row when one of them is reinstalled. Runsysprep /generalize(Windows) or clear/etc/machine-id(Linux) before sealing an image, and install the agent after provisioning.
Related
- Guide H.1 — Deploy your first agent — install the agent and its three services.
- Uninstall an agent — remove the agent, including its data directories and logs.
- X.3 — Troubleshooting — symptom-cause-fix entries for the most common agent problems.
- X.8.5 — Remote Screen Control prerequisites on Linux — packages, the first connection and the checks for a Linux device without a picture.
- A.9 — Logging & protocols — the server-side
Loggingpage, the counterpart to on-device debug logs.