Debug an agent

This guide turns on debug logging for the NetLock RMM agent services on a device, walks through reproducing a problem while logs are being written, and shows where to collect the log files from. By default the agents write no log files at all — not even errors — so an empty or missing Logs folder on a device is normal, not a fault. Debug logging is the on-device counterpart to the server-side Logging page described in A.9.

Each of the three agent services — Comm, Remote, and Health — checks once at process startup whether a file named debug.txt exists in its data directory. The file's content is ignored; an empty file is enough. Because the check runs only at startup, creating or deleting debug.txt takes effect only after the service restarts.

Before you start

• You have administrator or root access on the target device. The agent data directories are protected on every supported OS, so both creating debug.txt and restarting the services require elevation.
• You know which agent you need to debug. Comm Agent handles the server connection, policies, jobs, and sensors; Remote Agent handles remote control sessions; Health Agent supervises the other two. When in doubt, enable debugging on all three.
• Plan to turn debugging off again. Debug logs grow without rotation or a size limit — do not leave debug.txt in place on a production device after you have collected what you need.

The debug.txt file goes into the agent's data directory. The directory names contain spaces on every OS, so always quote the paths in a shell:

• Windows — C:\ProgramData\0x101 Cyber Security\NetLock RMM\<Agent>\debug.txt
• Linux — /var/0x101 Cyber Security/NetLock RMM/<Agent>/debug.txt
• macOS — /Library/Application Support/0x101 Cyber Security/NetLock RMM/<Agent>/debug.txt

where <Agent> is Comm Agent, Remote Agent, or Health Agent.

Note: The Health agent checks the other services every 60 seconds and automatically starts a stopped Comm or Remote service. Always use a single restart command rather than stopping a service, waiting, and starting it again — otherwise the watchdog may race you.

Steps

1. Create an empty debug.txt in the data directory of each agent you want to debug.

Windows (PowerShell, elevated):

New-Item -ItemType File -Path "C:\ProgramData\0x101 Cyber Security\NetLock RMM\Comm Agent\debug.txt"
New-Item -ItemType File -Path "C:\ProgramData\0x101 Cyber Security\NetLock RMM\Remote Agent\debug.txt"
New-Item -ItemType File -Path "C:\ProgramData\0x101 Cyber Security\NetLock RMM\Health Agent\debug.txt"

Linux (Bash, with sudo):

sudo touch "/var/0x101 Cyber Security/NetLock RMM/Comm Agent/debug.txt"
sudo touch "/var/0x101 Cyber Security/NetLock RMM/Remote Agent/debug.txt"
sudo touch "/var/0x101 Cyber Security/NetLock RMM/Health Agent/debug.txt"

macOS (Terminal, with sudo):

sudo touch "/Library/Application Support/0x101 Cyber Security/NetLock RMM/Comm Agent/debug.txt"
sudo touch "/Library/Application Support/0x101 Cyber Security/NetLock RMM/Remote Agent/debug.txt"
sudo touch "/Library/Application Support/0x101 Cyber Security/NetLock RMM/Health Agent/debug.txt"

2. Restart the agent services so they pick up the file. When you enabled debugging on more than one agent, simply restart all three.

Windows (PowerShell, elevated):

Restart-Service NetLock_RMM_Agent_Comm
Restart-Service NetLock_RMM_Agent_Remote
Restart-Service NetLock_RMM_Agent_Health

Linux (Bash, with sudo):

sudo systemctl restart netlock-rmm-agent-comm
sudo systemctl restart netlock-rmm-agent-remote
sudo systemctl restart netlock-rmm-agent-health

macOS (Terminal, with sudo):

sudo launchctl kickstart -k system/com.netlock.rmm.agent.comm
sudo launchctl kickstart -k system/com.netlock.rmm.agent.remote
sudo launchctl kickstart -k system/com.netlock.rmm.agent.health

3. Reproduce the problem. Trigger the failing action again — run the job, start the remote session, wait for the sync cycle — so the agents write log lines while the problem happens.

4. Collect the logs. Each agent writes JSON-lines .txt files into a Logs subfolder next to its debug.txt (for example C:\ProgramData\0x101 Cyber Security\NetLock RMM\Comm Agent\Logs\). The key files are Debug.txt and Error.txt, plus topic files such as Jobs.txt, Sensors.txt, and PowerShell.txt on the Comm agent or Remote_Control.txt on the Remote agent. Copy or zip the entire Logs folder per agent rather than picking individual files.

Tip: Independent of debug mode, the services' standard output is available too. On Linux, use journald: journalctl -u netlock-rmm-agent-comm (likewise for the remote and health units). On macOS, the LaunchDaemons log to /var/log/netlock_rmm_agent.comm.log and /var/log/netlock_rmm_agent.comm_error.log (likewise .remote / .health).

5. Disable debugging again. Delete debug.txt from each data directory and restart the services with the same commands as in step 2. Debug logs grow unbounded, so do not skip this on a production device. You can delete the collected Logs folders afterwards to reclaim disk space.

Verify it worked

• After step 2, the Logs subfolder appears in each agent's data directory and Debug.txt inside it grows as the agent works — the Comm agent writes lines on every sync cycle, so within a minute or two the file should have content.
• After step 5, the services run without writing new lines — the timestamps on the files in Logs stop advancing.

Troubleshooting

• No log output appears even though debug.txt exists. The services were not restarted after the file was created — the check runs only at process startup. Restart all three services as in step 2. On Windows, also check the file name: with hidden file extensions enabled, Explorer's New → Text Document silently creates debug.txt.txt, which the agent ignores. Create the file from PowerShell as shown above to be safe.
• The agent is offline or cannot reach the server — is debugging pointless? No. The logs are written locally on the device, independent of the server connection, so connection problems are exactly what debug logging is for. Collect the Logs folders directly from the device (step 4) instead of expecting anything to show up in the Console.
• The Remote agent's Error.txt shows INADEQUATE_SECURITY in Service.Setup_SignalR, but the device is online. The device runs Windows 8.1 / Server 2012 R2 or older behind a reverse proxy that enforces the HTTP/2 cipher-suite rules; agents from version 3.2.0.4 on are not affected. See X.3.4 — Remote control for the proxy-side workaround for older agents.
• The device is behind an HTTP proxy — which proxy did the agent actually pick? With debug logging on, the Comm, Remote and Health agents write the decision to Debug.txt under Helper.Proxy_Resolver.Create_Proxy on every start: the mode (auto, manual, none), the proxy address, the authentication method and, if one is used, the user name. The password is never written. No such line means no proxy applies and the agent connects directly, which in auto mode is also what you see on a device that reaches the backend without one. The reachability check itself logs under Helper.Connectivity_Probe, including a proxy that answered 407. The settings the agent reads come from proxy.json next to server_config.json; see Devices behind an HTTP proxy.
• A device turns up under Unauthorized Devices although nobody deauthorized it. The on-device logs cannot show the reason, because that decision is made on the server. Read the device's authentication log in the Console first — see the next section.

When a device becomes unauthorized: read the Authentication tab first

A device that suddenly waits under Devices → Unauthorized Devices although nobody deauthorized it is the one agent problem the on-device logs cannot explain, because the decision is made on the server. From version 3.2.0.4 on, the server writes every agent check-in that it did not accept as a plain success to a per-device authentication log, and the Console shows it in two places:

• Devices → select the device → tab Authentication, for a device that is (again) authorized.
• Devices → Unauthorized Devices → row action Authentication log, for a device that just lost its authorization.

Both require the permission "Authentication log" in the device information tabs group of the user settings; accounts that hold the "General" permission received it with the upgrade.

Each row names the endpoint the request hit, the remote address, the agent version, the hardware ID the agent sent next to the one that was stored for the device, and a reason. Repeats of the same state within ten minutes are folded into one row whose counter grows, so a device that polls every 30 seconds while it waits for approval produces one row per ten minutes rather than twenty. The moment authentication succeeds again is recorded as "Authentication succeeded again". Successful check-ins of an authorized device are not listed, so an empty tab means the device has authenticated without trouble.

The log is written by the server, so agents need no update for it. Rows are kept for 90 days by default (Settings → Retention & cleanup). The server records nothing while the table does not exist yet and says so once in its log, which is why the web console is deployed before the server.

How the hardware ID is computed and stored

Next to the access key, every device identity carries a hardware ID. The server compares it with the stored one on every check-in and treats a difference as a reason to deauthorize (see the table above); it is also the value behind {{device.hwid}} in scripts. The access key remains the identifier to use when you refer to a device from outside.

Windows. From version 3.3.0.1 on, the Comm agent computes the ID from the SMBIOS system UUID (Win32_ComputerSystemProduct.UUID), the mainboard serial (Win32_BaseBoard.SerialNumber), the BIOS serial (Win32_BIOS.SerialNumber) and the processor IDs of all processors (Win32_Processor.ProcessorId, sorted). Placeholders that firmware ships instead of a serial — To be filled by O.E.M., Default string, System Serial Number, an empty value and similar — count as absent. When at least one of the UUID and the two serials is present, the ID is of tier hardware; a whitebox that has only placeholders gets tier os_bound, computed from the processor IDs and the Windows MachineGuid, which is unique per Windows installation and renewed by sysprep /generalize. Both tiers are a SHA-256 hash written as 64 hexadecimal characters; the previous format had 48 characters on x64 and 32 on arm64, so the two are easy to tell apart in the authentication log. Disks are no input any more, so a USB stick at boot, a docking station or a changed boot order no longer affect the ID.

The value is computed once and stored in hardware_id.json in the Comm agent data directory (C:\ProgramData\0x101 Cyber Security\NetLock RMM\Comm Agent\), protected in the same way as server_config.json next to it. On every later start the agent reads the file instead of querying WMI, so a WMI query that is slow or fails at boot cannot change the ID. The file also records the SMBIOS UUID the ID was computed with: when the UUID the machine reports differs from the recorded one — a VM copied with a new UUID, an image moved to other hardware, a replaced mainboard — the agent computes the ID afresh and overwrites the file, which is what keeps clones apart. A VM restored from a backup in place keeps its UUID and with it its identity.

• Deleting the file makes the agent compute the value again on its next start. On unchanged hardware the result is the same ID and the server sees no change, so this is safe when you suspect a corrupted file.
• A full uninstall removes the file together with the access key; an agent update or a repair with fix keeps it. After a reinstall with a fresh installer the agent arrives at the same ID, so the server matches the existing device row by name and hardware ID as described in the table above.
• Golden images must not contain server_config.json or hardware_id.json. Clones that carry both share one access key; those whose hypervisor kept the UUID also share the ID and merge into one device row, those with a new UUID compute their own ID and show up as alternating IDs from several addresses. Uninstall the agent before you seal the image and install it on each clone after provisioning.
• No ID yet. When WMI is unavailable at start and no file exists, the agent sends its identity without a hardware ID and retries in the background. A device that is already authorized stays authorized; a new device is created only once the ID could be computed.

Migration from the previous format. An updated agent sends the new ID together with the values the previous format could have produced on the machine. The server matches the new ID first; when instead one of the previous values equals the stored ID, it rewrites the stored ID in place, leaves the authorization untouched, remembers the previous value with the device, and writes the row "Hardware ID migrated to the new format", which shows the old and the new value side by side. A device that was already unauthorized because of a flipping ID is migrated the same way and needs one manual re-authorization, after which it stays stable. A stored value that none of the sent candidates reproduces — one computed with a disk that has since been removed, for example — takes the "Hardware ID changed" path once; re-authorize the device and it is stable from then on. Scripts that read {{device.hwid}} see the value change once at that moment.

Upgrade in the usual order — web console, then server, then agents. The web console adds the column in which the migration records the previous value, and an older agent against the new server behaves as before; a device is migrated the first time its updated agent checks in.

Linux, macOS and Android. Unchanged: the agents on these platforms keep the identity they had, send no migration data, and produce no authentication log rows because of this change.

Related

• Guide H.1 — Deploy your first agent — install the agent and its three services.
• Uninstall an agent — remove the agent, including its data directories and logs.
• X.3 — Troubleshooting — symptom-cause-fix entries for the most common agent problems.
• A.9 — Logging & protocols — the server-side Logging page, the counterpart to on-device debug logs.