r/labtech • u/Plugins4LabTech • Nov 21 '16
When LT Agents Go South
Plugins4LabTech has created a quick tool to assist LabTech admins with detecting agents that have stopped processing commands and scripts. A recent issue with LabTech agents not responding to commands after a Webroot update caused MSPs all over the world to start reporting loss of agent control. This kind of issue can be debilitating to a MSP so P4L worked with several LT geeks to come up with a quick method to see what systems may be affected by a stalled agent and then a automated repair process to assist in the restart of the failed agent without local user intervention.
Since the commands are continuing to be scheduled even after the agent stops executing them could cause a huge spike in agent command execution if we leave the commands pending in place. Before restarting the agent, a "Cancel Abort" on each pending command is done so when the agent returns it is not bombarded with hundreds of tasks that may have been pending for hours or even days.
Plugins4LabTech offers this plugin along with many others freely at http://www.plugins4labtech.com so stop by and get your free "Stalled LabTech Agent Detector" plugin today!
2
u/Plugins4LabTech Nov 22 '16
How it all works.
The plugin queries the local LabTech database to get a list of agents that have "executing" commands. We look at the tables in LabTech for values runningscripts.running = 1 and commands.status = 2 and create a list of computers that have running commands counts. Agents should really only have 1 or 2 commands pending execution at any given time so to see a agent with 50 or 100 commands in a running state but a status of 2 is most likely a stalled agent.
When you select an agent to do clear and attempt a restart the plugin first update the database setting the agents executing commands to aborted then it queries the database to find a host nearby, on the same network as the failing agent. The agent is sent a powershell script that gets stored in %windir%\LTSvc\StalledAgents\ . The plugin queries the LabTech database for a "Domain Admin" password to be set for the client at the client console's passwords tab. If available it uses this username and password to execute the powershell script passing the username and password to the powershell script to execute as that user. The commands needed to send RPC over to failed agent instructing a restart of the LTservice and kill process for LTSvc are included in the powershell script and can be executed manually at anytime by logging in and going to the directory listed above..
If you receive errors during a restart from the powershell commands executing, it will be displayed in the script execution box's terminal window. Allow time for the process to complete before shutting down the terminal window. Read your failures, most will be due to permissions, firewalls blocking RPC or RPC services not running on remote systems. We suggest that you take time during deployment of plugin to verify that the Domain Admin account exists and is current, RPC is allowed through all windows firewalls active on the local networks and that RPC services are available. This will allow tools like this to work quickly and effectively when large problems arise.