Issue
Agents fails to discover during a task.
Status: Failed
Result: Cannot Find Agent
Return code: 1110
Agent Discovery Process
When a scheduled task is set to Push, these are the steps it goes through to discover a client to provide the task to.
Logs
These logs will be used in diagnosing this issue:
- C:\ProgramData\Landesk\log\PolicyTaskHandler.log
- C:\Program Files\LANDesk\ManagementSuite\log\PolicyTaskHandler.exe.log
Is the machine online?
Yes - Device is online
If the machine is online, and continues to fail discovery, continue on to Setup a Discovery Troubleshooting test.
No - Device is not online
The most common reason for a machine failing to be discovered is that it is offline. A machine that is not online, cannot be discovered.
Can you ping the device?
Can you validate through some other method it is online?
Setup Discovery Troubleshooting Test
When the LDMS core is attempting to discover an agent, it will send a 'discovery' packet to the client. When the client receives the discovery packet, it will respond with information about the device to let the Core know who received the discover request.
Follow these steps to setup a Discovery Troubleshooting Test.
Enable verbose policy task handler logging
When the Core sends a discovery packet, it will log the attempt in the PolicyTaskHandler.exe.log, if verbose logging for this process has been enabled.
- On the LDMS Core, in the Scheduled Task panel, select the Configuration Settings option (gear icon) and choose Default Scheduled Task Settings
- In the Default scheduled task settings window under Accelerated Push, check the box for Enable verbose policy task handler logging
- Click Save
Note: The changes will only take effect on Tasks started after the change has been saved.
Setup Wireshark on core
Because the Core is involved in a large amount of network traffic, we want to filter down the traffic that Wireshark actually listens for.
- In Wireshark choose Capture | Options.
- In the Wireshark: Capture Options window choose Capture Filter:
- In the Wireshark: Capture Filter - Profile: Default window click the New button
- Select New Filter at the bottom of the list
- Set the following -
- Filter name: Arp or UDP Port 9595
- Filter string: arp or udp port 9595
- (The filter string is case sensitive, so make it just as listed above)
- The entry in the list will now reflect the Filter Name entered
- Click Ok
- In the Wireshark: Capture Options window, the Capture filter will show the Arp or UDP Port 9595 filter.
Note: If the bar is green the filter is acceptable. If the bar is red, the filter is malformed. Go back and make it match exactly as above
- Click Start to begin the Capture
Example of malformed filter.
End Other PolicyTaskHandler Processes
If you have ran several tasks, you may have multiple instances of PolicyTaskHandler.exe running, all of which are sending discovery packets.
To limit the amount of traffic we gather in Wireshark, and in the PolicyTaskHandler.exe.log, we want to end any other PolicyTaskHandler.exe processes running. This can be done through Task Manager.
Note: Ending this process will stop any agent discovery occuring on that particular task. Because terminating the process will stop discovery, the Scheduled Task may stay in a 'Discovering Agent' state indefinitely. The task will need to be re-ran later for a status change.
Create Scheduled Task to Test
You can use an existing scheduled task if desired, so long as you terminated it's running PolicyTaskHandler.exe process as outlined above.
- Schedule the task, and assign machines to be pushed to.
- Right click the task and choose Info
- Take note of the TaskID
- We will use this when looking through logs to identify which lines the task belongs to
Were Discovery Attempts Logged?
When the Core attempts to discover a machine, it will log the machine info attempting to be discovered in the PolicyTaskHandler.exe.log (so long as verbose logging was enabled).
07/20/2015 07:00:33.2837 INFO 6812:1 RollingLog : [Task: 7zip 9.20 - 7/20/2015 7:00:18 AM, TaskID: 1541, ProcID: 6812] : Discover: Discovering machine: [96-AGENT] using it's known ip address [10.14.130.61]...
07/20/2015 07:00:35.2996 INFO 6812:1 RollingLog : [Task: 7zip 9.20 - 7/20/2015 7:00:18 AM, TaskID: 1541, ProcID: 6812] : Discover: Successfully discovered machine: [96-AGENT]
Yes
Continue to Were Discovery Packets Sent?
No
- Verify that Verbose Logging is enabled
- Run the task again
- Were Discovery Attempts Logged?
If Discovery attempts were still not logged, verify that new entries are being logged at all to the PolicyTaskHandler.exe.log.
If PolicyTaskHandler.exe runs, it should log accordingly, and utilize the verbose option if enabled. If yours is not doing this, then it is likely that now PolicyTaskHandler.exe isn't getting to run which means the task will not go to 'Unable to discover agent'.
Verify the Landesk Scheduler Service is running.
Review other logs to identify if there are any errors that may indicate why PolicyTaskHandler was unable to run.
Example: TaskHandlerProxy.exe.log shows PolicyTaskHandler.exe couldn't be found.
07/24/2015 08:03:28.0657 INFO 8044:1 RollingLog : TaskHandlerProxy: C:\Program Files\LANDesk\ManagementSuite\PolicyTaskHandler.exe /TASKID=2554
07/24/2015 08:03:28.2991 INFO 8044:1 RollingLog : TaskHandlerProxy: run process exception - The system cannot find the file specified
Were Discovery Packets Sent?
To verify the traffic we are looking at is for our test task specifically, do the following:
- In the PolicyTaskHandler.exe.loglook for the first line that lists Discovering Machine for our TaskID
Note: Our task ID is 1547
RollingLog : [Task: 7zip 9.20 - 7/24/2015 7:55:58 AM, TaskID: 1547, ProcID: 10500] : Discover: Discovering machine: [TABLET] using it's known ip address [10.0.0.6]...
- In the Wireshark trace, filter for only discovery packets for the first IP address that belongs to our task.
Example: udp.port == 9595 && ip.addr == 10.0.0.6
- The filtered trace should show the Source as the LDMS Core, and the Destination as the client IP address we got from the PolicyTaskHandler.exe.log
Example: LDMS Core - 10.14.130.58 Client - 10.0.0.6
Repeat this check against the 2nd and 3rd IP addresses as listed in PolicyTaskHandler.exe.log for the taskID.
Spot check a couple more from the list as well, sometimes only a single packet is sent to the first IP which is easier to identify by checking several in the list.
Yes - Discovery Packets Sent
If the couple IP's checked did send discovery packets, continue to: Did the Client Respond to Discovery?
No - Discovery Packets Not Sent / Single Discovery Packet Sent
PolicyTaskHandler.exe.log showed attempting to discover devices, but there were no discovery packets (shown on udp.port == 9595) going to it.
No ARP Response
When determining who to send the discovery packet to, Windows will check if the client IP address is in the same subnet as the sender. If the Core and client are in the same subnet, an ARP Request will go out, asking who currently has the target IP Address. If any machines on the subnet receive the ARP Request and contain information on who has the IP address, they respond with the client information so Windows can route the packet to the client.
Example: The Core (10.14.130.58) is trying to discover Client (10.14.130.61). Because they are in the same subnet, an ARP Request is issued asking who has the 10.14.130.61 address. An ARP Response is sent back from a machine on the subnet indicating which device has the IP, and the Discovery packet is issued.
If there is no ARP Response to the request, Windows will not know where to send the discovery packet, and it will not be sent.
Example: The Core (10.14.130.58) is trying to discover Client (10.14.130.250). Because they are in the same subnet, an ARP Request is issued asking who has the 10.14.130.250 address. No machines send an ARP Response to indicate who has the IP, so no discovery packet is sent.
If no discovery packets were sent to any of the clients on the task, but PolicyTaskHandler.exe.log shows that we attempted to discover a device by IP, Windows may have not got
Single Discovery Packet
ARP Responses Received, No Discovery Packet Sent or Single Discovery Packet Sent
- PolicyTaskHandler.exe.log shows many discovery attempts for a task
- Wireshark shows a single Discovery Packet go out
-or-
- PolicyTaskHandler.exe.log shows many discovery attempts for a task
- Wireshark shows ARP Responses to local subnet ARP Requests, but no Discovery Packets go out
This is typically caused by 3rd party security software on the LDMS core stopping the UDP Traffic, typically due to its resembelance to a UDP Flood Attack. We have seen Symantec Endpoint Protection (SEP) behave in this manner.
07/24/2015 08:44:10.4463 INFO 18096:1 RollingLog : [Task: 7zip 9.20 - 7/24/2015 7:55:58 AM, TaskID: 2554, ProcID: 18096] : Discover: Discovering machine: [TABLET] using it's known ip address [10.0.0.6]...
07/24/2015 08:44:18.7586 INFO 18096:1 RollingLog : [Task: 7zip 9.20 - 7/24/2015 7:55:58 AM, TaskID: 2554, ProcID: 18096] : Discover: Discovering machine: [SQLTEMP] using it's known ip address [10.0.0.7]...
07/24/2015 08:44:30.8520 INFO 18096:1 RollingLog : [Task: 7zip 9.20 - 7/24/2015 7:55:58 AM, TaskID: 2554, ProcID: 18096] : Discover: Discovering machine: [SALLYPC] using it's known ip address [10.0.0.8]...
07/24/2015 08:44:39.1643 INFO 18096:1 RollingLog : [Task: 7zip 9.20 - 7/24/2015 7:55:58 AM, TaskID: 2554, ProcID: 18096] : Discover: Discovering machine: [TOMPC] using it's known ip address [10.0.0.9]...
07/24/2015 08:44:52.0079 INFO 18096:1 RollingLog : [Task: 7zip 9.20 - 7/24/2015 7:55:58 AM, TaskID: 2554, ProcID: 18096] : Discover: Discovering machine: [TESTWINTEST] using it's known ip address [10.14.130.51]...
07/24/2015 08:45:00.3360 INFO 18096:1 RollingLog : [Task: 7zip 9.20 - 7/24/2015 7:55:58 AM, TaskID: 2554, ProcID: 18096] : Discover: Discovering machine: [WIN7X64] using it's known ip address [10.14.130.52]...
07/24/2015 08:45:12.4291 INFO 18096:1 RollingLog : [Task: 7zip 9.20 - 7/24/2015 7:55:58 AM, TaskID: 2554, ProcID: 18096] : Discover: Discovering machine: [Lab-pc] using it's known ip address [10.14.130.55]...
07/24/2015 08:45:25.3191 INFO 18096:1 RollingLog : [Task: 7zip 9.20 - 7/24/2015 7:55:58 AM, TaskID: 2554, ProcID: 18096] : Discover: Discovering machine: [BETA] using it's known ip address [10.14.130.55]...
07/24/2015 08:45:33.6634 INFO 18096:1 RollingLog : [Task: 7zip 9.20 - 7/24/2015 7:55:58 AM, TaskID: 2554, ProcID: 18096] : Discover: Discovering machine: [WIN8X64] using it's known ip address [10.14.130.56]...
To correct this issue:
- Configure the security software to allow UDP Traffic out from LDMS
-or-
- Remove the software from the LDMS Core
Did the Client Respond to Discovery?
After the Core sends a discovery packet to the client, it looks for a response from the client indicating it is online and available.
In the wireshark trace, apply a filter to show only discovery packets and the client's ip address.
Example: udp.port == 9595 && ip.addr == 10.14.130.61
- There should be a discovery packet going from the LDMS Core (Source) to the Client (Destination). This is reflected in the first line of the screenshot.
- There should be a response from the Client (Source) to the LDMS Core (Destination).
Yes - Client responded
Continue to Client responds but is still marked 'Failed to Discover'
No - Client did not respond
Continue to Does the Client receive the discovery packet?
Does the Client receive the discovery packet?
Verify the client is actually online. If it is not online, it cannot receive the discovery packet.
If the device is online, install Wireshark on the client and trace the discovery attempt.
Example: udp.port == 9595 && ip.addr == 10.14.130.61
{steps on manually doing a single discovery)
Note: Core IP = 10.14.130.58, Client IP = 10.14.130.61
No discovery packet received
Discovery Packet received, but no response returned to Core
Yes - Client received discovery packet
If wireshark shows that the Client did see the discovery packet, but did not respond to it, this could indicate an issue with the Agent. Things to try:
- Verify the LANDesk(R) Management Agent is started.
- Reboot the client
If the agent services are all running, and continue to not return a response to the discovery packet, check Client side logs under C:\ProgramData\Landesk\Logs for any errors.
No - Client did not receive discovery packet
If wireshark does not show that a discovery packet was received, this is an indication that the packet is not routing to the client correctly. This could be caused by:
- Firewall blocking packets/ports
- Network routing packets to wrong machines
Correct this issue internally to allow the client to receive the packet.
Client responds but is still marked 'Failed to Discover
In LDMS 9.6, discovery attempts now include client validation. This helps to verify that the machine responding to the Discovery attempt, is the machine we are intending to reach.
This is done by parsing the Clients Discovery response, which contains inventory information about the device who is responding.
You can simulate the discovery and view the response with the following command: pds2dis.exe ping {client ip}
Example: C:\Program Files\LANDesk\ManagementSuite\pds2dis.exe ping 10.14.130.61
Response:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<discover>
<response><address>10.14.130.61</address><XHSH>3D820E0A</XHSH><APID>ping</APID><MAID>FE495D5D-404A-234C-9CAB-697F258E8DB7</MAID><FQCN>96-Agent.evdomain.local</FQCN><MACA>000C29C44B0A</MACA><MASK>255.255.255.00.0.0.0</MASK><AGRP>EVDOMAIN</AGRP><OSFM index="7">Win32</OSFM><OSNM index="4">WinNT</OSNM><OSVR>060223f0</OSVR><CERT>;55e3c398</CERT><PONG size="8">Q0JBOAAAAAA=</PONG></response>
</discover>
If the response contains information that does not match the clients Inventory information in the LDMS core, it will be considered the 'wrong machine' and marked as 'failed to discover'.
This can be viewed within the C:\ProgramData\Landesk\Log\PolicyTaskHandler.log
07/20/2015 07:00:33.2837 PDS2CallbackFunc: not the same deviceId - ip=3d820e0a
Note: The ip address is listed here as a hexidecimal representation of the IPv4 address. Visit this article on how to convert the values to Dotted IPv4 addresses.
How to Convert IP from Hex to Dotted Format
Check Client with Real-time Discovery
A device that was PolicyTaskHandler.exe.log shows as failed to discover, but that Wireshark shows as having responded to the Discovery likely has Inventory data that conflicts with what was returned in the discovery.
This can be checked quickly by:
- Right click the client in Inventory and choose Scheduled Tasks and Diagnostics
- In the Scheduled tasks and diagnostics window click the Real-time Discovery button (globe in upper left corner)
- The Discovery Information window will open. At this point, the LDMS Core is sending Discovery packets to the client, waiting for returned xml data.
- All fields stay at 'No data' - This indicates that we never got a response from the Client. This machine would be marked 'Failed to Discover'.
- Discovery response XML populates, and all fields remain same color - This indicates that the Client responded, and appears to be the Client we were looking for. This machine would be a successful Discovery.
- Discovery response XML populates, and one or more fields are red - This indicates that the Client responded, but the response contained data that conflicts what we have in Inventory. This machine would be marked as Failed to Discover, and not be given the task.
If Real-time Discovery shows a mismatch, correct the inventory issues at hand in order to Discover the affected client.
