Why & How of Monitoring Workstations with Log & Event Manager

In case you missed it, the Log & Event Manager team has recently rolled out new pricing related to monitoring workstation nodes. The goal of this addition is to make it much more affordable for you to monitor workstations together with your servers and network devices in LEM - or even by themselves, if you're solely workstation-minded. It's still the same LEM with the same features and functionality, this just makes it much more possible for you to extend your investment.

So, what does that really mean? What would you want to monitor from workstations? And, how do you do that with LEM?

Issues Specific to Workstations

Traditionally we focus a lot on servers, but realistically workstations are both the entry point to the network from a security perspective and more systems that require maintenance. As you think about moving away from reactive network/systems/security management to proactive network/systems/security management, workstations are a critical part of our enterprises.

Security

From a security perspective, workstations do give you an entry point to the network, and can serve as a gateway to a veritable feast of data. Helpful: your customers and users can access the network quickly and easily from their system to do their jobs well. Not helpful: they have access to so much information and systems that they can also do some serious damage.

Things to monitor:

• Unexpected users logging on to workstations that are more likely to have sensitive information - C-level, VP, and IT administrators. Create a group of users that SHOULD have access to these systems and look for authentication activity (logons and failures) that are to those systems but not from those users.
• Other forms of unexpected logon activity depending on your environment - logons to workstations after hours if you're in a fairly controlled environment, remote logons if you don't use VPN access or users don't use RDP
• Changes (create, update, delete) to local accounts and groups, especially Local Admins and accounts that won't inherit your domain policies and settings
• System changes, like installation of unexpected software and changes to local policies
• Usage of removable USB disk and networking devices
• Launch of prohibited applications (IM, games, etc)
• Patterns of behavior that are not unusual in the one-off case but are in excess, like failed logons

Changes and Issues

Monitoring log data from workstations can also grant you insight into the state of the system - if a user calls and complains about something not working correctly, the event log and recent history of activity can provide a lot of useful data.

Things to monitor:

• Software installation, successful and failed
• Installation of Windows/OS updates, especially failed updates
• Changes to system policies and configurations (enable/disable of Windows Firewall, enable/disable of audit policy)
• Failures related to services starting/stopping
• For Windows, "Critical", "Error", and "Warning" events in general in the System and Application logs

Active Responses & Workstations

Useful active responses and scenarios for workstations include:

• Detecting suspicious (or unapproved) processes and killing them (by name or ID)
• Disabling networking on a workstation after detecting a malware infection (to isolate from the network)
• Detaching a USB device that's not approved - this one can be done whether the agent is connected or not with our USB local whitelisting policy
• Detecting unexpected or inappropriate network, proxy, or file activity and sending a popup to the workstation notifying the user they've been spotted
• Removing unapproved users from Local Admins automatically, or disabling local users if they are created

Combining Workstation and Network/Server Data

In some cases, data specific to workstations is actually centralized at the server or network device, but you might not have thought about specifics of things to look for for workstations or endpoint issues. There's also some cool things you can do if you correlate activity across multiple sources.

Centralized Events

• Anti-Virus and DLP: It's most common for your anti-virus and DLP solutions to log centrally, rather than at the endpoint themselves. These events can provide critical insight into security issues directly at the workstation.
  • Look especially for viruses that are "left alone" (not cleaned, not quarantined) and unexpected data that has moved from the endpoint.
• Look for Firewall/router data that indicates a workstation:
  • attempting to make outbound connections to unexpected ports
  • bypassing your proxy server for port 80 traffic
  • making excessive repeated outbound attempts to a single source/destination/port
• If you have a web proxy, use that data to monitor repeated attempts to access blocked content, repeated download attempts for viruses or other suspicious content, downloads of executables
• A fair amount of your domain controller and other server activity is related to access from workstations (since that's where your users are, after all). You can use this to extend your monitoring of certain types of logon activity that comes from clients and software solutions that are not directly logged at the workstation.
• DHCP/DNS issues can surface at the DHCP/DNS server side, but indicate workstation problems. With DHCP, especially, you can track whether your server has seen a request for a lease and what the response actually was (before you break out the packet capturing tools to dig deeper).

Correlated Activity

• Correlate authentication activity across servers and other workstations that indicates logon attempts from a single source, which can be symptomatic of an infection or exposure
• If you monitor file access, monitor for excessive deletes or copies from a single system, and potentially correlate with the USB activity from the workstation itself to indicate files copied from a server all the way to a USB drive
• Combine suspicious activity to create a more conclusive case that something's wrong - for example, combine excessive logon failures to multiple systems on the network with excessive outbound traffic or combine virus/malware activity with executable downloads

Where to go in LEM

If you want to be alerted when above activity occurs (via e-mail) or automatically respond to the workstation, you need to go to Rules (Build>Rules). Most of the items above are really good candidates for rules. Other areas to look in will be:

• Rule Library/NATO5 Rules > Agent: Especially "Detach USB" rules, "Windows Disk Nearly Full", "Keylogger Process Launch", "Authentication Traffic but no Agent"
• Rule Library/NATO5 Rules > Active Responses: Especially "Kill Suspicious Process", "Game Application Launch", "Remote Desktop After Business Hours", "Restart Stopped AV"
• Rule Library/NATO5 Rules > Authentication: Especially "Logon Attempt outside of Time Restrictions", "User Logon After Hours", "User Logon but no Agent"
• Rule Library/NATO5 Rules > Change Management: If you're interested in tracking workstation changes, many of the same rules apply here, or will indicate activity coming from workstations.
• Rule Library/NATO5 Rules > Spyware
• Rule Library/NATO5 Rules > Virus/Worm: Especially "AV Update Failure" and "Virus Attack - Bad State"

If you want to search for activity that has occurred based on a workstation's name and/or IP address, you want to go to nDepth (Explore>nDepth).

• To search for any events that contain the workstation's name or IP, just type it in the search box - this searches globally.
• To search for any events from a specific workstation, use the DetectionIP field (or InsertionIP, they'll usually be the same on workstations)
• To search for any events that came from, were going to, or were created by, a workstation's name or IP,  use the combined IP Address field

If you want to monitor workstations in real time, you can use the widgets in Ops Center to view trends and anomalies, and you can use filters in Monitor to, well, monitor for different categories of activity. Good candidates for filters are things like:

• Activity from high-profile workstations
  • Create a Tool/Connector Profile or a User-Defined Group with your workstations in it
  • Build a filter for Any Alert.DetectionIP = <group>
  • This will be fairly high-traffic depending on the number, so you might need to narrow the focus to specific types of events.
• Logon failures only to workstations
  • Create a Tool/Connector Profile (or multiples) with your workstations grouped together
  • Build a filter for UserLogonFailure.DetectionIP = <group> (if you have more than one, OR them together)
  • If you want to look for workstations generating logon failures on other systems, use UserLogonFailure.SourceMachine = <group>
  • If you only want to monitor interactive logons (RDP or local), use UserLogonFailure.LogonType = *interactive*
• Workstation change activity
  • Again, Create a Tool/Connector Profile (or multiples) with your workstations grouped together
  • Build a filter for [Change Management Events].DetectionIP = <group?> (if you have more than one, OR them together)

Tips for Managing Workstations in LEM

1. Deploy agents many at a time using the remote agent installer, by deploying the agent with your workstation image, or by using the local installer in "silent" mode and using it with your software distribution tools.
   1. If you're using the remote agent installer and have remote sites, a helpful tip is to copy the installer to a system (e.g. server) local to that remote site, then push out from there.
   2. KBs: SolarWinds Knowledge Base :: How to include the LEM Agent in a Windows image, SolarWinds Knowledge Base :: Using the SolarWinds LEM Agent Installer non-interactively
2. Use Tool/Connector Profiles to group agents together. This serves the purpose of grouping AND maintaining a standard configuration template. Configure a single agent, then create a new tool/connector profile and add all of the similar agents with it.
3. If you have mixed Windows environments, look out for configuring the "wrong" connectors for the Windows Security Log. You'll see Internal alerts that suggest you should configure the opposite connector (there's one for XP/2003 and earlier, and one for Vista and later).
4. The Manage>Nodes grid can be sorted, sliced, and diced, to take inventory of what's connected and not. The new "Node Health" widget in our upcoming release (RC info available here) will show the last time data was received from nodes, which is helpful. There's also a couple of agent health reports in LEM Reports that can help track down agent connectivity and events.

The recent thwackCamp presentation on the Top 10 Things Logs Can Do for You might have some additional ideas to help spark your creativity in monitoring workstations and your enterprise holistically with LEM.

What about you? Do you monitor workstations? Is there anything you'd like to monitor but aren't sure how? Haven't heard about LEM Workstation Edition and want to know more about what it means? Drop a comment here or feel free to start your own discussion thread over in the Log & Event Manager space.

Shameless Plug: Other SolarWinds Products for Workstations

While we're on the topic, here's some other good stuff for workstations that will help extend what you get with LEM even further:

• Patch Manager: not just help with managing your windows patches, but helping address third party patching issues. On top of the fact that Acrobat, Flash, and Java have had a ton of security issues, a lot of malware out there still exploits old holes that are fixed with patches. Keep it up to date from one place.
• DameWare: DameWare is a handy remote management tool. Once you've identified a problem with LEM, if you want to investigate at the endpoint or respond beyond LEM's built-in active responses, DameWare can help.
• NetFlow Traffic Analyzer: if you've got bandwidth problems with workstations, use NTA to track down who is consuming it. LEM can help on a time & frequency basis and can do some basic top talker stuff with NetFlow/sFlow, but NTA is all flow all the time.
• User Device Tracker: UDT helps you determine what user owned an IP address/hostname/MAC address over time. If you've found a historical issue on LEM and want to trace that IP back to a user, UDT can tell you where they were.
• LANsurveyor: map out your network and figure out the logistical layout of devices. If you've got workstations, chances are there's enough of them that you'd like to know what and where they are connected. LEM doesn't have built-in network diagrams itself, but this can help you make sense of what's out there.