Databl love to see customers implementing clever security controls. Databl also love to threat hunt using Splunk. Sometimes on a threat hunt we find some low hanging fruit and like to offer quick win solutions to keep our customers more secure.
LAPS (Local Administrator Password Solution) is one of those clever security controls we see implemented regularly. LAPS is a Microsoft tool that manages local administrator passwords of domain join systems. The tool provides the ability to change the local administrator password on a regular schedule, for example every thirty days. The rotated passwords are stored in Active Directory and only accessible by ACL.
By installing the LAPS tool you will be adding some schema extensions to computer objects. The computer object schema extensions are ‘ms-mcs-AdmPwd’ (the local administrator password) and ‘ms-mcs-AdmPwdExpirationTime’ (the password expiration time or when the LAPS client will reset the local administrator password next.).
Active Directory Monitoring in Splunk
Databl see a lot of customers monitoring Active Directory with Splunk and rightly so. A lot of customers use Splunk to monitor Active Directory which helps support maintaining it’s integrity, health and security. Customers typically monitor Active Directory using Splunk by enabling the “[admon://NearestDC]” input. This input is typically associated with the “ActiveDirectory” sourcetype, which we search for later in this article. There are some great examples of this inputs.conf configuration here: Admon example configurations
How do I know if Splunk is storing LAPS clear text passwords?
Databl typically see Splunk Universal Forwarders deployed on domain controllers running as the local ‘SYSTEM’ account. Although this will allow the Splunk software to work and collect the events you require for monitoring, in this instance you could be collecting a bit more than desired. By default the ‘SYSTEM’ account can read those new schema extensions you recently added ( ms-mcs-AdmPwd’ and ‘ms-mcs-AdmPwdExpirationTime’ ). Databl recommends to customers they run the Splunk service as a domain managed service account. Of course there are some requirements you need to consider when creating a managed service account to run Splunk so we have provided a great link here: Splunk managed service accounts.
Run the below search to check if your Splunk instance has any clear text local administrator passwords stored. The search will provide a stats table of all the clear text password collected via “[admon://*]” in Splunk grouped by host. It will be pretty obvious if they are present. If you can’t find any results, alter your search slightly. Maybe extend the timeframe, remove the “stats” or just search for events, also you might not see any results as “[admon://*]” is not actually enabled.
index=* sourcetype=ActiveDirectory ms-Mcs-AdmPwd=*
| stats values(ms-Mcs-AdmPwd) AS ms-Mcs-AdmPwd by hostThe ActiveDirectory event you’re looking for looks a bit like the below. You might see clear text passwords by looking at the value of (ms-Mcs-AdmPwd) field;
05/18/2021 13:34:15.347
dcName=test.testdomain.local
admonEventType=Update
Names:
objectCategory=CN=Computer,CN=Schema,CN=Configuration,DC=testdomain,DC=local
name=TESTACCOUNT0001
displayName=TESTACCOUNT0001$
distinguishedName=CN=TESTACCOUNT0001,OU=Desktops,OU=Computers,OU=TEST Exploration,OU=TEST Group,DC=testdomain,DC=local
cn=TESTACCOUNT0001
Object Details:
sAMAccountType=805306369
sAMAccountName=TESTACCOUNT0001$
logonCount=440
accountExpires=Never Expires
objectSid=S-1-5-21-2739638291-1717091454-2333388744-12404
primaryGroupID=515
pwdLastSet=06:33.51 PM, Sun 05/09/2021
lastLogon=12:55.11 PM, Tue 05/18/2021
lastLogoff=0
badPasswordTime=0
countryCode=0
codePage=0
badPwdCount=0
userAccountControl=4096
objectGUID=6b072253-8ce2-463f-82d4-ae07151ef0db
whenChanged=01:34.15 PM, Tue 05/18/2021
whenCreated=11:38.31 AM, Mon 12/07/2020
objectClass=top|person|organizationalPerson|user|computer
Event Details:
uSNChanged=58923199
uSNCreated=47952499
instanceType=4
Additional Details:
ms-Mcs-AdmPwd=KJ%HGVg2374#@$@!
ms-Mcs-AdmPwdExpirationTime=132658760552956999
msDS-SupportedEncryptionTypes=28
msDS-AuthenticatedAtDC=CN=TESTMACHINE01,OU=Domain Controllers,DC=testdomain,DC=local
lastLogonTimestamp=08:46.17 PM, Mon 05/17/2021
dSCorePropagationData=20210122054926.0Z|20201230022458.0Z|20201207042845.0Z|16010101181633.0Z
isCriticalSystemObject=FALSE
servicePrincipalName=CmRcService/TESTACCOUNT0001.testdomain.local|TERMSRV/TESTACCOUNT0001.testdomain.local|RestrictedKrbHost/TESTACCOUNT0001.testdomain.local|HOST/TESTACCOUNT0001.testdomain.local|CmRcService/TESTACCOUNT0001|TERMSRV/TESTACCOUNT0001|RestrictedKrbHost/TESTACCOUNT0001|HOST/TESTACCOUNT0001
dNSHostName=TESTACCOUNT0001.testdomain.local
operatingSystemVersion=10.0 (19042)
operatingSystem=Windows 10 Enterprise
localPolicyFlags=0
userCertificate=TESTTESTTESTTEST
description=TEST Media Workstation S/N: TESTHow do I resolve this?
If your search results did return some clear text local administrator passwords, then read on. There are several ways you can resolve this issue, below is a relatively quick way in Splunk.
Option 1 – Anonymise the clear text password in Splunk using props.conf and SEDCMD.
You can deploy a props.conf configuration utilising “SEDCMD” and have it apply to the sourcetype “ActiveDirectory”. This means, as the data is streaming into Splunk and before it ends up on disk “indexed”, it will be anonymised. To do this we utilise the Splunk “SEDCMD”. This is much like the sed command you would find on a *nix based system. Ideally the props.conf file should be placed on a Splunk Heavy Forwarder on it’s path towards the Indexer, you could place this configuration on an indexer, but to avoid potential performance issues try use a Splunk Heavy Forwarder.
The “SEDCMD” is very useful for anonymising data and altering raw events for several other reasons. Splunk document this command here: ( just search for ‘SEDCMD’ in these docs ).
High level explanation of the below SEDCMD
SEDCMD-anonymiseLaps = ‘s/ms-Mcs-AdmPwd\=.*/ms-Mcs-AdmPwd=####!!!!!#####/g’
- ‘s/‘ = we are informing SEDCMD to do a replace rather than a string substitution.
- ‘ms-Mcs-AdmPwd\=.*/‘ = this is the regex we are trying to match and replace.
- ‘ms-Mcs-AdmPwd=####!!!!!#####/‘ = this is the string we are replacing our regex match with.
- ‘g‘ = this means, replace all matches, rather than a specific match, where we would need to provide an integer.
You could deploy this via a specific barebones app and server class then manage it centrally via deployment server where appropriate/possible. That way if you need to anonymise other events you can more centrally.
# props.conf on a HF or Indexer (preferably HF).
[ActiveDirectory]
SEDCMD-anonymiseLaps = 's/ms-Mcs-AdmPwd\=.*/ms-Mcs-AdmPwd=####!!!!!#####/g'Once this configuration has been deployed to your Splunk Heavy Forwarder, make sure to restart Splunk service.
Verify that your clear passwords are no longer clear text and successfully been anonymised. Run the following search and your LAPS passwords should look like this instead
ms-Mcs-AdmPwd=####!!!!!#####
index=* sourcetype=ActiveDirectory ms-Mcs-AdmPwd=*
| stats values(ms-Mcs-AdmPwd) AS ms-Mcs-AdmPwd by hostSomething else to note.
All the clear text passwords up until the point when you configured the new anonymising props.conf configuration are still present. That’s because they were already indexed (placed on disk) before the config was implemented. Splunk cannot retrospectively anonymise data, so you will need to make a decision on what to do next.
To help you decide what to do next, take a look at the clean index command and delete commands here as a start: How to clean and delete Splunk data
That’s all for now folks, If you need any assistance with Splunk for security, Splunk for ITOPS or considering using Splunk as part of a observability development program please reach out to the team!