Anonymise your clear text LAPS passwords in Splunk

splunk-laps-anonymise-flow

Databl love to see customers implementing clever security controls. Databl also love to threat hunt using Splunk. Sometimes on a threat hunt we find some low hanging fruit and like to offer quick win solutions to keep our customers more secure.

LAPS (Local Administrator Password Solution) is one of those clever security controls we see implemented regularly. LAPS is a Microsoft tool that manages local administrator passwords of domain join systems. The tool provides the ability to change the local administrator password on a regular schedule, for example every thirty days. The rotated passwords are stored in Active Directory and only accessible by ACL.

By installing the LAPS tool you will be adding some schema extensions to computer objects. The computer object schema extensions are ‘ms-mcs-AdmPwd’ (the local administrator password) and ‘ms-mcs-AdmPwdExpirationTime’ (the password expiration time or when the LAPS client will reset the local administrator password next.).

Active Directory Monitoring in Splunk

Databl see a lot of customers monitoring Active Directory with Splunk and rightly so. A lot of customers use Splunk to monitor Active Directory which helps support maintaining it’s integrity, health and security. Customers typically monitor Active Directory using Splunk by enabling the “[admon://NearestDC]” input. This input is typically associated with the “ActiveDirectory” sourcetype, which we search for later in this article. There are some great examples of this inputs.conf configuration here: Admon example configurations

How do I know if Splunk is storing LAPS clear text passwords?

Databl typically see Splunk Universal Forwarders deployed on domain controllers running as the local ‘SYSTEM’ account. Although this will allow the Splunk software to work and collect the events you require for monitoring, in this instance you could be collecting a bit more than desired. By default the ‘SYSTEM’ account can read those new schema extensions you recently added ( ms-mcs-AdmPwd’ and ‘ms-mcs-AdmPwdExpirationTime’ ). Databl recommends to customers they run the Splunk service as a domain managed service account. Of course there are some requirements you need to consider when creating a managed service account to run Splunk so we have provided a great link here: Splunk managed service accounts.

Run the below search to check if your Splunk instance has any clear text local administrator passwords stored. The search will provide a stats table of all the clear text password collected via “[admon://*]” in Splunk grouped by host. It will be pretty obvious if they are present. If you can’t find any results, alter your search slightly. Maybe extend the timeframe, remove the “stats” or just search for events, also you might not see any results as “[admon://*]” is not actually enabled.

index=* sourcetype=ActiveDirectory ms-Mcs-AdmPwd=*
| stats values(ms-Mcs-AdmPwd) AS ms-Mcs-AdmPwd by host

The ActiveDirectory event you’re looking for looks a bit like the below. You might see clear text passwords by looking at the value of (ms-Mcs-AdmPwd) field;

05/18/2021 13:34:15.347
dcName=test.testdomain.local
admonEventType=Update
Names:
        objectCategory=CN=Computer,CN=Schema,CN=Configuration,DC=testdomain,DC=local
        name=TESTACCOUNT0001
        displayName=TESTACCOUNT0001$
        distinguishedName=CN=TESTACCOUNT0001,OU=Desktops,OU=Computers,OU=TEST Exploration,OU=TEST Group,DC=testdomain,DC=local
        cn=TESTACCOUNT0001
Object Details:
        sAMAccountType=805306369
        sAMAccountName=TESTACCOUNT0001$
        logonCount=440
        accountExpires=Never Expires
        objectSid=S-1-5-21-2739638291-1717091454-2333388744-12404
        primaryGroupID=515
        pwdLastSet=06:33.51 PM, Sun 05/09/2021
        lastLogon=12:55.11 PM, Tue 05/18/2021
        lastLogoff=0
        badPasswordTime=0
        countryCode=0
        codePage=0
        badPwdCount=0
        userAccountControl=4096
        objectGUID=6b072253-8ce2-463f-82d4-ae07151ef0db
        whenChanged=01:34.15 PM, Tue 05/18/2021
        whenCreated=11:38.31 AM, Mon 12/07/2020
        objectClass=top|person|organizationalPerson|user|computer
Event Details:
        uSNChanged=58923199
        uSNCreated=47952499
        instanceType=4
Additional Details:
        ms-Mcs-AdmPwd=KJ%HGVg2374#@$@!
        ms-Mcs-AdmPwdExpirationTime=132658760552956999
        msDS-SupportedEncryptionTypes=28
        msDS-AuthenticatedAtDC=CN=TESTMACHINE01,OU=Domain Controllers,DC=testdomain,DC=local
        lastLogonTimestamp=08:46.17 PM, Mon 05/17/2021
        dSCorePropagationData=20210122054926.0Z|20201230022458.0Z|20201207042845.0Z|16010101181633.0Z
        isCriticalSystemObject=FALSE
        servicePrincipalName=CmRcService/TESTACCOUNT0001.testdomain.local|TERMSRV/TESTACCOUNT0001.testdomain.local|RestrictedKrbHost/TESTACCOUNT0001.testdomain.local|HOST/TESTACCOUNT0001.testdomain.local|CmRcService/TESTACCOUNT0001|TERMSRV/TESTACCOUNT0001|RestrictedKrbHost/TESTACCOUNT0001|HOST/TESTACCOUNT0001
        dNSHostName=TESTACCOUNT0001.testdomain.local
        operatingSystemVersion=10.0 (19042)
        operatingSystem=Windows 10 Enterprise
        localPolicyFlags=0
        userCertificate=TESTTESTTESTTEST
        description=TEST Media Workstation S/N: TEST

How do I resolve this?

If your search results did return some clear text local administrator passwords, then read on. There are several ways you can resolve this issue, below is a relatively quick way in Splunk.

Option 1 – Anonymise the clear text password in Splunk using props.conf and SEDCMD.

You can deploy a props.conf configuration utilising “SEDCMD” and have it apply to the sourcetype “ActiveDirectory”. This means, as the data is streaming into Splunk and before it ends up on disk “indexed”, it will be anonymised. To do this we utilise the Splunk “SEDCMD”. This is much like the sed command you would find on a *nix based system. Ideally the props.conf file should be placed on a Splunk Heavy Forwarder on it’s path towards the Indexer, you could place this configuration on an indexer, but to avoid potential performance issues try use a Splunk Heavy Forwarder.

anonymise laps passwords splunk
High level Example

The “SEDCMD” is very useful for anonymising data and altering raw events for several other reasons. Splunk document this command here: ( just search for ‘SEDCMD’ in these docs ).

High level explanation of the below SEDCMD

SEDCMD-anonymiseLaps = ‘s/ms-Mcs-AdmPwd\=.*/ms-Mcs-AdmPwd=####!!!!!#####/g’

  • s/‘ = we are informing SEDCMD to do a replace rather than a string substitution.
  • ms-Mcs-AdmPwd\=.*/‘ = this is the regex we are trying to match and replace.
  • ms-Mcs-AdmPwd=####!!!!!#####/‘ = this is the string we are replacing our regex match with.
  • g‘ = this means, replace all matches, rather than a specific match, where we would need to provide an integer.

You could deploy this via a specific barebones app and server class then manage it centrally via deployment server where appropriate/possible. That way if you need to anonymise other events you can more centrally.

# props.conf on a HF or Indexer (preferably HF). 
[ActiveDirectory]
SEDCMD-anonymiseLaps = 's/ms-Mcs-AdmPwd\=.*/ms-Mcs-AdmPwd=####!!!!!#####/g'

Once this configuration has been deployed to your Splunk Heavy Forwarder, make sure to restart Splunk service.

Verify that your clear passwords are no longer clear text and successfully been anonymised. Run the following search and your LAPS passwords should look like this instead
ms-Mcs-AdmPwd=####!!!!!#####

index=* sourcetype=ActiveDirectory ms-Mcs-AdmPwd=*
| stats values(ms-Mcs-AdmPwd) AS ms-Mcs-AdmPwd by host

Something else to note.

All the clear text passwords up until the point when you configured the new anonymising props.conf configuration are still present. That’s because they were already indexed (placed on disk) before the config was implemented. Splunk cannot retrospectively anonymise data, so you will need to make a decision on what to do next.

To help you decide what to do next, take a look at the clean index command and delete commands here as a start: How to clean and delete Splunk data

That’s all for now folks, If you need any assistance with Splunk for security, Splunk for ITOPS or considering using Splunk as part of a observability development program please reach out to the team!