Update:  To see all articles in this series click here.

No matter how bad he wants to an elephant simply cannot drive a Volkswagen.  He just won’t fit in that space.  People can drive VWs, because that is who the VW was designed to accommodate.  There is nothing worse than frustrated elephants.  What does this have to do with PowerShell or Active Directory?  I’m glad you asked.

The Problem:  Token Size and SID History

When users become members of too many groups their access token grows so large that it no longer fits inside some of the default OS setting.  Users can see issues logging in, applying group policy, and authenticating to web servers.  There are several other good articles that explain this scenario, and they are linked at the end of this post.

Token size issues can usually be attributed to a combination of three scenarios:

1. SID history left over from AD migration projects
2. Poor group nesting strategy
3. Old groups that are never purged

Today we will address the SID history scenario with a PowerShell script that documents the extent of SID history in your environment and creates a SID mapping file for use with the ADMT to migrate resources to the new SIDs.

Tell me if this scenario has ever happened at your company.  You are doing an AD migration with the ADMT (or a similar product).  You migrated the users and groups.  The project schedule was falling behind.  You take a shortcut with the member servers.  Instead of using the ADMT security translation process you just rejoin the servers to the new domain, leaving all of the old ACLs and SIDs untouched.  You leave SID history in place and move on to the next big project and deadline.

Management is happy that you met the deadline, but little do they know that you created an entire zoo full of elephants whose VWs are starting to feel kind of tight.

This scenario is common, and I want to explain the risks here.

1. Down the road you are likely to encounter token size issues, because now all users and groups have at least two SIDs.
2. If you decide to clean up SID history in the future you have the potential of orphaning users from their data, because the old SIDs were still giving them access.
3. THE BIG ONE:  When the project is decommissioned and the ADMT is blown away you lose all of the OldSID/NewSID migration data needed to clean the ACL SIDs on your migrated resource servers.

Many companies that I work with find themselves in this place.  Users are calling the help desk with mysterious symptoms, and they are not sure what is causing the problem.  Often it is token size, and SID history is a significant component.  At this point you could purge the SID history, but then you could lose access to all of the migrated data with old SIDs in the ACLs.  The best way forward is to go back and finish the AD migration by doing security translation on all of the migrated resources.

The Solution

The following steps will rid your environment of SID history:

1. Identify all servers in the environment that were around during the AD migration(s).
2. Install the latest version of the ADMT on a member server in your domain (NOT A DC, follow the guide linked below).
3. Run the PowerShell script from this article to create a SID mapping file.
4. Run the ADMT security translation wizard against each of the old servers, and use the SID mapping file when prompted to retrieve objects for security translation.  View the ADMT log files to see where changes were made.  (I started to write a PowerShell script to do the security translation, and I got it working for file share permissions by editing SDDL strings.  That was pretty cool, but it would take too much time to rewrite all seven steps built into the ADMT wizard.)  The ADMT wizard will clean SIDs in the following locations:
   1. Files And Folders
   2. Local Groups
   3. Printers
   4. Registry
   5. Shares
   6. User Profiles
   7. User Rights
5. In phases purge SID history from users and groups, starting with a small test population and then going by department until it is entirely removed from the environment.  Use targeted LDAP query strings to feed the PowerShell script linked at the end of this article to do your purges.  DO NOT PURGE ALL SID HISTORY AT ONCE.  That could be a resume-generating event if you missed some resources in the re-migration.  Do it in smaller batches to be safe.  Rerun the script in this article to see where SID history remains.

Enter PowerShell.  Since the ADMT database is long gone, we have to rebuild the SID mapping between old and new SIDs.  Lucky for us this data is all present in AD, because you told the ADMT to keep it.  We just need to put it into a file that the ADMT wants, called the SID mapping file.  This file format is simply a CSV file where the first column contains the old SID and the second column contains either the new SID or the new user name in DOMAIN\USERNAME format.  The script in today’s post will do this for you.  This script is entirely safe for your environment, since it makes no changes and only reads data.

In addition to the mapping file the script will also generate a CSV report of all SID history in your domain.  It includes the following columns:  samAccountName, DisplayName, objectClass, OldSID, NewSID, DistinguishedName.  Use this report before the clean up to assess the scope of the issue.  Run it again after the clean up to make sure you got it all.

Another handy way to view SID history for a specific user is with NTDSUTIL.  You can use the command Group Membership Evaluation to view all of a user’s groups, and it will tell you which ones come from SID history.  This is another process to spot check SID history before and after the cleanup.

The Script

This script is rather simple really.  Initially I thought you could dump all SID history with a PowerShell one-liner.  And you may, but the trick is that the SID history attribute is a multi-value field.  This syntax will not work:

Get-ADObject -LDAPFilter "(sidHistory=*)" -Property sidHistory | Export-CSV SIDHistory.csv

This will get you a query of the objects, but the SID history column on every row will say “Microsoft.ActiveDirectory.Management.ADPropertyValueCollection”.

Author's note:  See the new-improved version of this script as a one-liner here.

To do this properly and make the report generation easy I used two loops.  The outside loop iterates through all of the objects with SID history, and the inside loop iterates through each of the multi-value SID history attribute values.  Then I capture the properties for the report into a custom object for easy manipulation.  I create the custom object using a PowerShell technique called “splatting”.  (Read more about that v2 feature over on the “Hey, Scripting Guy!” blog here.)  This report treats the multi-value attributes as separate rows so that we get a one-to-one list of OldSID/NewSID.  Finally I dump the SID mapping data into two separate CSV files for your convenience:  a detailed report and an ADMT SID mapping file.  For the ADMT mapping file we have to do a quick cleanup of the quotes, because the ADMT doesn’t like those in the input file.

<#----------------------------------------------------------------------------- Get-SIDHistory.ps1 Ashley McGlone, Microsoft PFE http://blogs.technet.com/b/ashleymcglone April, 2011 This script queries Active Directory for SID history in order to build a SID mapping file for use with the ADMT to do security translation, especially in situations where the ADMT database has been lost. In addition to the mapping file it also generates a full SID history report for viewing in Excel. This script must be run from a machine that has the Active Directory module for PowerShell installed (ie. Windows 7 or Windows Server 2008 R2 with RSAT). You must also have either a Windows Server 2008 R2 domain controller, or an older domain controller with the Active Directory Management Gateway Service (AD Web Service) installed. For more information on ADMGS see: http://blogs.technet.com/b/ashleymcglone/archive/2011/03/17/step-by-step-how-to-use-active-directory-powershell-cmdlets-against-2003-domain-controllers.aspx -----------------------------------------------------------------------------#>Import-ModuleActiveDirectory#Create a blank array to hold our SID Map data$arySIDMap=@()#Query SID history, current SID, and related fields from AD$ADQuery=Get-ADObject-LDAPFilter"(sIDHistory=*)"-PropertyobjectClass,` samAccountName,DisplayName,objectSid,sIDHistory,distinguishedname#Loop through each AD object returnedForEach($rowin$ADQuery){#SID history is a multi-valued attribute, so loop through each entry.ForEach($SIDin$row.sIDHistory){#Arrange the data we want into a custom object$objTemp=New-ObjectPSObject-Property@{objectClass=$row.objectClass;OldSID=$SID;NewSID=$row.objectSID;samAccountName=$row.samAccountName;DisplayName=$row.displayName;DistinguishedName=$row.distinguishedName}#Use array addition to add the new object to our SID Map array$arySIDMap+=$objTemp}}#Create a full SID History report file for reference in Excel$arySIDMap|Export-CSV.\SID_History_Report.csv-NoTypeInformation#Create a SID Mapping text file for use with ADMT$arySIDMap|Select-ObjectOldSID,NewSID|Export-CSV.\SIDMapping1.csv-NoTypeInformation#Peel out the quotes from the mapping file, because ADMT does not like those.Get-Content.\SIDMapping1.csv|ForEach-Object{$_.Replace("`"","")}|Set-Content.\SIDMapping.csvRemove-Item.\SIDMapping1.csv"Output complete:""SID_History_Report.csv - full SID History report for reference in Excel""SIDMapping.csv - file for use with ADMT to do security translation"# ><>

Conclusion

I hope this long-winded explanation resolves your SID history issues and informs your future AD migration projects.  Who knew that PowerShell was a weight-loss product for elephants?!  Now your tokens can fit inside the VW.

Links

• Active Directory Migration Tool version 3.2
  • http://www.microsoft.com/downloads/en/details.aspx?FamilyID=20C0DB45-DB16-4D10-99F2-539B7277CCDB
• Active Directory Migration Tool (ADMT) Guide: Migrating and Restructuring Active Directory Domains
  • http://www.microsoft.com/downloads/en/details.aspx?FamilyID=6D710919-1BA5-41CA-B2F3-C11BCB4857AF
• KB327825 - New resolution for problems with Kerberos authentication when users belong to many groups (Good explanation of calculating token size)
  • http://support.microsoft.com/kb/327825
• Large Security Tokens and Kernel Memory Exhaustion
  • http://blogs.technet.com/b/exchange/archive/2005/12/28/416551.aspx
• Kerberos Authentication Problem with Active Directory (Excellent explanation of token size)
  • http://blogs.technet.com/b/surama/archive/2009/04/06/kerberos-authentication-problem-with-active-directory.aspx
• Step-by-Step: How to use Active Directory PowerShell cmdlets against 2003 domain controllers
  • http://blogs.technet.com/b/ashleymcglone/archive/2011/03/17/step-by-step-how-to-use-active-directory-powershell-cmdlets-against-2003-domain-controllers.aspx
• TechNet:  Selectively purge SID history with PowerShell
  • http://technet.microsoft.com/en-us/library/powershell_remove_sid_history(WS.10).aspx

Wouldn’t you know it.  The very same day I posted the SID history script I found a shorter way to do it.  I'm calling a "do over".

I love the PowerShell.com email tip-of-the-day.  On that particular day the tip was how to use the ExpandProperty switch on the Select-Object cmdlet.  This was new for me, and exactly what I needed.  Let me explain.

When you display a list of object properties at the PowerShell console sometimes one of those properties may be what we call “multi-valued”.  That means the property contains multiple entries in an array or object collection.  A good example of this is Get-PSProvider.  The Drives column usually returns multiple values in a single row, like the drives for the Registry {HKLM, HKCU}.  In order to expand those multiple values into single rows we pipe the output to Select-Object -ExpandProperty like this:

Get-PSProvider | Select-Object -ExpandProperty Drives

This is very handy for Active Directory, because several user attributes are multi-value properties (ie. postalAddress, sIDHistory, description, userCertificate, etc.).  By expanding these properties in our query we can get one row for each SID history value that is present, making the result set more friendly for exporting and reporting.  This eliminates the need to loop through the values and populate a custom object as I did in the first version of the script.

That brings us to the revised SID history script.  Because we’re going to format the output into two separate files we’ll do the AD query only once and capture it into a variable.  This is obviously faster than doing the query twice and piping it out twice.  The one-liner takes the query, expands the SID history property, selects and names the columns, and then exports it to CSV.  I have put carriage returns after each pipe simply to make it more readable.

I love PowerShell!

<#-----------------------------------------------------------------------------
Get-SIDHistory.ps1
Ashley McGlone, Microsoft PFE
http://blogs.technet.com/b/ashleymcglone
May, 2011

This script queries Active Directory for SID history in order to build a SID
mapping file for use with the ADMT to do security translation, especially in
situations where the ADMT database has been lost.  In addition to the mapping
file it also generates a full SID history report for viewing in Excel.

This script must be run from a machine that has the Active Directory module
for PowerShell installed (ie. Windows 7 with RSAT or Windows Server 2008 R2).
You must also have either a Windows Server 2008 R2 domain controller, or an
older domain controller with the Active Directory Management Gateway Service
(AD Web Service) installed.  For more information on ADWS see:
http://blogs.technet.com/b/ashleymcglone/archive/2011/03/17/step-by-step-how-to-use-active-directory-powershell-cmdlets-against-2003-domain-controllers.aspx
-----------------------------------------------------------------------------#>Import-ModuleActiveDirectory#Query SID history, current SID, and related fields from AD$ADQuery=Get-ADObject-LDAPFilter"(sIDHistory=*)"-PropertyobjectClass,`samAccountName,DisplayName,objectSid,sIDHistory,distinguishedname#Create a full SID History report file for reference in Excel$ADQuery|Select-Object*-ExpandPropertysIDHistory|Select-ObjectobjectClass,@{name="OldSID";expression={$_.Value}},`@{name="NewSID";expression={$_.objectSID}},samAccountName,DisplayName,`DistinguishedName|Export-CSVSIDReport.csv-NoTypeInformation#Create a SID Mapping text file for use with ADMTGet-ADObject-LDAPFilter"(sIDHistory=*)"-PropertyobjectSID,sIDHistory|Select-Object*-ExpandPropertysIDHistory|Select-Object@{name="OldSID";expression={$_.Value}},`@{name="NewSID";expression={$_.objectSID}}|Export-CSVSIDMap0.csv-NoTypeInformation#Peel out the quotes from the mapping file, because ADMT does not like those.Get-Content.\SIDMap0.csv|ForEach-Object{$_.Replace("`"","")}|Set-Content.\SIDMap.csvRemove-Item.\SIDMap0.csv"Output complete:""SIDReport.csv  - full SID History report for reference in Excel""SIDMap.csv     - file for use with ADMT to do security translation"#   ><>

More bad Chuck Norris one liners:

1. You don't get access unless Chuck Norris says you get access.
2. The Chuck Norris array starts and ends on 1, because Chuck Norris is never a zero.
3. Chuck Norris' PowerShell console NEVER shows red text.

This is part two in a series on translating SIDs in ACLs.  Start by reading Part One if you haven't done so already.  Today we're going to unleash a Chuck Norris round house kick on old SIDs stinking up our file servers.  It's time to take out the trash.

The Scenario

In my former post on SID history I explained…

Tell me if this scenario has ever happened at your company. You are doing an AD migration with the ADMT (or a similar product). You migrated the users and groups. The project schedule was falling behind. You take a shortcut with the member servers. Instead of using the ADMT security translation process you just rejoin the servers to the new domain, leaving all of the old ACLs and SIDs untouched. You leave SID history in place and move on to the next big project and deadline.

The Requirements

We need a solution that will

• Scan a file server
• Find all of the NTFS permissions (ACLs)
• Find the SIDs in those permissions (ACEs)
• Compare each ACE SID to our SID history mapping file to see if it needs swapped
• Swap the old SID with the new SID

An Added Twist… The NAS

The ADMT (Active Directory Migration Tool) is the ideal solution, well almost.  An added complexity here is that many companies migrate Windows file servers to a NAS (network attached storage).  The ADMT does security translation by deploying an agent that runs on Windows servers and translates all of the SIDs locally.  Since a NAS is not a Windows server, the ADMT is powerless to translate the SIDs stored there.  The security translation agent cannot run on a NAS.  I know what you're thinking, though.  "From a Windows server just map the NAS as a drive or mount it under a local folder.  Then the ADMT will see it."  That's what I thought, too.  We're both wrong.  After rereading the ADMT guide, trying a few work-arounds, and confirming this internally I have realized that the ADMT simply cannot work with a NAS.  (If you've done it I would love to hear your story.)

Then I turned to a host of utilities that have been around for years:  SIDWALK, SHOWACCS, SUBINACL, and ICACLS.  After hours studying and tinkering with these EXEs I was not able to find a satisfactory fit for our requirements.  Although they support SID translation and some form of an input file each tool was limited in some way or simply too out-dated to be of use.  Now please understand these are still amazing utilities, but for our specific scenario I could not get them to work.

Finally I considered the NAS vendor's ACL migration utility.  Most storage vendors have a suite of file migration utilities, including SID translation.  I found the tools in this case to be simplified and not scalable.  We need more POWER!

Enter PowerShell (aka Chuck Norris)

Our script today will help find and translate those old NTFS SIDs lingering behind on file server ACLs.

Basically the script traverses an NTFS path (local or UNC) and swaps any ACE SIDs that it finds in the SID mapping file.  At the end it outputs a text file verbose log and a CSV with all folders and any SID mappings identified.  Use the WhatIf switch to run in audit mode and check for SID history without making any changes.  In the interest of efficiency we only scan folder permissions; you can modify the script quickly to do both files and folders if desired.  We also ignore owner information in the ACL.

The Round House Kick

The Big O Notation to check every SID in every ACL against every row of the SID mapping file would be quite large.  (This is the problem with some of the EXEs we mentioned earlier.)  The beauty of PowerShell is that we can do the SID mapping quickly with a hash table, and that is the heart of the magic.

First, we read the mapping file into a hash table:

$SIDMapHash=@{}Import-CSV$MapFile|ForEach-Object{$SIDMapHash.Add($_.OldSID,$_.NewSID)}

The key If statement buried in the heart of our SID searching loop simply does a Contains method against the hash table:

If ($SIDMapHash.Contains($ACLEntrySID)) {#We have a winner}

This lightning-fast technique saves us from traversing the entire mapping file each time we search for a SID match.  Each time we find a match we do the old switcheroo with the old SID and new SID in the ACE.

$SDDLSplit[$i]=$SDDLSplit[$i].Replace($ACLEntrySID,$SIDMapHash.($ACLEntrySID))

When we finish swapping SIDs in the ACEs we write back the change with Set-ACL.

$acl.SetSecurityDescriptorSddlForm($NewSDDL)Set-Acl-Path$acl.path-ACLObject$acl

See the full code in the attached zip file, including sample output.  There is also a fancy progress bar as we scan the folders.  Think of the progress dots as Chuck Norris' foot flying to the target.  I finally followed The Scripting Guy's advice and took the time to add all the bells and whistles... comment-based help, advanced functions, parameter validation, etc.  With each new post I'll build this function library into a full dojo of karate moves.

Explicit Instructions

As always you should test scripts like this in a lab or on a small subset of data until you are comfortable with what it will do in production.  I have also provided a couple backup options in these steps:

1. Make sure you have a full backup from the night before.
2. Notify your backup administrator that you could potentially trigger a significant list of file backups if your software treats ACL changes as backup triggers.
3. For the PowerShell console use Run As with sufficient credentials to edit security on all subfolders in the path specified.
4. Use ICACLS to save a permissions template so that it can be reapplied in the event of an issue.
5. Dot-source the script file (dot space dot backslash script file):
      PS C:\>. .\SIDHistory.ps1
6. For instructions use Get-Help -Full to read about each function:
      PS C:\>Get-Help Export-SIDMapping -Full
      PS C:\>Get-Help Convert-SIDHistoryNTFS -Full
7. Create a SID mapping file:
      PS C:\>Export-SIDMapping
8. Review the report files, proceed with WhatIf first.
9. PS C:\>Convert-SIDHistoryNTFS \\server\share\path -WhatIf
10. Review the report files, decide to remove WhatIf and commit the changes.
11. PS C:\>Convert-SIDHistoryNTFS \\server\share\path
12. Review the report files.
13. Have your users test file access and login script drive mappings to make sure everyone is still happy.

Coming Up Next

Today's function library is not a full replacement for SIDWALK or ADMT.  It just does NTFS permissions for wherever you point it.  Next time we'll look at how to get a list of Domain SIDs in your environment so that you can interpret where SID history is coming from.  Now go unleash your Norris moves on the NAS.

For More Information

• Get-Help about_Hash_Tables
• Get-Help about_Scripts     (more on dot sourcing)
• Get-Help about_Functions
• Get-Help about_Functions_Advanced
• Get-Help about_Functions_Advanced_Parameters
• Get-Help about_Comment_Based_Help

 PS... I'm running out of Chuck Norris lines.  Add yours in the comments below, and I'll put them in the next post.

This weekend I'll be celebrating Christmas in the US.  Today Santa brought me a 1500VA 865W UPS for my home lab server.  Now I don't have to worry about those winter power hiccups rebooting all of my HyperV guests.  Best of all the scripting elves will be able to write presents without interruption.

So in PowerShell style here's a little Christmas gift for you.

Introduction

This post is the fifth in the "SID Walker, Texas Ranger" series on SID history remediation with PowerShell.  Today we're wrapping up with a handy summary of each post in the series. We will also take the function library we've been using and upgrade it to a PowerShell module. Then we'll walk through the entire SID history remediation process using the provided cmdlets in this module.

The Story So Far

Those of you who follow my blog know that I have been stuck on this theme of SID history for several months now.  Why?  Because I see this quite frequently with customers, and I want to offer some practical guidance on dealing with it.  Here is a summary of the blog series that brought us to today's module:

1. Using PowerShell to resolve Token Size issues caused by SID history
   Prior to starting the module development this post explained the background of token size issues as related to SID history.  I provided the basic SID history query that we use to produce the report and some great links for more information on token size.
2. Do Over: SID History One-Liner
   As a follow up to the Token Size post I re-wrote the SID history report query as a one-liner.
3. PowerShell: SID Walker, Texas Ranger (Part 1)
   This time we looked at Get-ACL and parsing SDDL strings, a warm up for the next post.
4. PowerShell: SID Walker, Texas Ranger (Part 2)
   Next I wrote a function to swap SID history entries in ACLs/ACEs.  This compensates for a gap in the ADMT, because it cannot migrate SID history for file shares hosted on a NAS.
5. PowerShell: SID Walker, Texas Ranger (Part 3): Exporting Domain SIDs and Trusts
   Looking at raw SIDs in a report is not very friendly, so I wrote a function that translates domain SIDs into domain names.  This makes the SID history report more meaningful when you can see the name of the domain from whence they came.  Enumerating all forest trusts and their domain SIDs required using some .NET ninja skills.
6. How To Remove SID History With PowerShell
   To round out the functions I provided Get-SIDHistory and Remove-SIDHistory, emphasizing that this is the LAST step in the process.  I leveraged the previous domain SID function to even give us the ability to remove SID history selectively by old domain name.

I suggest that you go back and read all of the articles linked above. They will give you much more insight into the SID history cleanup process and the nuances of the provided functions. Then skim through the ADMT Guide to get familiar with the big picture.

All of these functions are now wrapped up in the module provided in today's blog post.

Installing the Module

If you've never installed a module there really isn't much to it.  Here's what you do:

1. Create the module folder (adjust Documents path if necessary):
   New-Item -Type Directory -path "$home\Documents\WindowsPowerShell\Modules\SIDHistory"
2. Download the attached ZIP file at the bottom of this article.
3. Unzip the contents into this path:
   C:\Users\<username>\Documents\WindowsPowerShell\Modules\SIDHistory\
4. Fire up the PowerShell console or ISE.
5. Import-Module ActiveDirectory (This is a prerequisite.)
6. Import-Module SIDHistory

Now you can use Get-Command and Get-Help to unwrap the present and see what's inside:

• Get-Command -Module SIDHistory
• Get-Help Get-SIDHistory -Full
• Get-Help Get-SIDHistory -Online
• Get-Command -Module SIDHistory | Get-Help -Full | More

You can use Get-Help -Full for each of the included functions to find syntax and descriptions.

Using the Module

The outline below will guide you through the process of using the functions to help remediate SID history.  Run them in this order.

• Start up:
• Import-Module ActiveDirectory
• Import-Module SIDHistory
• Get the SID history report:
• Export-DomainSIDs
• Export-SIDMapping
• Update-SIDMapping
• Open the SIDReportUpdated.csv file in Excel to see all of the SID history in your environment.
• Keep an archive copy of these output files for documentation at the end of the project.
• Use the ADMT for server migration:
• Use the SIDMap.csv file with the ADMT to migrate servers with SID history.  This file recovers your OldSID/NewSID data from former migrations so that you can finish security translation on servers.
• NAS permission migration:
• If you have NAS-based file shares, migrate SID history of NTFS shares this way:
• Run with -WhatIf the first time to see if there is any SID history to translate.
• Convert-SIDHistoryNTFS \\server\share\path –WhatIf
• Review the report files.  Run again without -WhatIf to actually update the ACLs.
• Convert-SIDHistoryNTFS \\server\share\path
• Review the report files.
• Confirm share file access with affected users and groups.
• Remove the SID history:
• Confirm that you have good backups of Active Directory system state on two DCs in every domain.  You should always have a backout plan in case you missed some SID history remediation.
• Once SID history remediation is verified on all servers you can begin removing SID history in phases.  First, use Get-SIDHistory to target the removal population with a specific query.  Second, pipe the output to Remove-SIDHistory.  Here are some examples:
• Get-SIDHistory –MemberOf AccountingDept
• Get-SIDHistory –MemberOf AccountingDept | Remove-SIDHistory
• Get-SIDHistory –DomainName alpineskihouse.com
• Get-SIDHistory –DomainName alpineskihouse.com | Remove-SIDHistory
• See the help for extensive filtering capabilities of Get-SIDHistory.
• Check your work:
• Make an archive copy of your first SIDReportUpdated.csv and SIDMap.csv files.
• Export-SIDMapping
• Use the SIDReport.csv file as an audit to see where SID history remains.
• Repeat the migration and removal processes until this report comes back empty.
• Remediation (ie. Damage Control):
• Assuming that your security translation was thorough then you should not see any issues.
• If the help desk only gets a couple calls:
• Manually clean up the permissions on a case-by-case basis.
• If the help desk gets hammered with calls, then you have a couple options:
• Identify the scope of impact and remediate those servers by doing more security translation.  You may have missed a few.
• Do an AD authoritative restore to recover the SID history of impacted users.

The functions provided in this module will give you added visibility into the status of your SID history throughout the process and an easy way to target removal in the final phase.

Conclusion

This SID history project has been a lot of fun, and I'm sure there's more we could do with it.  I have a few ideas of my own, but I would like to hear your feedback.  What challenges have you encountered with SID history remediation?  Where do you think PowerShell could help?  Leave a comment below and let me know.

Additional Reading

• Get-Help about_Modules
• Active Directory Migration Tool (ADMT) Guide: Migrating and Restructuring Active Directory Domains

TechReady

At Microsoft we have an internal event called TechReady that happens twice each year.  It is similar to our public TechEd event.  Approximately 5,000 Microsoft technical field people from all over the world gather to learn the details of the latest product releases and dialog with the product groups.  It is a fantastic event, and this July I'll be presenting a session called "Active Directory PowerShell Notes From the Field".  I'll be covering three case studies related to schema updates, DNS migrations, and Active Directory SID history.  I will conclude the session with a quick look at the future of DCPROMO, REPADMIN, and DNSCMD in Windows Server 2012, comparing the new PowerShell ways to do Active Directory administration in the next release.  If you are going to TechReady then look me up.

More SID History

I get emails frequently from folks who have read my series of articles on Active Directory SID history, so I'm guessing that is a good theme to continue.  Working with a customer recently I was able to enhance the functionality in the Active Directory SID history PowerShell module that I posted last December.  Today's post will publish and discuss those improvements.

Version 1.4
June, 2012

Functions added in this release:
Export-SIDHistoryShare
Merge-CSV

Functions modified in this release:
Convert-SIDHistoryNTFS
Export-SIDMapping
Update-SIDMapping
Get-SIDHistory

Fixes:
Removed Test-Path validation on NewReport parameter of Update-SIDMapping.
Added file validation for DomainFile parameter of Get-SIDHistory.

It's All About The Customer

Imagine if…

• your company acquired an average of 12 other companies each year.
• your forest had over 35 domains (consolidated from 50+).
• your forest had over 80 trusts to other domains you still need to migrate.
• your forest had over 172,000 instances of SID history.
• you lived in the ADMT from sun up to sun down.
• token bloat plagued your support engineers with mystery troubleshooting.

Obviously I can't make this stuff up.  Those points describe a customer who recently invited me to help them develop a process to identify and remediate SID history in their forest.  This also gave them a process going forward that would help them do future domain migrations in a way that would minimize the impact of SID history.

Enter PowerShell

I knew the SID history PowerShell module could help them, but I also knew that some improvements would be necessary to help them scale to the size of data collection they would need.  Here's what I did:

• I added a function called Export-SIDHistoryShare that will collect SID history information from share permissions.  (Note that it will not convert SID history as the NTFS function does.)
• I modified the NTFS SID translation report CSV generated by Convert-SIDHistoryNTFS so that the columns would match the output from the new share CSV output.  This involved adding ACLType and DateTimeStamp.
• Both NTFS and Share reports now include a date time stamp so that the CSV data can be imported into a larger database multiple times.  Different scans of the same data will be distinguished by the date time stamp.
• I modified the Export-SIDMapping and Update-SIDMapping to include a date time stamp field as well for the same reasons.
• I created a new function called Merge-CSV that will take all of the NTFS and Share SID history reports and roll them into a single, large CSV file for importing to a database (like Microsoft Access or Microsoft SQL).
• This is huge!  I created a quick proof-of-concept Access database that would relate the forest object SID history data against the SID history discovered in shares and NTFS permissions on servers.  This is super powerful for analyzing the impact of SID history.  Now you can produce reports that tell you:
  • which former domain has the most SID history
  • which groups with SID history are most used on your file servers
  • which file servers need ACL translation with the ADMT
  • where to target your remediation efforts for the largest impact
  • where to begin the cleanup with the users and groups whose SID history is not in use
  • etc.
• I also resolved one issue with the NewReport parameter of the Update-SIDMapping function and resolved one issue with the DomainFile parameter of the Get-SIDHistory function. (Special thanks to Andrew Hill and Greg Jaworski for their feedback.)

These changes added the scalability and flexibility for the customer to begin inventorying hundreds for servers for SID history and manage all the data with a full database they would implement later on their own.  I love the simplicity of CSV in PowerShell!

The New Process

My last post on SID history has the overall steps for installing the module and completing the SID history remediation so I will not repeat that content here.  But I do want to list out the steps for you to automate the NTFS and share SID history data collection:

• Inventory SID history in share and NTFS ACLs:
  • Run these commands against the servers and share paths where you want to check for
    SID history:
    • Convert-SIDHistoryNTFS \\servername\share\path –WhatIf
    • Export-SIDHistoryShare servername
    • NOTE: You will need to run this under an administrative account that has permissions to view all of the recursive subfolders.
  • Repeat these steps for all servers where you want to collect data.  See the example code below for another way to automate mass data collection.
  • Now merge all of the output into a single CSV file for analysis:
    • Put all of the Share and NTFS CSV output files into a new working folder.
    • Merge-CSV –Path "C:\Temp\WorkingFolder"
    • This creates a merged CSV called ACL_SID_History_xxxxxxxxxxxxxxxx.csv.
• Copy these two files to the folder with the provided Access database:
  • SIDReportUpdated.csv
  • ACL_SID_History_xxxxxxxxxxxxxxxx.csv
• Rename the ACL_SID_History_xxxxxxxxxxxxxxxx.csv file to ACL_SID_History.csv.
• Open the Access database.
• In order for the linked tables in Access to see these new files you must repair the links and point them to the files in your working folder.
  • Right click the table ACL_SID_History on the left.
  • Choose “Linked Table Manager”.
  • Check the box at the bottom “Always prompt for new location”.
  • Check the box beside each of the two tables.
  • Click OK.
  • Browse to the each of the two CSV files that you just copied into the database folder. Pay special attention to the title bar of the File browser dialog box. Make sure that you choose the file that matches the name in the title bar.
  • Close the Linked Table Manager dialog box.
• Now you can double click any of the example queries on the left to see the data analysis.  You can also create your own queries and custom reports.

Automating the Automation

It would be quite time consuming to run the NTFS and share scan commands one at a time against all of your servers.  Instead try these handy PowerShell routines to make the data collection go faster.  As these processes usually take hours to scan large file shares it would be a good idea to let them run over a night or weekend.

# Sample code to scan a group of Windows servers in a text file.$Servers=Get-Content".\servers.txt"ForEach($ComputerNamein$Servers){Export-SIDHistoryShare$ComputerName# Grab only file shares that were manually created$Shares=Get-WmiObjectWin32_Share-ComputerName$ComputerName-Filter"Type=0"ForEach($Sharein$Shares){Convert-SIDHistoryNTFS"\\$ComputerName\$($Share.Name)"-WhatIf}}# Sample code to scan a list of share paths in a text file.# This can work for Windows or NAS servers.$Shares=Get-Content".\shares.txt"ForEach($Sharein$Shares){Convert-SIDHistoryNTFS$Share-WhatIf}

Where can I get all this PowerShell goodness?

I have updated this code at the TechNet Script Gallery.  Attached to that entry you'll find the following in a single compressed file:

• Updated PowerShell module for AD SID history
• Sample output data files
• Sample Access database (You must update the linked tables to point to the CSV files.  See instructions above.)
• Sample script file for using the module

Extract all of the files into a working folder.  Install the module.  The usual disclaimers apply: this is sample code for use at your own risk.  Enjoy!

But Wait!  There's More…

If you would like me or another Microsoft PFE to visit your company and help you with the ideas discussed on this blog, please contact your Premier Technical Account Manager (TAM).  We would love to come see you.

Follow me on Twitter: @GoateePFE.

To see all of the articles in this series visit http://aka.ms/SIDHistory.

I would like to thank everyone who has been using the Active Directory SIDHistory PowerShell module and sending me feedback.  Your input helps guide future releases like the one I am publishing today.

I’ve been sitting on some updates for a while, because I prefer to release code that has been field-tested.  I also wanted to time this release with the upcoming PowerShell Deep Dives book where I have a chapter discussing the origins of this module.  The last update was version 1.4 in June of 2012.  This is update 1.5 in July of 2013.

Summary of Changes

I am excited to announce the following key improvements in this release:

• SID history updates in ACLs can be added instead of replaced.
• Create SID map files for security translation without needing SID history.
• Track down old domains after their trusts have been removed.
• Get error logging for file server paths that fail the ACL scan.
• Automate SID history data collection across many servers and shares.

Change Details

This release includes some significant changes and additions, which I have highlighted below. Here is a list of the functions in this release:

PS C:\> Get-Command -Module SIDHistory

CommandType Name                     ModuleName
----------- ----                     ----------
Function    Convert-SIDHistoryNTFS   SIDHistory
Function    Export-DomainSIDs        SIDHistory
Function    Export-SIDHistoryShare   SIDHistory
Function    Export-SIDMapping        SIDHistory
Function    Export-SIDMappingCustom  SIDHistory
Function    Get-ADObjectADSI         SIDHistory
Function    Get-DomainSIDWordCount   SIDHistory
Function    Get-SIDHistory           SIDHistory
Function    Get-SIDHistoryDuplicates SIDHistory
Function    Merge-CSV                SIDHistory
Function    Remove-SIDHistory        SIDHistory
Function    Search-SIDHistoryACL     SIDHistory
Function    Update-SIDMapping        SIDHistory
Function    Watch-Job                SIDHistory

Due to the large number of changes I am not going to include code samples for each function in this article.  Please use Get-Help -Full to see complete details and examples for each module member.

PS C:\> Get-Help Convert-SIDHistoryNTFS -Full

PS C:\> Get-Help Export-SIDMappingCustom -Full

PS C:\> Get-Help Get-DomainSIDWordCount -Full

Etc...

Functions Added

Export-SIDMappingCustom

• I had a customer who was not able to use the Active Directory Migration Tool for a domain migration.  The newly acquired subsidiary was not allowed to create a trust for compliance reasons.  Obviously we need a trust to do a migration.  Or do we?
• To work around the situation I wrote a function that will map SIDs between accounts in two different domains where there is no SID history.  The trick is having a common, unique attribute like samAccountName, employeeID, employeeNumber, mail, etc.
• The customer manually exported the accounts from the old domain and imported them into the new domain.  Next they made sure that their HR system populated the same unique Employee ID in the new domain.  Now they had a common, unique key between the two domains.
• In the old 2003 domain we used an ADSI script to create a file containing the objectSID and employeeID attributes.  (See Get-ADObjectADSI described below.)  This is all we needed to create our mapping file.  We copied this file across international borders to the local new domain.
• Now we can use the new Export-SIDMappingCustom function to create a mapping file between the exported SID data from the old domain and the live accounts in the new domain.  This is only one possible scenario with the function.  It can use any combination of live connection or export file from the old and new domains.
• In this particular case the customer only wanted to migrate the file server from the old domain.  With this new SID mapping file they were able to run Convert-SIDHistoryNTFS against the old server to re-ACL the resources.  This is POWERFUL.  Essentially, you can do a NAS ACL migration without SID history in place.
• This works for both users and groups.  At another customer they wanted to migrate several domain local groups.  The groups had been recreated with the same name in the new domain without SID history.  The Export-SIDMappingCustom function was able to create a mapping file by matching the group names between both old and new live domains and then putting both SIDs into a mapping file.

Get-SIDHistoryDuplicates

• There are processes to resolve duplicate SIDs on accounts but not for duplicate SID history.
• This should never never never happen.  But it happened for one customer, the same customer where I discovered 100 unique old domains.  We’re not sure how the duplicate SID history entries got there, because there were years of migration history with a variety of migration tools.
• Duplicates in SID history are a violation of everything we know to be good and true.  There can only be one.  If duplicates are found, then it would invalidate the SID mapping process.
• To find these I wrote this function to generate a report.  Once identified, the customer chose to simply delete the accounts with the duplicates, because they were old empty groups.

Search-SIDHistoryACL

• In this module two main functions do all of the work to discover SID history on shares and files:  Convert-SIDHistoryNTFS and Export-SIDHistoryShare.  In order to fully document SID history on your file servers you need to call each of these functions for each share on each server.  That’s a lot of calls.
• This is a “meta-function” that calls these two functions for a batch of servers or shares listed in either an OU or a text file.  Using PowerShell background jobs it scans them all in parallel to reduce the discovery time required.
• My only caution is that there is no throttling built in.  Run these in small batches of servers until you see how the performance works out with your hardware.
• Use the following Watch-Job function to manage these background jobs.

Watch-Job

• This function conveniently manages receiving the output from the jobs spun up by Search-SIDHistoryACL.  It displays a PowerShell progress bar indicating how many jobs are still running.  Once they are all complete it will receive the output into a consolidated job log and report on each job’s start and end times.  All of the ACL output files are rendered separately by their respective jobs.
• Optionally you can have the function clear the job queue with the -Remove switch.
• This is a utility function, and it could be modified easily for many other background job scenarios.

Get-DomainSIDWordCount

• The Export-DomainSIDs function crawls all trusts in the forest to identify domain names and domain SIDs.  When old trusts to migrated domains are gone you can no longer identify where SID history originated… until now.
• The function creates a list of word counts gleaned from popular string attributes found on accounts from each old domain SID identified in the forest.  This gives you clues about where the accounts in the unidentified SID history domain may have originated.  For example, employees migrated from the old ContosoPartner domain may have that company name string in their description, department, display name, or OU path.  By finding and counting the common strings across all migrated accounts you can usually identify the old domain.
• This process gives you the information needed to manually update the DomainSIDs.csv file.  After updating the file re-run the Update-SIDMapping function to add these old domain names to your master SID history report.
• Pipe the output to Export-CSV or Out-GridView for easy viewing.

Get-ADObjectADSI

• I wrote this utility function for a customer with legacy DCs where they were not going to install the AD web service.  They installed PowerShell, and we used this function to create the account listing for Export-SIDMappingCustom.  It uses ADSI to mimic Get-ADObject, but it was a point solution not intended for full parity with the AD module cmdlet.  This function is intentionally undocumented, because most environments will not need it.  Feel free to modify it to meet your needs.

Functions Modified

Export-SIDMapping

• I added description and whenCreated attributes to the SIDReport.CSV.  These properties help identify accounts in the list, especially old accounts that are likely stale.
• By mistake I was running the AD query twice in this function, so I corrected it to only run once.  Obviously this will greatly improve performance.

Update-SIDMapping

• Added description and whenCreated to SIDReportUpdated.CSV for same reasons listed above.

Convert-SIDHistoryNTFS

• Added -Add switch.  The previous functionality did an ACE replace only.  Now you have the option of doing a replace or an add.  This way you can co-exist with both old and new SIDs temporarily until you are ready to completely remove all old SIDs from the ACLs.  Running without the -Add switch gives you a replace, leaving the new SID and removing the old SID.
• As a result of the new -Add switch I had to modify the CSV output and log files to report the status of both Old SID and New SID in the ACLs.  The “Both” column will be true if both the old and new SIDs are present in the ACL.  The “Operation” column will now say “Add” or “Replace”.  Added new column for NewDomainSID.
• After more field testing I discovered that there were some issues scanning folders.  In prior releases you would see red text in the PowerShell console, but you didn’t know where the errors occurred. I added error logging so that you get a list of the reasons and the paths where the ACL scans fail down inside the folder tree.  Primarily I’ve found three issues:
• 1. Path too long.  I spent a lot of time investigating this issue.  The Get-ACL -Path parameter is limited to paths of 260 characters or less.  I found this TechNet Wiki article and this blog series, but none of the proposed work-arounds were applicable.  As a temporary work-around if you run into this problem you may be able to map a drive down farther in the folder structure of longer paths and run the scan again for that newly mapped path.  This is due to an underlying .NET limitation.  Unfortunately this error only gets logged when running in PowerShell v3.  PowerShell v2 will not report this error in a way that we can trap.
• 2. Invalid character in path.  I discovered an issue with Get-ACL that does not return an object when the path contains a ‘[‘ character.  After researching this on the Microsoft PowerShell Connect site I found that it is resolved in PowerShell v3 with the new -LiteralPath parameter.  For now I am logging the error until I update the module to use PowerShell v3 features.
• 3. Access denied.  Using this function you often find file shares where Administrator rights have been removed.  To resolve this either rescan the path in question with appropriate credentials or modify the permissions on the path.

Export-SIDHistoryShare

• Modified CSV columns to match the output from Convert-SIDHistoryNTFS.  This allows all results to be consolidated into a single CSV report.

Time-Saving Tip

When using Convert-SIDHistoryNTFS -WhatIf to scan servers and shares it can take hours or days depending on the size of the environment.  Sometimes you may only find a few SIDHistory entries that need to be remediated.  Rather than rescan the entire server or share to fix these ACLs you can target the SID history cleanup this way to save time:

PS C:\> Import-CSV ACL_SID_History.csv |
 Select-Object -Unique -ExpandProperty Folder |
 ForEach-Object {Convert-SIDHistoryNTFS $_}

In this example ACL_SID_History.csv is the NTFS permission scan output from Convert-SIDHistoryNTFS -WhatIf.  We simply get a unique list of folders and feed those into the conversion function.  Obviously this is much faster than rescanning every ACL of every folder in the share subtree.

What’s next?

I have a long list of pending enhancements for the module, but my top priority is to update it with PowerShell v3 features:

• Use -LiteralPath in place of -Path to avoid the ‘[‘ issue in Convert-SIDHistoryNTFS.
• Rewrite the Export-SIDHistoryShare function using the new SmbShare module.
• Create a function that will update share ACLs using the new SmbShare module.
• Create a -Server parameter for Get-SIDHistory and Export-SIDMapping to target a specific DC.
• Capture all output streams correctly for Watch-Job logging.

Please use the comment area below to tell me what features you would like to see improved or added in the module.  I would also appreciate any feedback for how this has helped your projects.

Download

You can download the updated Active Directory SIDHistory PowerShell module from the TechNet Script Gallery.

Update:  To see all articles in this series click here.

No matter how bad he wants to an elephant simply cannot drive a Volkswagen.  He just won’t fit in that space.  People can drive VWs, because that is who the VW was designed to accommodate.  There is nothing worse than frustrated elephants.  What does this have to do with PowerShell or Active Directory?  I’m glad you asked.

The Problem:  Token Size and SID History

When users become members of too many groups their access token grows so large that it no longer fits inside some of the default OS setting.  Users can see issues logging in, applying group policy, and authenticating to web servers.  There are several other good articles that explain this scenario, and they are linked at the end of this post.

Token size issues can usually be attributed to a combination of three scenarios:

1. SID history left over from AD migration projects
2. Poor group nesting strategy
3. Old groups that are never purged

Today we will address the SID history scenario with a PowerShell script that documents the extent of SID history in your environment and creates a SID mapping file for use with the ADMT to migrate resources to the new SIDs.

Tell me if this scenario has ever happened at your company.  You are doing an AD migration with the ADMT (or a similar product).  You migrated the users and groups.  The project schedule was falling behind.  You take a shortcut with the member servers.  Instead of using the ADMT security translation process you just rejoin the servers to the new domain, leaving all of the old ACLs and SIDs untouched.  You leave SID history in place and move on to the next big project and deadline.

Management is happy that you met the deadline, but little do they know that you created an entire zoo full of elephants whose VWs are starting to feel kind of tight.

This scenario is common, and I want to explain the risks here.

1. Down the road you are likely to encounter token size issues, because now all users and groups have at least two SIDs.
2. If you decide to clean up SID history in the future you have the potential of orphaning users from their data, because the old SIDs were still giving them access.
3. THE BIG ONE:  When the project is decommissioned and the ADMT is blown away you lose all of the OldSID/NewSID migration data needed to clean the ACL SIDs on your migrated resource servers.

Many companies that I work with find themselves in this place.  Users are calling the help desk with mysterious symptoms, and they are not sure what is causing the problem.  Often it is token size, and SID history is a significant component.  At this point you could purge the SID history, but then you could lose access to all of the migrated data with old SIDs in the ACLs.  The best way forward is to go back and finish the AD migration by doing security translation on all of the migrated resources.

The Solution

The following steps will rid your environment of SID history:

1. Identify all servers in the environment that were around during the AD migration(s).
2. Install the latest version of the ADMT on a member server in your domain (NOT A DC, follow the guide linked below).
3. Run the PowerShell script from this article to create a SID mapping file.
4. Run the ADMT security translation wizard against each of the old servers, and use the SID mapping file when prompted to retrieve objects for security translation.  View the ADMT log files to see where changes were made.  (I started to write a PowerShell script to do the security translation, and I got it working for file share permissions by editing SDDL strings.  That was pretty cool, but it would take too much time to rewrite all seven steps built into the ADMT wizard.)  The ADMT wizard will clean SIDs in the following locations:
   1. Files And Folders
   2. Local Groups
   3. Printers
   4. Registry
   5. Shares
   6. User Profiles
   7. User Rights
5. In phases purge SID history from users and groups, starting with a small test population and then going by department until it is entirely removed from the environment.  Use targeted LDAP query strings to feed the PowerShell script linked at the end of this article to do your purges.  DO NOT PURGE ALL SID HISTORY AT ONCE.  That could be a resume-generating event if you missed some resources in the re-migration.  Do it in smaller batches to be safe.  Rerun the script in this article to see where SID history remains.

Enter PowerShell.  Since the ADMT database is long gone, we have to rebuild the SID mapping between old and new SIDs.  Lucky for us this data is all present in AD, because you told the ADMT to keep it.  We just need to put it into a file that the ADMT wants, called the SID mapping file.  This file format is simply a CSV file where the first column contains the old SID and the second column contains either the new SID or the new user name in DOMAIN\USERNAME format.  The script in today’s post will do this for you.  This script is entirely safe for your environment, since it makes no changes and only reads data.

In addition to the mapping file the script will also generate a CSV report of all SID history in your domain.  It includes the following columns:  samAccountName, DisplayName, objectClass, OldSID, NewSID, DistinguishedName.  Use this report before the clean up to assess the scope of the issue.  Run it again after the clean up to make sure you got it all.

Another handy way to view SID history for a specific user is with NTDSUTIL.  You can use the command Group Membership Evaluation to view all of a user’s groups, and it will tell you which ones come from SID history.  This is another process to spot check SID history before and after the cleanup.

The Script

This script is rather simple really.  Initially I thought you could dump all SID history with a PowerShell one-liner.  And you may, but the trick is that the SID history attribute is a multi-value field.  This syntax will not work:

Get-ADObject -LDAPFilter "(sidHistory=*)" -Property sidHistory | Export-CSV SIDHistory.csv

This will get you a query of the objects, but the SID history column on every row will say “Microsoft.ActiveDirectory.Management.ADPropertyValueCollection”.

Author's note:  See the new-improved version of this script as a one-liner here.

To do this properly and make the report generation easy I used two loops.  The outside loop iterates through all of the objects with SID history, and the inside loop iterates through each of the multi-value SID history attribute values.  Then I capture the properties for the report into a custom object for easy manipulation.  I create the custom object using a PowerShell technique called “splatting”.  (Read more about that v2 feature over on the “Hey, Scripting Guy!” blog here.)  This report treats the multi-value attributes as separate rows so that we get a one-to-one list of OldSID/NewSID.  Finally I dump the SID mapping data into two separate CSV files for your convenience:  a detailed report and an ADMT SID mapping file.  For the ADMT mapping file we have to do a quick cleanup of the quotes, because the ADMT doesn’t like those in the input file.

<#-----------------------------------------------------------------------------
Get-SIDHistory.ps1
Ashley McGlone, Microsoft PFE
http://blogs.technet.com/b/ashleymcglone
April, 2011

This script queries Active Directory for SID history in order to build a SID
mapping file for use with the ADMT to do security translation, especially in
situations where the ADMT database has been lost. In addition to the mapping
file it also generates a full SID history report for viewing in Excel. This
script must be run from a machine that has the Active Directory module for
PowerShell installed (ie. Windows 7 or Windows Server 2008 R2 with RSAT).
You must also have either a Windows Server 2008 R2 domain controller, or an
older domain controller with the Active Directory Management Gateway Service
(AD Web Service) installed. For more information on ADMGS see:
http://aka.ms/ADPS2003
-----------------------------------------------------------------------------#>Import-ModuleActiveDirectory#Create a blank array to hold our SID Map data$arySIDMap=@()#Query SID history, current SID, and related fields from AD$ADQuery=Get-ADObject-LDAPFilter"(sIDHistory=*)"-PropertyobjectClass,` samAccountName,DisplayName,objectSid,sIDHistory,distinguishedname#Loop through each AD object returnedForEach($rowin$ADQuery){#SID history is a multi-valued attribute, so loop through each entry.ForEach($SIDin$row.sIDHistory){#Arrange the data we want into a custom object$objTemp=New-ObjectPSObject-Property@{objectClass=$row.objectClass;OldSID=$SID;NewSID=$row.objectSID;samAccountName=$row.samAccountName;DisplayName=$row.displayName;DistinguishedName=$row.distinguishedName}#Use array addition to add the new object to our SID Map array$arySIDMap+=$objTemp}}#Create a full SID History report file for reference in Excel$arySIDMap|Export-CSV.\SID_History_Report.csv-NoTypeInformation#Create a SID Mapping text file for use with ADMT$arySIDMap|Select-ObjectOldSID,NewSID|Export-CSV.\SIDMapping1.csv-NoTypeInformation#Peel out the quotes from the mapping file, because ADMT does not like those.Get-Content.\SIDMapping1.csv|ForEach-Object{$_.Replace("`"","")}|Set-Content.\SIDMapping.csvRemove-Item.\SIDMapping1.csv"Output complete:""SID_History_Report.csv - full SID History report for reference in Excel""SIDMapping.csv - file for use with ADMT to do security translation"# ><>

Conclusion

I hope this long-winded explanation resolves your SID history issues and informs your future AD migration projects.  Who knew that PowerShell was a weight-loss product for elephants?!  Now your tokens can fit inside the VW.

Links

• Active Directory Migration Tool version 3.2
  • http://www.microsoft.com/downloads/en/details.aspx?FamilyID=20C0DB45-DB16-4D10-99F2-539B7277CCDB
• Active Directory Migration Tool (ADMT) Guide: Migrating and Restructuring Active Directory Domains
  • http://www.microsoft.com/downloads/en/details.aspx?FamilyID=6D710919-1BA5-41CA-B2F3-C11BCB4857AF
• KB327825 - New resolution for problems with Kerberos authentication when users belong to many groups (Good explanation of calculating token size)
  • http://support.microsoft.com/kb/327825
• Large Security Tokens and Kernel Memory Exhaustion
  • http://blogs.technet.com/b/exchange/archive/2005/12/28/416551.aspx
• Kerberos Authentication Problem with Active Directory (Excellent explanation of token size)
  • http://blogs.technet.com/b/surama/archive/2009/04/06/kerberos-authentication-problem-with-active-directory.aspx
• Step-by-Step: How to use Active Directory PowerShell cmdlets against 2003 domain controllers
  • http://blogs.technet.com/b/ashleymcglone/archive/2011/03/17/step-by-step-how-to-use-active-directory-powershell-cmdlets-against-2003-domain-controllers.aspx
• TechNet:  Selectively purge SID history with PowerShell
  • http://technet.microsoft.com/en-us/library/powershell_remove_sid_history(WS.10).aspx

Wouldn’t you know it.  The very same day I posted the SID history script I found a shorter way to do it.  I'm calling a "do over".

I love the PowerShell.com email tip-of-the-day.  On that particular day the tip was how to use the ExpandProperty switch on the Select-Object cmdlet.  This was new for me, and exactly what I needed.  Let me explain.

When you display a list of object properties at the PowerShell console sometimes one of those properties may be what we call “multi-valued”.  That means the property contains multiple entries in an array or object collection.  A good example of this is Get-PSProvider.  The Drives column usually returns multiple values in a single row, like the drives for the Registry {HKLM, HKCU}.  In order to expand those multiple values into single rows we pipe the output to Select-Object -ExpandProperty like this:

Get-PSProvider | Select-Object -ExpandProperty Drives

This is very handy for Active Directory, because several user attributes are multi-value properties (ie. postalAddress, sIDHistory, description, userCertificate, etc.).  By expanding these properties in our query we can get one row for each SID history value that is present, making the result set more friendly for exporting and reporting.  This eliminates the need to loop through the values and populate a custom object as I did in the first version of the script.

That brings us to the revised SID history script.  Because we’re going to format the output into two separate files we’ll do the AD query only once and capture it into a variable.  This is obviously faster than doing the query twice and piping it out twice.  The one-liner takes the query, expands the SID history property, selects and names the columns, and then exports it to CSV.  I have put carriage returns after each pipe simply to make it more readable.

I love PowerShell!

<#-----------------------------------------------------------------------------
Get-SIDHistory.ps1
Ashley McGlone, Microsoft PFE
http://blogs.technet.com/b/ashleymcglone
May, 2011

This script queries Active Directory for SID history in order to build a SID
mapping file for use with the ADMT to do security translation, especially in
situations where the ADMT database has been lost.  In addition to the mapping
file it also generates a full SID history report for viewing in Excel.

This script must be run from a machine that has the Active Directory module
for PowerShell installed (ie. Windows 7 with RSAT or Windows Server 2008 R2).
You must also have either a Windows Server 2008 R2 domain controller, or an
older domain controller with the Active Directory Management Gateway Service
(AD Web Service) installed.  For more information on ADWS see:
http://blogs.technet.com/b/ashleymcglone/archive/2011/03/17/step-by-step-how-to-use-active-directory-powershell-cmdlets-against-2003-domain-controllers.aspx
-----------------------------------------------------------------------------#>Import-ModuleActiveDirectory#Query SID history, current SID, and related fields from AD$ADQuery=Get-ADObject-LDAPFilter"(sIDHistory=*)"-PropertyobjectClass,`samAccountName,DisplayName,objectSid,sIDHistory,distinguishedname#Create a full SID History report file for reference in Excel$ADQuery|Select-Object*-ExpandPropertysIDHistory|Select-ObjectobjectClass,@{name="OldSID";expression={$_.Value}},`@{name="NewSID";expression={$_.objectSID}},samAccountName,DisplayName,`DistinguishedName|Export-CSVSIDReport.csv-NoTypeInformation#Create a SID Mapping text file for use with ADMTGet-ADObject-LDAPFilter"(sIDHistory=*)"-PropertyobjectSID,sIDHistory|Select-Object*-ExpandPropertysIDHistory|Select-Object@{name="OldSID";expression={$_.Value}},`@{name="NewSID";expression={$_.objectSID}}|Export-CSVSIDMap0.csv-NoTypeInformation#Peel out the quotes from the mapping file, because ADMT does not like those.Get-Content.\SIDMap0.csv|ForEach-Object{$_.Replace("`"","")}|Set-Content.\SIDMap.csvRemove-Item.\SIDMap0.csv"Output complete:""SIDReport.csv  - full SID History report for reference in Excel""SIDMap.csv     - file for use with ADMT to do security translation"#   ><>

More bad Chuck Norris one liners:

1. You don't get access unless Chuck Norris says you get access.
2. The Chuck Norris array starts and ends on 1, because Chuck Norris is never a zero.
3. Chuck Norris' PowerShell console NEVER shows red text.

This is part two in a series on translating SIDs in ACLs.  Start by reading Part One if you haven't done so already.  Today we're going to unleash a Chuck Norris round house kick on old SIDs stinking up our file servers.  It's time to take out the trash.

The Scenario

In my former post on SID history I explained…

Tell me if this scenario has ever happened at your company. You are doing an AD migration with the ADMT (or a similar product). You migrated the users and groups. The project schedule was falling behind. You take a shortcut with the member servers. Instead of using the ADMT security translation process you just rejoin the servers to the new domain, leaving all of the old ACLs and SIDs untouched. You leave SID history in place and move on to the next big project and deadline.

The Requirements

We need a solution that will

• Scan a file server
• Find all of the NTFS permissions (ACLs)
• Find the SIDs in those permissions (ACEs)
• Compare each ACE SID to our SID history mapping file to see if it needs swapped
• Swap the old SID with the new SID

An Added Twist… The NAS

The ADMT (Active Directory Migration Tool) is the ideal solution, well almost.  An added complexity here is that many companies migrate Windows file servers to a NAS (network attached storage).  The ADMT does security translation by deploying an agent that runs on Windows servers and translates all of the SIDs locally.  Since a NAS is not a Windows server, the ADMT is powerless to translate the SIDs stored there.  The security translation agent cannot run on a NAS.  I know what you're thinking, though.  "From a Windows server just map the NAS as a drive or mount it under a local folder.  Then the ADMT will see it."  That's what I thought, too.  We're both wrong.  After rereading the ADMT guide, trying a few work-arounds, and confirming this internally I have realized that the ADMT simply cannot work with a NAS.  (If you've done it I would love to hear your story.)

Then I turned to a host of utilities that have been around for years:  SIDWALK, SHOWACCS, SUBINACL, and ICACLS.  After hours studying and tinkering with these EXEs I was not able to find a satisfactory fit for our requirements.  Although they support SID translation and some form of an input file each tool was limited in some way or simply too out-dated to be of use.  Now please understand these are still amazing utilities, but for our specific scenario I could not get them to work.

Finally I considered the NAS vendor's ACL migration utility.  Most storage vendors have a suite of file migration utilities, including SID translation.  I found the tools in this case to be simplified and not scalable.  We need more POWER!

Enter PowerShell (aka Chuck Norris)

Our script today will help find and translate those old NTFS SIDs lingering behind on file server ACLs.

Basically the script traverses an NTFS path (local or UNC) and swaps any ACE SIDs that it finds in the SID mapping file.  At the end it outputs a text file verbose log and a CSV with all folders and any SID mappings identified.  Use the WhatIf switch to run in audit mode and check for SID history without making any changes.  In the interest of efficiency we only scan folder permissions; you can modify the script quickly to do both files and folders if desired.  We also ignore owner information in the ACL.

The Round House Kick

The Big O Notation to check every SID in every ACL against every row of the SID mapping file would be quite large.  (This is the problem with some of the EXEs we mentioned earlier.)  The beauty of PowerShell is that we can do the SID mapping quickly with a hash table, and that is the heart of the magic.

First, we read the mapping file into a hash table:

$SIDMapHash=@{}Import-CSV$MapFile|ForEach-Object{$SIDMapHash.Add($_.OldSID,$_.NewSID)}

The key If statement buried in the heart of our SID searching loop simply does a Contains method against the hash table:

If ($SIDMapHash.Contains($ACLEntrySID)) {#We have a winner}

This lightning-fast technique saves us from traversing the entire mapping file each time we search for a SID match.  Each time we find a match we do the old switcheroo with the old SID and new SID in the ACE.

$SDDLSplit[$i]=$SDDLSplit[$i].Replace($ACLEntrySID,$SIDMapHash.($ACLEntrySID))

When we finish swapping SIDs in the ACEs we write back the change with Set-ACL.

$acl.SetSecurityDescriptorSddlForm($NewSDDL)Set-Acl-Path$acl.path-ACLObject$acl

See the full code in the attached zip file, including sample output.  There is also a fancy progress bar as we scan the folders.  Think of the progress dots as Chuck Norris' foot flying to the target.  I finally followed The Scripting Guy's advice and took the time to add all the bells and whistles... comment-based help, advanced functions, parameter validation, etc.  With each new post I'll build this function library into a full dojo of karate moves.

Explicit Instructions

As always you should test scripts like this in a lab or on a small subset of data until you are comfortable with what it will do in production.  I have also provided a couple backup options in these steps:

1. Make sure you have a full backup from the night before.
2. Notify your backup administrator that you could potentially trigger a significant list of file backups if your software treats ACL changes as backup triggers.
3. For the PowerShell console use Run As with sufficient credentials to edit security on all subfolders in the path specified.
4. Use ICACLS to save a permissions template so that it can be reapplied in the event of an issue.
5. Dot-source the script file (dot space dot backslash script file):
      PS C:\>. .\SIDHistory.ps1
6. For instructions use Get-Help -Full to read about each function:
      PS C:\>Get-Help Export-SIDMapping -Full
      PS C:\>Get-Help Convert-SIDHistoryNTFS -Full
7. Create a SID mapping file:
      PS C:\>Export-SIDMapping
8. Review the report files, proceed with WhatIf first.
9. PS C:\>Convert-SIDHistoryNTFS \\server\share\path -WhatIf
10. Review the report files, decide to remove WhatIf and commit the changes.
11. PS C:\>Convert-SIDHistoryNTFS \\server\share\path
12. Review the report files.
13. Have your users test file access and login script drive mappings to make sure everyone is still happy.

Coming Up Next

Today's function library is not a full replacement for SIDWALK or ADMT.  It just does NTFS permissions for wherever you point it.  Next time we'll look at how to get a list of Domain SIDs in your environment so that you can interpret where SID history is coming from.  Now go unleash your Norris moves on the NAS.

For More Information

• Get-Help about_Hash_Tables
• Get-Help about_Scripts     (more on dot sourcing)
• Get-Help about_Functions
• Get-Help about_Functions_Advanced
• Get-Help about_Functions_Advanced_Parameters
• Get-Help about_Comment_Based_Help

 PS... I'm running out of Chuck Norris lines.  Add yours in the comments below, and I'll put them in the next post.

This weekend I'll be celebrating Christmas in the US.  Today Santa brought me a 1500VA 865W UPS for my home lab server.  Now I don't have to worry about those winter power hiccups rebooting all of my HyperV guests.  Best of all the scripting elves will be able to write presents without interruption.

So in PowerShell style here's a little Christmas gift for you.

Introduction

This post is the fifth in the "SID Walker, Texas Ranger" series on SID history remediation with PowerShell.  Today we're wrapping up with a handy summary of each post in the series. We will also take the function library we've been using and upgrade it to a PowerShell module. Then we'll walk through the entire SID history remediation process using the provided cmdlets in this module.

The Story So Far

Those of you who follow my blog know that I have been stuck on this theme of SID history for several months now.  Why?  Because I see this quite frequently with customers, and I want to offer some practical guidance on dealing with it.  Here is a summary of the blog series that brought us to today's module:

1. Using PowerShell to resolve Token Size issues caused by SID history
   Prior to starting the module development this post explained the background of token size issues as related to SID history.  I provided the basic SID history query that we use to produce the report and some great links for more information on token size.
2. Do Over: SID History One-Liner
   As a follow up to the Token Size post I re-wrote the SID history report query as a one-liner.
3. PowerShell: SID Walker, Texas Ranger (Part 1)
   This time we looked at Get-ACL and parsing SDDL strings, a warm up for the next post.
4. PowerShell: SID Walker, Texas Ranger (Part 2)
   Next I wrote a function to swap SID history entries in ACLs/ACEs.  This compensates for a gap in the ADMT, because it cannot migrate SID history for file shares hosted on a NAS.
5. PowerShell: SID Walker, Texas Ranger (Part 3): Exporting Domain SIDs and Trusts
   Looking at raw SIDs in a report is not very friendly, so I wrote a function that translates domain SIDs into domain names.  This makes the SID history report more meaningful when you can see the name of the domain from whence they came.  Enumerating all forest trusts and their domain SIDs required using some .NET ninja skills.
6. How To Remove SID History With PowerShell
   To round out the functions I provided Get-SIDHistory and Remove-SIDHistory, emphasizing that this is the LAST step in the process.  I leveraged the previous domain SID function to even give us the ability to remove SID history selectively by old domain name.

I suggest that you go back and read all of the articles linked above. They will give you much more insight into the SID history cleanup process and the nuances of the provided functions. Then skim through the ADMT Guide to get familiar with the big picture.

All of these functions are now wrapped up in the module provided in today's blog post.

Installing the Module

If you've never installed a module there really isn't much to it.  Here's what you do:

1. Create the module folder (adjust Documents path if necessary):
   New-Item -Type Directory -path "$home\Documents\WindowsPowerShell\Modules\SIDHistory"
2. Download the attached ZIP file at the bottom of this article.
3. Unzip the contents into this path:
   C:\Users\<username>\Documents\WindowsPowerShell\Modules\SIDHistory\
4. Fire up the PowerShell console or ISE.
5. Import-Module ActiveDirectory (This is a prerequisite.)
6. Import-Module SIDHistory

Now you can use Get-Command and Get-Help to unwrap the present and see what's inside:

• Get-Command -Module SIDHistory
• Get-Help Get-SIDHistory -Full
• Get-Help Get-SIDHistory -Online
• Get-Command -Module SIDHistory | Get-Help -Full | More

You can use Get-Help -Full for each of the included functions to find syntax and descriptions.

Using the Module

The outline below will guide you through the process of using the functions to help remediate SID history.  Run them in this order.

• Start up:
• Import-Module ActiveDirectory
• Import-Module SIDHistory
• Get the SID history report:
• Export-DomainSIDs
• Export-SIDMapping
• Update-SIDMapping
• Open the SIDReportUpdated.csv file in Excel to see all of the SID history in your environment.
• Keep an archive copy of these output files for documentation at the end of the project.
• Use the ADMT for server migration:
• Use the SIDMap.csv file with the ADMT to migrate servers with SID history.  This file recovers your OldSID/NewSID data from former migrations so that you can finish security translation on servers.
• NAS permission migration:
• If you have NAS-based file shares, migrate SID history of NTFS shares this way:
• Run with -WhatIf the first time to see if there is any SID history to translate.
• Convert-SIDHistoryNTFS \\server\share\path –WhatIf
• Review the report files.  Run again without -WhatIf to actually update the ACLs.
• Convert-SIDHistoryNTFS \\server\share\path
• Review the report files.
• Confirm share file access with affected users and groups.
• Remove the SID history:
• Confirm that you have good backups of Active Directory system state on two DCs in every domain.  You should always have a backout plan in case you missed some SID history remediation.
• Once SID history remediation is verified on all servers you can begin removing SID history in phases.  First, use Get-SIDHistory to target the removal population with a specific query.  Second, pipe the output to Remove-SIDHistory.  Here are some examples:
• Get-SIDHistory –MemberOf AccountingDept
• Get-SIDHistory –MemberOf AccountingDept | Remove-SIDHistory
• Get-SIDHistory –DomainName alpineskihouse.com
• Get-SIDHistory –DomainName alpineskihouse.com | Remove-SIDHistory
• See the help for extensive filtering capabilities of Get-SIDHistory.
• Check your work:
• Make an archive copy of your first SIDReportUpdated.csv and SIDMap.csv files.
• Export-SIDMapping
• Use the SIDReport.csv file as an audit to see where SID history remains.
• Repeat the migration and removal processes until this report comes back empty.
• Remediation (ie. Damage Control):
• Assuming that your security translation was thorough then you should not see any issues.
• If the help desk only gets a couple calls:
• Manually clean up the permissions on a case-by-case basis.
• If the help desk gets hammered with calls, then you have a couple options:
• Identify the scope of impact and remediate those servers by doing more security translation.  You may have missed a few.
• Do an AD authoritative restore to recover the SID history of impacted users.

The functions provided in this module will give you added visibility into the status of your SID history throughout the process and an easy way to target removal in the final phase.

Conclusion

This SID history project has been a lot of fun, and I'm sure there's more we could do with it.  I have a few ideas of my own, but I would like to hear your feedback.  What challenges have you encountered with SID history remediation?  Where do you think PowerShell could help?  Leave a comment below and let me know.

Additional Reading

• Get-Help about_Modules
• Active Directory Migration Tool (ADMT) Guide: Migrating and Restructuring Active Directory Domains

TechReady

At Microsoft we have an internal event called TechReady that happens twice each year.  It is similar to our public TechEd event.  Approximately 5,000 Microsoft technical field people from all over the world gather to learn the details of the latest product releases and dialog with the product groups.  It is a fantastic event, and this July I'll be presenting a session called "Active Directory PowerShell Notes From the Field".  I'll be covering three case studies related to schema updates, DNS migrations, and Active Directory SID history.  I will conclude the session with a quick look at the future of DCPROMO, REPADMIN, and DNSCMD in Windows Server 2012, comparing the new PowerShell ways to do Active Directory administration in the next release.  If you are going to TechReady then look me up.

More SID History

I get emails frequently from folks who have read my series of articles on Active Directory SID history, so I'm guessing that is a good theme to continue.  Working with a customer recently I was able to enhance the functionality in the Active Directory SID history PowerShell module that I posted last December.  Today's post will publish and discuss those improvements.

Version 1.4
June, 2012

Functions added in this release:
Export-SIDHistoryShare
Merge-CSV

Functions modified in this release:
Convert-SIDHistoryNTFS
Export-SIDMapping
Update-SIDMapping
Get-SIDHistory

Fixes:
Removed Test-Path validation on NewReport parameter of Update-SIDMapping.
Added file validation for DomainFile parameter of Get-SIDHistory.

It's All About The Customer

Imagine if…

• your company acquired an average of 12 other companies each year.
• your forest had over 35 domains (consolidated from 50+).
• your forest had over 80 trusts to other domains you still need to migrate.
• your forest had over 172,000 instances of SID history.
• you lived in the ADMT from sun up to sun down.
• token bloat plagued your support engineers with mystery troubleshooting.

Obviously I can't make this stuff up.  Those points describe a customer who recently invited me to help them develop a process to identify and remediate SID history in their forest.  This also gave them a process going forward that would help them do future domain migrations in a way that would minimize the impact of SID history.

Enter PowerShell

I knew the SID history PowerShell module could help them, but I also knew that some improvements would be necessary to help them scale to the size of data collection they would need.  Here's what I did:

• I added a function called Export-SIDHistoryShare that will collect SID history information from share permissions.  (Note that it will not convert SID history as the NTFS function does.)
• I modified the NTFS SID translation report CSV generated by Convert-SIDHistoryNTFS so that the columns would match the output from the new share CSV output.  This involved adding ACLType and DateTimeStamp.
• Both NTFS and Share reports now include a date time stamp so that the CSV data can be imported into a larger database multiple times.  Different scans of the same data will be distinguished by the date time stamp.
• I modified the Export-SIDMapping and Update-SIDMapping to include a date time stamp field as well for the same reasons.
• I created a new function called Merge-CSV that will take all of the NTFS and Share SID history reports and roll them into a single, large CSV file for importing to a database (like Microsoft Access or Microsoft SQL).
• This is huge!  I created a quick proof-of-concept Access database that would relate the forest object SID history data against the SID history discovered in shares and NTFS permissions on servers.  This is super powerful for analyzing the impact of SID history.  Now you can produce reports that tell you:
  • which former domain has the most SID history
  • which groups with SID history are most used on your file servers
  • which file servers need ACL translation with the ADMT
  • where to target your remediation efforts for the largest impact
  • where to begin the cleanup with the users and groups whose SID history is not in use
  • etc.
• I also resolved one issue with the NewReport parameter of the Update-SIDMapping function and resolved one issue with the DomainFile parameter of the Get-SIDHistory function. (Special thanks to Andrew Hill and Greg Jaworski for their feedback.)

These changes added the scalability and flexibility for the customer to begin inventorying hundreds for servers for SID history and manage all the data with a full database they would implement later on their own.  I love the simplicity of CSV in PowerShell!

The New Process

My last post on SID history has the overall steps for installing the module and completing the SID history remediation so I will not repeat that content here.  But I do want to list out the steps for you to automate the NTFS and share SID history data collection:

• Inventory SID history in share and NTFS ACLs:
  • Run these commands against the servers and share paths where you want to check for
    SID history:
    • Convert-SIDHistoryNTFS \\servername\share\path –WhatIf
    • Export-SIDHistoryShare servername
    • NOTE: You will need to run this under an administrative account that has permissions to view all of the recursive subfolders.
  • Repeat these steps for all servers where you want to collect data.  See the example code below for another way to automate mass data collection.
  • Now merge all of the output into a single CSV file for analysis:
    • Put all of the Share and NTFS CSV output files into a new working folder.
    • Merge-CSV –Path "C:\Temp\WorkingFolder"
    • This creates a merged CSV called ACL_SID_History_xxxxxxxxxxxxxxxx.csv.
• Copy these two files to the folder with the provided Access database:
  • SIDReportUpdated.csv
  • ACL_SID_History_xxxxxxxxxxxxxxxx.csv
• Rename the ACL_SID_History_xxxxxxxxxxxxxxxx.csv file to ACL_SID_History.csv.
• Open the Access database.
• In order for the linked tables in Access to see these new files you must repair the links and point them to the files in your working folder.
  • Right click the table ACL_SID_History on the left.
  • Choose “Linked Table Manager”.
  • Check the box at the bottom “Always prompt for new location”.
  • Check the box beside each of the two tables.
  • Click OK.
  • Browse to the each of the two CSV files that you just copied into the database folder. Pay special attention to the title bar of the File browser dialog box. Make sure that you choose the file that matches the name in the title bar.
  • Close the Linked Table Manager dialog box.
• Now you can double click any of the example queries on the left to see the data analysis.  You can also create your own queries and custom reports.

Automating the Automation

It would be quite time consuming to run the NTFS and share scan commands one at a time against all of your servers.  Instead try these handy PowerShell routines to make the data collection go faster.  As these processes usually take hours to scan large file shares it would be a good idea to let them run over a night or weekend.

# Sample code to scan a group of Windows servers in a text file.$Servers=Get-Content".\servers.txt"ForEach($ComputerNamein$Servers){Export-SIDHistoryShare$ComputerName# Grab only file shares that were manually created$Shares=Get-WmiObjectWin32_Share-ComputerName$ComputerName-Filter"Type=0"ForEach($Sharein$Shares){Convert-SIDHistoryNTFS"\\$ComputerName\$($Share.Name)"-WhatIf}}# Sample code to scan a list of share paths in a text file.# This can work for Windows or NAS servers.$Shares=Get-Content".\shares.txt"ForEach($Sharein$Shares){Convert-SIDHistoryNTFS$Share-WhatIf}

Where can I get all this PowerShell goodness?

I have updated this code at the TechNet Script Gallery.  Attached to that entry you'll find the following in a single compressed file:

• Updated PowerShell module for AD SID history
• Sample output data files
• Sample Access database (You must update the linked tables to point to the CSV files.  See instructions above.)
• Sample script file for using the module

Extract all of the files into a working folder.  Install the module.  The usual disclaimers apply: this is sample code for use at your own risk.  Enjoy!

But Wait!  There's More…

If you would like me or another Microsoft PFE to visit your company and help you with the ideas discussed on this blog, please contact your Premier Technical Account Manager (TAM).  We would love to come see you.

Follow me on Twitter: @GoateePFE.

To see all of the articles in this series visit http://aka.ms/SIDHistory.

I would like to thank everyone who has been using the Active Directory SIDHistory PowerShell module and sending me feedback.  Your input helps guide future releases like the one I am publishing today.

I’ve been sitting on some updates for a while, because I prefer to release code that has been field-tested.  I also wanted to time this release with the upcoming PowerShell Deep Dives book where I have a chapter discussing the origins of this module.  The last update was version 1.4 in June of 2012.  This is update 1.5 in July of 2013.

Summary of Changes

I am excited to announce the following key improvements in this release:

• SID history updates in ACLs can be added instead of replaced.
• Create SID map files for security translation without needing SID history.
• Track down old domains after their trusts have been removed.
• Get error logging for file server paths that fail the ACL scan.
• Automate SID history data collection across many servers and shares.

Change Details

This release includes some significant changes and additions, which I have highlighted below. Here is a list of the functions in this release:

PS C:\> Get-Command -Module SIDHistory

CommandType Name                     ModuleName
----------- ----                     ----------
Function    Convert-SIDHistoryNTFS   SIDHistory
Function    Export-DomainSIDs        SIDHistory
Function    Export-SIDHistoryShare   SIDHistory
Function    Export-SIDMapping        SIDHistory
Function    Export-SIDMappingCustom  SIDHistory
Function    Get-ADObjectADSI         SIDHistory
Function    Get-DomainSIDWordCount   SIDHistory
Function    Get-SIDHistory           SIDHistory
Function    Get-SIDHistoryDuplicates SIDHistory
Function    Merge-CSV                SIDHistory
Function    Remove-SIDHistory        SIDHistory
Function    Search-SIDHistoryACL     SIDHistory
Function    Update-SIDMapping        SIDHistory
Function    Watch-Job                SIDHistory

Due to the large number of changes I am not going to include code samples for each function in this article.  Please use Get-Help -Full to see complete details and examples for each module member.

PS C:\> Get-Help Convert-SIDHistoryNTFS -Full

PS C:\> Get-Help Export-SIDMappingCustom -Full

PS C:\> Get-Help Get-DomainSIDWordCount -Full

Etc...

Functions Added

Export-SIDMappingCustom

• I had a customer who was not able to use the Active Directory Migration Tool for a domain migration.  The newly acquired subsidiary was not allowed to create a trust for compliance reasons.  Obviously we need a trust to do a migration.  Or do we?
• To work around the situation I wrote a function that will map SIDs between accounts in two different domains where there is no SID history.  The trick is having a common, unique attribute like samAccountName, employeeID, employeeNumber, mail, etc.
• The customer manually exported the accounts from the old domain and imported them into the new domain.  Next they made sure that their HR system populated the same unique Employee ID in the new domain.  Now they had a common, unique key between the two domains.
• In the old 2003 domain we used an ADSI script to create a file containing the objectSID and employeeID attributes.  (See Get-ADObjectADSI described below.)  This is all we needed to create our mapping file.  We copied this file across international borders to the local new domain.
• Now we can use the new Export-SIDMappingCustom function to create a mapping file between the exported SID data from the old domain and the live accounts in the new domain.  This is only one possible scenario with the function.  It can use any combination of live connection or export file from the old and new domains.
• In this particular case the customer only wanted to migrate the file server from the old domain.  With this new SID mapping file they were able to run Convert-SIDHistoryNTFS against the old server to re-ACL the resources.  This is POWERFUL.  Essentially, you can do a NAS ACL migration without SID history in place.
• This works for both users and groups.  At another customer they wanted to migrate several domain local groups.  The groups had been recreated with the same name in the new domain without SID history.  The Export-SIDMappingCustom function was able to create a mapping file by matching the group names between both old and new live domains and then putting both SIDs into a mapping file.

Get-SIDHistoryDuplicates

• There are processes to resolve duplicate SIDs on accounts but not for duplicate SID history.
• This should never never never happen.  But it happened for one customer, the same customer where I discovered 100 unique old domains.  We’re not sure how the duplicate SID history entries got there, because there were years of migration history with a variety of migration tools.
• Duplicates in SID history are a violation of everything we know to be good and true.  There can only be one.  If duplicates are found, then it would invalidate the SID mapping process.
• To find these I wrote this function to generate a report.  Once identified, the customer chose to simply delete the accounts with the duplicates, because they were old empty groups.

Search-SIDHistoryACL

• In this module two main functions do all of the work to discover SID history on shares and files:  Convert-SIDHistoryNTFS and Export-SIDHistoryShare.  In order to fully document SID history on your file servers you need to call each of these functions for each share on each server.  That’s a lot of calls.
• This is a “meta-function” that calls these two functions for a batch of servers or shares listed in either an OU or a text file.  Using PowerShell background jobs it scans them all in parallel to reduce the discovery time required.
• My only caution is that there is no throttling built in.  Run these in small batches of servers until you see how the performance works out with your hardware.
• Use the following Watch-Job function to manage these background jobs.

Watch-Job

• This function conveniently manages receiving the output from the jobs spun up by Search-SIDHistoryACL.  It displays a PowerShell progress bar indicating how many jobs are still running.  Once they are all complete it will receive the output into a consolidated job log and report on each job’s start and end times.  All of the ACL output files are rendered separately by their respective jobs.
• Optionally you can have the function clear the job queue with the -Remove switch.
• This is a utility function, and it could be modified easily for many other background job scenarios.

Get-DomainSIDWordCount

• The Export-DomainSIDs function crawls all trusts in the forest to identify domain names and domain SIDs.  When old trusts to migrated domains are gone you can no longer identify where SID history originated… until now.
• The function creates a list of word counts gleaned from popular string attributes found on accounts from each old domain SID identified in the forest.  This gives you clues about where the accounts in the unidentified SID history domain may have originated.  For example, employees migrated from the old ContosoPartner domain may have that company name string in their description, department, display name, or OU path.  By finding and counting the common strings across all migrated accounts you can usually identify the old domain.
• This process gives you the information needed to manually update the DomainSIDs.csv file.  After updating the file re-run the Update-SIDMapping function to add these old domain names to your master SID history report.
• Pipe the output to Export-CSV or Out-GridView for easy viewing.

Get-ADObjectADSI

• I wrote this utility function for a customer with legacy DCs where they were not going to install the AD web service.  They installed PowerShell, and we used this function to create the account listing for Export-SIDMappingCustom.  It uses ADSI to mimic Get-ADObject, but it was a point solution not intended for full parity with the AD module cmdlet.  This function is intentionally undocumented, because most environments will not need it.  Feel free to modify it to meet your needs.

Functions Modified

Export-SIDMapping

• I added description and whenCreated attributes to the SIDReport.CSV.  These properties help identify accounts in the list, especially old accounts that are likely stale.
• By mistake I was running the AD query twice in this function, so I corrected it to only run once.  Obviously this will greatly improve performance.

Update-SIDMapping

• Added description and whenCreated to SIDReportUpdated.CSV for same reasons listed above.

Convert-SIDHistoryNTFS

• Added -Add switch.  The previous functionality did an ACE replace only.  Now you have the option of doing a replace or an add.  This way you can co-exist with both old and new SIDs temporarily until you are ready to completely remove all old SIDs from the ACLs.  Running without the -Add switch gives you a replace, leaving the new SID and removing the old SID.
• As a result of the new -Add switch I had to modify the CSV output and log files to report the status of both Old SID and New SID in the ACLs.  The “Both” column will be true if both the old and new SIDs are present in the ACL.  The “Operation” column will now say “Add” or “Replace”.  Added new column for NewDomainSID.
• After more field testing I discovered that there were some issues scanning folders.  In prior releases you would see red text in the PowerShell console, but you didn’t know where the errors occurred. I added error logging so that you get a list of the reasons and the paths where the ACL scans fail down inside the folder tree.  Primarily I’ve found three issues:
• 1. Path too long.  I spent a lot of time investigating this issue.  The Get-ACL -Path parameter is limited to paths of 260 characters or less.  I found this TechNet Wiki article and this blog series, but none of the proposed work-arounds were applicable.  As a temporary work-around if you run into this problem you may be able to map a drive down farther in the folder structure of longer paths and run the scan again for that newly mapped path.  This is due to an underlying .NET limitation.  Unfortunately this error only gets logged when running in PowerShell v3.  PowerShell v2 will not report this error in a way that we can trap.
• 2. Invalid character in path.  I discovered an issue with Get-ACL that does not return an object when the path contains a ‘[‘ character.  After researching this on the Microsoft PowerShell Connect site I found that it is resolved in PowerShell v3 with the new -LiteralPath parameter.  For now I am logging the error until I update the module to use PowerShell v3 features.
• 3. Access denied.  Using this function you often find file shares where Administrator rights have been removed.  To resolve this either rescan the path in question with appropriate credentials or modify the permissions on the path.

Export-SIDHistoryShare

• Modified CSV columns to match the output from Convert-SIDHistoryNTFS.  This allows all results to be consolidated into a single CSV report.

Time-Saving Tip

When using Convert-SIDHistoryNTFS -WhatIf to scan servers and shares it can take hours or days depending on the size of the environment.  Sometimes you may only find a few SIDHistory entries that need to be remediated.  Rather than rescan the entire server or share to fix these ACLs you can target the SID history cleanup this way to save time:

PS C:\> Import-CSV ACL_SID_History.csv |
 Select-Object -Unique -ExpandProperty Folder |
 ForEach-Object {Convert-SIDHistoryNTFS $_}

In this example ACL_SID_History.csv is the NTFS permission scan output from Convert-SIDHistoryNTFS -WhatIf.  We simply get a unique list of folders and feed those into the conversion function.  Obviously this is much faster than rescanning every ACL of every folder in the share subtree.

What’s next?

I have a long list of pending enhancements for the module, but my top priority is to update it with PowerShell v3 features:

• Use -LiteralPath in place of -Path to avoid the ‘[‘ issue in Convert-SIDHistoryNTFS.
• Rewrite the Export-SIDHistoryShare function using the new SmbShare module.
• Create a function that will update share ACLs using the new SmbShare module.
• Create a -Server parameter for Get-SIDHistory and Export-SIDMapping to target a specific DC.
• Capture all output streams correctly for Watch-Job logging.

Please use the comment area below to tell me what features you would like to see improved or added in the module.  I would also appreciate any feedback for how this has helped your projects.

Download

You can download the updated Active Directory SIDHistory PowerShell module from the TechNet Script Gallery.

PowerShell WMF 5.0 RTM Video Training now available!

PowerShell WMF 5.0 has reached RTM, and I am excited to announce video training!  Microsoft Premier customers now have access to 11 hours of PowerShell WMF 5.0 RTM video training through the Premier Workshop Library on Demand (WLOD) subscription.  We recorded seven modules on topics from beginner to advanced.  This training builds on our PowerShell Part 1 and Part 2 titles to cover new features in WMF 5.0.

Microsoft Premier Workshop Library on Demand

Microsoft Premier customers can get full-length, quality PowerShell training directly from Microsoft via streaming video at your convenience.

As a Microsoft Premier Field Engineer I teach many PowerShell workshops. Traditionally this has been classroom work, but over the last few years we have expanded our online offerings. We teach live remote workshops over Skype and use cloud-hosted labs. We also record training videos for you to stream on your schedule.

We call the online training videos Education as a Service, and we have given it the brand name Premier Workshop Library on Demand (WLOD).  Check out a YouTube video explaining it here. The video training is an add-on item for Premier contracts. Not all Premier customers may have purchased this option, but they can always add it.

These two screenshots are from the sessions with my peer, Jeff Dykstra, when we recorded WorkshopPLUS – Windows PowerShell v5 Features.

Content Outline

Here is an outline of the content covered in each module:

Module 1: Introduction. Get an overview of Windows PowerShell v5, the requirements, terminology, and implementation considerations. Identify new features in the console and ISE for easier scripting.

Module 2: New Cmdlets, Parameters, and Language. Begin using new cmdlets for file compression. Copy files securely over WinRM. Quickly parse text files into object-oriented data without regular expressions. Generate cmdlets from an OData endpoint.

Module 3: Package Management, PowerShellGet. Understand the new command-line tools for discovering, installing, and inventorying software, modules, and scripts. Publish your own internal PowerShell repository for modules and scripts.

Module 4: Security. Encrypt files using Cryptographic Message Syntax (CMS). Enforce system-wide transcription through Group Policy. Enabled detailed logging of script blocks executed on a server. Combat script-based malware with new features in Windows 10 and Windows Server 2016.

Module 5: Remote Debugging. Debug PowerShell scripts in other runspaces: background jobs, remote machines, and Desired State Configuration (DSC). Edit scripts on Windows Server Core remotely using the improved PowerShell ISE and new psEdit command.

Module 6: Classes. Leverage familiar developer syntax to create powerful custom objects in PowerShell v5. Create enums, properties, methods and constructors. Understand inheritance and interfaces.

Module 7: JEA – Just Enough Administration. Use PowerShell’s built-in Role Based Access Control (RBAC) to whitelist only cmdlets essential to remote administration tasks. Implement virtual accounts to minimize exposure of domain credentials on key servers.

How do I get it?

Microsoft Premier customers can subscribe to this offering; talk to your Premier Technical Account Manager (TAM) for more information.  If you would like to become a Premier customer go here.

To access the video courses follow these steps:

• Go to https://services.premier.microsoft.com
• Click Sign In
• Log in with your Microsoft Account (Windows Live ID) that has been granted access to Workshop Library On-Demand. For questions contact your Premier TAM.
• Click on My Education
• Use the Category filter on the left to find Windows Server titles
• Scroll through the list or use CTRL+F in the browser to find the specific PowerShell titles
• The course title is WorkshopPLUS – Windows PowerShell v5 Features

In addition to this title, you will have access to watch hundreds of Microsoft Premier workshops as many times as you like.

Enjoy!

Hi folks. Quick checkpoint here to let you know where we can meet up on the road. I’ll be presenting at three events in March and April. Hope to see you there.

These events are a great way to grow your skills and your career. It’s not too late to register for the Florida or Ohio events.

Here are the details and the topics I’ll be presenting:

PowerShell Saturday #PSSAT010

March 19 – Tampa, Florida – http://powershellsaturday.com/010/

• Top Five Questions Customers Ask About PowerShell Desired State Configuration (DSC)
• Troubleshooting and Debugging DSC

PowerShell and DevOps Global Summit @PSHSummit

April 4-6 – Bellevue, Washington – http://powershell.org/wp/summit/

• Active Directory Forensics with PowerShell
• Active Directory DSC

LRITA Technology Conference @LRITA_org

April 14 – Lima, Ohio – http://lrita.org/

• Active Directory Forensics with PowerShell

People frequently ask me, “Where can I learn more about PowerShell?” Today’s post will highlight popular video training options available directly from Microsoft.

Microsoft Virtual Academy (MVA)

There are many free, publicly available PowerShell video courses on Microsoft Virtual Academy. These cover a range of topics from core PowerShell features to Desired State Configuration (DSC), Active Directory, Exchange, SharePoint, Office 365 and more. Here are the most popular titles:

• Getting Started with PowerShell 3.0 Jump Start
  Start here as a beginner. Meet Jeffrey Snover. Learn it straight from the source. Even though “3.0” is in the title, this content is completely relevant today.
• Advanced Tools & Scripting with PowerShell 3.0 Jumpstart
  Second day of PowerShell basics, continuation of the first link. Learn about functions and modules. Great stuff!
• What’s New in PowerShell v5   NEW!
  Learn about the new features in PowerShell v5 from celebrities on the PowerShell team and PowerShell MVPs.
• Getting Started with PowerShell Desired State Configuration (DSC)
  Free DSC training recorded midway through the WMF 5.0 release cycle.
• Advanced PowerShell Desired State Configuration (DSC) and Custom Resources
  Second day of DSC training recorded midway through the WMF 5.0 release cycle.
• Building Your Datacenter One DSC Resource at a Time
  Practical training on PowerShell DSC resources from the experts.
• Using PowerShell for Active Directory
  One day of training I recorded to help everyone from beginner to advance learn how to script for Active Directory.
• Managing Exchange Online Using PowerShell
  Great training from one of my PFE peers.
• Deploying & Managing SharePoint 2013 with PowerShell
  Another example of domain-specific PowerShell training on MVA.
• …and many more!

MSDN Channel9

Another avenue for free online PowerShell training is Channel9. Here you will find PowerShell video training from a number of sources and in a variety of languages in addition to English. This is a great place to find recorded sessions from TechEd and Ignite events.

edX

For a while Microsoft offered a paid PowerShell class on edX. That course has been archived now, but you can look there for popular Microsoft technology courses. Many of these are free.

Microsoft Premier Workshop Library on Demand

You have many choices for training these days, both online and in the classroom. Microsoft Virtual Academy titles usually move fast to cover a lot of content. These Premier titles will dive deeper and slow down the pace to help with learning. Customers prefer training from Microsoft PFEs, because they get knowledge that comes from real-world experience of engineers working with customers of all sizes.

Microsoft Premier customers can get full-length, quality PowerShell training directly from Microsoft via streaming video at your convenience.The video training is an add-on item for Premier contracts. Not all Premier customers may have purchased this option, but they can always add it.

As a Microsoft Premier Field Engineer I teach many PowerShell workshops. Traditionally this has been classroom work, but over the last few years we have expanded our online offerings. We teach live remote workshops over Skype and use cloud-hosted labs. We also record training videos for you to stream on your schedule.

We call the online training videos Education as a Service and have given it the brand name Premier Workshop Library on Demand (WLOD).  Check out a YouTube video explaining it here.

What is available?

Find below the current list of PowerShell video workshops available for Premier customers through WLOD. The links below go directly to the courses once you are logged into the Premier Services portal.

• Windows PowerShell v4.0 for the IT Professional, Part 1 – 14.5 hours – https://aka.ms/wpsp1
  Beginners start here. My buddy Gary and I will teach you all the essentials to start your journey as a PowerShell scripter. This is our most popular course of all Premier workshops!
• Windows PowerShell v4.0 for the IT Professional, Part 2 – 12.5 hours – https://aka.ms/wpsp2
  Gary and I continue with deeper content on advanced functions, regular expressions, debugging, error handling, etc.
• WorkshopPLUS – Windows PowerShell v5 Features – 11 hours – https://aka.ms/wpsv5
  This course covers the new PowerShell features released since the recording of the Part 1 and Part 2 workshops. Find detailed training on the new features in PowerShell WMF 5.0.
• Desired State Configuration with PowerShell 5.0 – 17 hours – https://aka.ms/wpsdsc
  Everyone is talking about it. Learn it here. Ben and I spent time with customers in the field doing DSC, and then we wrote this course for you to learn it.

Here I have only listed the PowerShell titles. Essentially all of our Premier workshop titles are available on video now, so PowerShell is only the tip of the iceberg.

We’ve received a great deal of positive feedback from customers who love this stuff.

How do I get it?

Microsoft Premier customers with the WLOD subscription can access this offering; talk to your Premier Technical Account Manager (TAM) for more information and pricing.  If you would like to become a Premier customer go here.

To access the video courses follow these steps:

• Go to https://services.premier.microsoft.com
• Click Sign In
• Log in with your Microsoft Account (Windows Live ID) that has been granted access to Workshop Library On-Demand. For questions contact your Premier TAM.
• Click on My Education
• Use the Category filter on the left to find Windows Server titles
• Scroll through the list or use CTRL+F in the browser to find the specific PowerShell titles

You will have access to watch hundreds of Microsoft Premier workshop titles.

Microsoft Premier Webcast: PowerShell Ramp-Up Series

The Premier Webcast – PowerShell Ramp-Up Series is a collection of webcasts created with the goal of teaching from basic to more advanced PowerShell skills. No prior knowledge is necessary.

Beginning April 1, 2016 we have a special three month window of on-demand PowerShell training webcasts. Topics include:

• Fundamentals to Advanced
• Desired State Configuration
• Advanced Functions
• Workflow
• Modules
• Office 365
• SQL
• System Center
• SharePoint
• Security
• And more…

This PowerShell Ramp-up Series offering allows developers and administrators to logon and view 16+ webcasts with a duration of approximately 2-hours each, which totals approximately 36 hours of PowerShell training. The content can be accessed anytime throughout the 3-month availability. Access starts on the first day of the first month and expires on the last day of the third month.

Sessions are delivered via on-demand streaming. In the recorded webcast, the instructor provides a combination of PowerPoint and demos in a rich and effective learning experience. Delivery time for each webcast is 90-120 minutes and the series contains a minimum of 16+ webcasts.

Contact your Microsoft Services Representative or TAM for more information and pricing.

What about non-Microsoft video training?

YouTube

Of course YouTube is packed with other video options as well. I especially want to point out that you can find most of the PowerShell Summit presentations there. MVPs, Microsoft PowerShell team members, and other members of the community present fantastic topics that are usually 300-400 level at this annual event previously held in North America and Europe. You will see expert demos in these videos that you will find nowhere else.

Others

There are many other video training options from MVPs and partners, ranging from free to paid. As a Microsoft employee I cannot endorse any one particular offering. However, I can tell you that many of the PowerShell MVPs record training for PluralSight. I have not watched any of that content, but I know many of these folks and trust them to do a great job.

Bonus Content: PowerScripting Podcast

One of my favorite resources for learning PowerShell prior to all of the video training options we have today has been the PowerScripting Podcast. This resource was created by two MVPs out of Atlanta, Jon and Hal. You can listen to interviews with many celebrities and MVPs in the world of PowerShell. They even have a chat room you can attend during the live podcast recordings to ask questions of the guests. I highly recommend this resource to customers as I deliver PowerShell training.

Get started today!

Surviving and thriving in the IT industry requires constant learning. PowerShell is a core skillset. Regardless of your current knowledge level of PowerShell, these resources will raise your game. Learn something new today!

That Active Directory Test Lab You Have Always Wanted

As a Microsoft Premier Field Engineer I have visited countless customers to assess and assist their Active Directory environment. Sadly, the majority of those customers do not have an Active Directory test environment. Today I am going to show you how to quickly deploy and destroy your own Active Directory test lab using Azure Resource Manager and PowerShell.

Build 10 new AD labs a day if you want. It is crazy easy and crazy fast using this script. Imagine going to your morning status meeting, and then coming back to an Active Directory domain ready for you to do your worst. When you are done, a single line of PowerShell erases the entire environment.

PowerShell and DevOps Global Summit 2016

Watch the demo and explanation of this code from the 2016 PowerShell Summit.

Quickly? Azure? In the same sentence?

Wait! Keep reading! I promise this is painless. I have done all the heavy lifting for you. All you need is an Azure subscription.

• Microsoft Enterprise Agreements include Azure.
• MSDN and Visual Studio subscriptions include Azure.
• You can use a free trial of Azure.
• Or you can outright buy your own paid subscription.

This solution is a mash up of the following technologies:

• Azure
• Azure DSC extension
• PowerShell DSC
• Azure Resource Manager (ARM) templates

Gone are the days of stumbling through the Azure interface to provision network, storage, compute, and sanity. Now using ARM templates you can spin up a complete environment with a couple lines of PowerShell. Yes, it can be that easy.

What’s in the secret sauce?

I started out by snagging a copy of a free AzureRM template for a domain controller. There is a large repository of sample templates on GitHub. Check it out.

I wanted to build my own DSC configuration for the domain, so I started with the included DSC configuration and added my own twist using a newer version of the xActiveDirectory DSC resource module (available on PowerShell Gallery and GitHub).

I watched Trevor Sullivan’s presentation on deploying AzureRM templates with PowerShell.

I asked my PFE buddy Ben Wilkinson for a head start with the calling code. He showed me how to get the Azure cmdlets installed and launch the template build.

I read a bunch of blog posts and documentation. You know how it is learning something new. You bang your head against the docs and your keyboard until you get a breakthrough. Here are some links that helped me:

https://azure.microsoft.com/en-us/blog/azps-1-0/
https://blogs.msdn.microsoft.com/powershell/2016/02/26/arm-dsc-extension-settings/
https://blogs.msdn.microsoft.com/powershell/2014/08/07/introducing-the-azure-powershell-dsc-desired-state-configuration-extension/
https://azure.microsoft.com/en-us/documentation/articles/automation-dsc-onboarding/
http://blogs.msdn.com/b/golive/archive/2015/09/01/developing-dsc-scripts-for-the-azure-resource-manager-dsc-extension.aspx
https://blogs.msdn.microsoft.com/powershell/2014/11/20/release-history-for-the-azure-dsc-extension/
https://blogs.msdn.microsoft.com/powershell/2015/06/17/azure-dsc-extension-v2-0-released/
https://blogs.msdn.microsoft.com/powershell/2014/09/10/secure-credentials-in-the-azure-powershell-desired-state-configuration-dsc-extension/

Then I hacked at it for a day until it worked.  :–)>  Now you get the benefit of a fully-functional solution.

After you install the Azure cmdlets you have just three commands to deploy your lab, and then one to blow it away when you are done:

• Login-AzureRmAccount
• New-AzureRmResourceGroup
• New-AzureRmResourceGroupDeployment
• Remove-AzureRmResourceGroup

Anyone working with AzureRM templates will probably tell you to use Visual Studio to create and edit them. And that is probably the easier way. However, I know many of you as infrastructure ops are not comfortable with Visual Studio dev tools. I wanted to prove that it can be done without the overhead of a massive Visual Studio install and license. I did some of my editing in the free, light-weight editor Visual Studio Code, but you could just as easily use Notepad or the PowerShell ISE to edit the json file.

How do I make my AD lab?

Usually my blog posts go into great detail and documentation for each script that I publish. In this case you can watch the video above for all the details. You will need WMF 5.0 installed before you get started.

I put the whole thing on GitHub for you to learn from. All you have to do is download CallingScript.ps1, edit the naming strings for uniqueness, and then run it a few lines at a time. The comments in the script will guide you. In 30 minutes you will have a populated AD test lab in Azure that you can connect to with RDP (remote desktop protocol). You could even copy and paste more scripts into the VM for further configuration or testing. Below is a screenshot of the finished domain controller.

When you have more time, download all the files from my GitHub and dive into the DSC configuration and azuredeploy.json parts. Using the techniques I demonstrate in this code, you can tweak the domain sample data however you like. Then you can host your own version of these files on GitHub or Azure storage. You could even take just the DSC piece to build your own test lab in a local VM.

One parting piece of advice…

If you do not destroy the Azure resource group at the end of the day, then be sure to shutdown and deprovision the VMs in Azure. You pay for them as long as they are running. You can do this via the Azure portal or script. Consider yourself warned. Enjoy!

Is PowerShell a vulnerability?

Does your organization forbid the use of PowerShell remoting? Has InfoSec blocked remote server administration with PowerShell? Do you want to understand PowerShell security better? This post is for you.

Notes from the field…

As a Premier Field Engineer I have delivered PowerShell engagements to countless companies and teams. Over the years I have consistently received questions around PowerShell remoting security. More recently I have worked with multiple customers who had been tasked with restricting PowerShell capabilities in their environment. The conversation usually starts with one of these lines:

• “InfoSec will not let us turn on PowerShell remoting.”
• “Our last audit said that PowerShell needs to be locked down on all servers.”
• “The CIO went to a security conference and then banned PowerShell from the environment.”

Now, be aware that these same companies will leave any one or more of these remote management ports open:

• Remote Desktop Protocol (RDP)
• Remote WMI access over RPC, clear text by default, random ports
• Remote event log management
• Remote service management
• SMB file share access
• PSEXEC

But they are afraid of:

• PowerShell remoting, always encrypted, single port 5985 or 5986, does all of the above

Here are the key points for a PowerShell security conversation:

• PowerShell is a neutral administration tool, not a vulnerability.
• PowerShell remoting respects all Windows authentication and authorization protocols. It requires local Administrators group membership by default.
• Hackers use PowerShell for the same reasons you do… because it is more convenient than twenty years of other popular command line tools.

Bottom Line

The improvements in WMF 5.0 (or WMF 4.0 with KB3000850) make PowerShell the worst tool of choice for a hacker when you enable script block logging and system-wide transcription. Hackers will leave fingerprints everywhere, unlike popular CMD utilities. For this reason, PowerShell should be the only tool you allow for remote administration. These features allow you to answer the classic questions who, what, when, where, and how for activities on your servers.

The links below provide documentation and training from Microsoft and other industry sources around securing PowerShell in the enterprise and enabling these key points above. Focus especially on the highlighted ones and anything from Lee Holmes.

Microsoft Resources

Download PowerShell WMF 5.0

Leverage all of the new security features in WMF 5.0.

http://microsoft.com/powershell

PowerShell Remoting Security Considerations

New security documentation from the PowerShell team. This is a start, and it will continue to be updated. Give this link to your InfoSec people who need more information.

https://msdn.microsoft.com/en-us/powershell/scripting/setup/winrmsecurity

PowerShell ♥ the Blue Team

Whitepaper by Lee Holmes “Scripting Security and Protection Advances in Windows 10” (PowerShell 5).

Give this to your InfoSec people, your manager, and your grandmother. Then implement it.

https://blogs.msdn.microsoft.com/powershell/2015/06/09/powershell-the-blue-team/

Just Enough Administration

JEA is how you restrict capabilities of remote PowerShell sessions. This is huge for securing your environment and a major thrust in Windows Server 2016 (available down level with WMF 5.0).

https://msdn.microsoft.com/powershell/jea/readme

Introducing the updated JEA Helper Tool

https://blogs.technet.microsoft.com/privatecloud/2015/12/20/introducing-the-updated-jea-helper-tool/

Windows Event Forwarding guidance

http://aka.ms/wef

Maslow’s Hierarchy of Security Controls

http://www.leeholmes.com/blog/2014/12/08/maslows-hierarchy-of-security-controls/

Use Windows Event Forwarding to help with intrusion detection

An operational guide for implementing WEF based on the experience of Microsoft IT in a large-scale environment

https://technet.microsoft.com/itpro/windows/keep-secure/use-windows-event-forwarding-to-assist-in-instrusion-detection

The Underhanded PowerShell Contest

Microsoft’s community effort to identify and defend against malicious PowerShell code.

https://github.com/PowerShell/underhanded-powershell

https://blogs.msdn.microsoft.com/powershell/2016/03/07/announcing-the-underhanded-powershell-contest/

Follow Lee Holmes on Twitter

PowerShell team security lead, Microsoft Lead Security Architect for Enterprise Cloud Group & Azure Stack. Constant stream of relevant info and links.

https://twitter.com/Lee_Holmes

PowerShell Remoting Exposed

An article that includes network traces comparing WMI, RPC, and WINRM/WSMAN PowerShell remoting protocols on the wire. It also includes a list of PowerShell help topics for more information.

https://blogs.technet.microsoft.com/ashleymcglone/2011/04/18/powershell-remoting-exposed-how-to-command-your-minions/

External Resources

NSA: Spotting the Adversary with Windows Event Log Monitoring

https://www.nsa.gov/ia/_files/app/spotting_the_adversary_with_windows_event_log_monitoring.pdf

Securing PowerShell in the Enterprise

Information security advice for all levels of government

Australian Signals Directorate, Australian Government Department of Defence

http://www.asd.gov.au/publications/protect/securing-powershell.htm

FireEye: Investigating PowerShell Attacks (BlackHat USA 2014)

https://www.fireeye.com/content/dam/fireeye-www/global/en/solutions/pdfs/wp-lazanciyan-investigating-powershell-attacks.pdf

Verizon 2016 Data Breach Investigations Report

Not PowerShell-specific, but good insight to breach techniques.

http://www.verizonenterprise.com/verizon-insights-lab/dbir/2016/

http://www.verizonenterprise.com/resources/reports/rp_DBIR_2016_Report_en_xg.pdf

Audio/Video

Microsoft Virtual Academy: What’s New in PowerShell v5

Lee Holmes and Ryan Puffer demonstrate security and JEA in WMF 5

http://aka.ms/MVAps5

PowerShell Security at DerbyCon 2016
Great list of security conference session recordings on the current landscape of PowerShell security. Includes talks by Jeffrey Snover and Lee Holmes.
https://blogs.msdn.microsoft.com/powershell/2016/09/27/powershell-security-at-derbycon

Security Weekly 460: Interview with Lee Holmes

https://www.youtube.com/watch?v=tzcabAVuJFw

RunAs Radio Podcast #471: Just Enough Admin and Windows Server 2016 with Jeffrey Snover

http://www.runasradio.com/default.aspx?ShowNum=471&utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+RunasRadio+%28RunAs+Radio+%28mp3%29%29

Microsoft Ignite 2015: JEA: A PowerShell Toolkit to Secure a Post-Snowden World

WMF 4.0 version of JEA

https://channel9.msdn.com/events/Ignite/2015/BRK2470

PowerShell Summit 2015: Managing PowerShell in the Enterprise Using Group Policy

Overview of GPOs involved for securing PowerShell

https://www.youtube.com/watch?v=NRnGP1RRNsM&index=20&list=PLfeA8kIs7CochwcgX9zOWxh4IL3GoG05P

PowerShell Summit 2015: Defending the Defenders Pt 1

Lee Holmes and MVP Jeff Hicks

https://www.youtube.com/watch?v=2HBTj0WU5lY&list=PLfeA8kIs7CochwcgX9zOWxh4IL3GoG05P&index=16

PowerShell Summit 2015: Defending the Defenders Pt 2

Lee Holmes and MVP Jeff Hicks

https://www.youtube.com/watch?v=mua7DhxamHs&list=PLfeA8kIs7CochwcgX9zOWxh4IL3GoG05P&index=17

PowerShell Summit 2015: Keeping Secrets

MVP Dave Wyatt

https://www.youtube.com/watch?v=Ta2hQHVKauo&index=19&list=PLfeA8kIs7CochwcgX9zOWxh4IL3GoG05P

Learning PowerShell Desired State Configuration

PowerShell DSC is one of my favorite topics to teach, because the technology is simply amazing. Usually I do not have enough time with a customer to teach all of the built-in resources. I would guess that the Script resource is one of the least understood. Today’s post outlines a simple example to enable the Windows Firewall. This sample will work on Windows Server 2008 R2 and above.

Automating Your Server Build Sheet

Last week I was on site with a customer doing a PS DSC POC (PowerShell Desired State Configuration Proof of Concept). We took their server build check sheet, identified DSC resources for each step, and then began converting their manual server config process into PowerShell DSC.

The customer was new to DSC and requested a sample of how they could use the built-in Script resource. One of the items on the build sheet was to make sure the Windows Firewall was turned on. This was the perfect opportunity for a Script resource example.

The Script Resource

When you cannot find a built-in DSC resource or community resource to do what you need you have two options for custom resource code:

1. Built-In Script Resource – if the code is relatively simple, maybe just a couple lines
2. Custom Resource – if the code requires complexities like helper functions, extensive variable use, etc.

The primary decision factor between these two is the level of effort and complexity required. If you can do it in a couple lines of code, then try using a Script resource first. If the code is more involved, then write a custom resource. Obviously a script resource is much less effort if you can get by with it.

In the case of the firewall example, there is a community resource xDSCFirewall that might suffice, but it does not work on Windows Server 2008 R2.

Script Resource Firewall Example

Here is an elegantly simple use of the Script resource. We could make it more complex in many ways, but I prefer to start out with a simple use case first. The code is its own documentation. Study it carefully.

Configuration FirewallScriptResource
{
    Import-DscResource -ModuleName PSDesiredStateConfiguration
    Node localhost
    {
        Script EnableFirewall
        {
            # Must return a hashtable with at least one key
            # named 'Result' of type String
            GetScript = {
                Return @{
                    Result = [string]$(netsh advfirewall show allprofiles)
                }
            }

            # Must return a boolean: $true or $false
            TestScript = {
                If ((netsh advfirewall show allprofiles) -like "State*off*") {
                    Write-Verbose "One or more firewall profiles are off"
                    Return $false
                } Else {
                    Write-Verbose "All firewall profiles are on"
                    Return $true
                }
            }

            # Returns nothing
            SetScript = {
                Write-Verbose "Setting all firewall profiles to on"
                netsh advfirewall set allprofiles state on
            }
        }
    }
}

FirewallScriptResource

Start-DscConfiguration -Path .\FirewallScriptResource -Wait -Verbose

Get-DscConfiguration

Notice the Write-Verbose statements. These are important for logging and output when the resource runs.

Notice that the GetScript, TestScript, and SetScript do not use a variable (except $true and $false). You could use custom variables in this code, for example, to reference parameters. See Example 2 of the Script resource documentation for how to use the $using: prefix with variables in a Script resource.

Take it to the next level.

There are many directions you could take this example:

• Add a parameter for Enable vs. Disable.
• Add a parameter for a specific firewall profile.
• Filter the output of NETSH to only show the profile status portion (netsh advfirewall show allprofiles | Where-Object {$_ -like ‘*Profile*’ -or $_ -like ‘State*’})
• Put the script resource into a composite resource to hide the Get/Test/Set code and keep your config looking clean.

Your Turn

Now it is your turn. Take this sample and implement your own script resources. Leave a comment below telling what you accomplished.

Best wishes on your adventure learning DSC!

The struggle is real.

Are you facing issues with PowerShell remoting and credentials? You remote into your jump box, but then any remoting beyond there gets a big red ACCESS DENIED. Maybe you’ve tried CredSSP, but people say that isn’t safe. Read today’s post for a completely legit, secure, safe, and easy way to enable Kerberos double hop for PowerShell remoting.

The Problem

It’s a tale as old as time:

• ServerA talks to ServerB
• ServerB talks to ServerC
• Access denied!

You would have better luck asking a cheerleader to the prom. We call this the kerberos double hop. Yeah, it’s like a dance.

The struggle is real. Just check out this forum post on PowerShell.org from last month. After many years of PowerShell remoting we are still searching for a secure method of passing credentials to that elusive ServerC.

Neo vs. the Architect

Many have come before you. Let’s look at some of the popular solutions for Kerberos double hop in PowerShell remoting:

Resource-Based Kerberos Constrained Delegation

Every release of Windows Server packs tons of new features, many that do not make big headlines. In my opinion this solution has been around four years now, and no one has uncovered its use for PowerShell remoting. I have researched this topic thoroughly, and I have not found anyone else online documenting this feature as a solution for PowerShell remoting.

Windows Server 2000 included Kerberos delegation (unconstrained). This allowed ServerB to delegate credentials anywhere else in the domain. Not good.

Windows Server 2003 modified this concept to constrained delegation, limiting delegation from ServerB to only designated service principal names (SPNs) on ServerC. This was much better, and it is common practice today. Many have struggled to get this working for PowerShell remoting, since it is not a documented solution.

Windows Server 2012 simplified the design by instead configuring the delegation on the computer object of ServerC, called resource-based delegation, specifying from whom it will receive delegated credentials. The same attribute can be set for user accounts and service accounts as well.

Resource-based Kerberos constrained delegation requires Windows Server 2012 or above for the servers involved, including at least one 2012 domain controller in each related domain. I am not going to include all of the details in this post, because the technology is well-documented. Read the links at the bottom of the article for all the particulars.

Show me some `Shell

This code for setting up the permissions requires a Windows box with the Windows Server 2012 Active Directory PowerShell RSAT available.

PS C:\> Add-WindowsFeature RSAT-AD-PowerShell

PS C:\> Import-Module ActiveDirectory

PS C:\> Get-Command -ParameterName PrincipalsAllowedToDelegateToAccount

CommandType Name                 ModuleName
----------- ----                 ----------
Cmdlet      New-ADComputer       ActiveDirectory
Cmdlet      New-ADServiceAccount ActiveDirectory
Cmdlet      New-ADUser           ActiveDirectory
Cmdlet      Set-ADComputer       ActiveDirectory
Cmdlet      Set-ADServiceAccount ActiveDirectory
Cmdlet      Set-ADUser           ActiveDirectory

Notice that some of the Active Directory cmdlets for Windows Server 2012 and above include a new parameter PrincipalsAllowedToDelegateToAccount. This parameter sets the Active Directory object attribute msDS-AllowedToActOnBehalfOfOtherIdentity. That attribute actually holds an access control list (ACL) determining who has permissions to delegate credentials to ServerC.

# Set up variables for reuse
$ServerA = $env:COMPUTERNAME
$ServerB = Get-ADComputer -Identity ServerB
$ServerC = Get-ADComputer -Identity ServerC

# Notice the StartName property of the WinRM Service: NT AUTHORITY\NetworkService
# This looks like the ServerB computer account when accessing other servers over the network.
Get-WmiObject Win32_Service -Filter 'Name="winrm"' -ComputerName $ServerB.name | fl *

The WinRM service by default runs as the NetworkService account. Therefore, we will allow ServerC to receive the computer object of ServerB for delegation.

# Grant resource-based Kerberos constrained delegation
Set-ADComputer -Identity $ServerC -PrincipalsAllowedToDelegateToAccount $ServerB

# Check the value of the attribute directly
$x = Get-ADComputer -Identity $ServerC -Properties msDS-AllowedToActOnBehalfOfOtherIdentity
$x.'msDS-AllowedToActOnBehalfOfOtherIdentity'.Access

# Check the value of the attribute indirectly
Get-ADComputer -Identity $ServerC -Properties PrincipalsAllowedToDelegateToAccount

The output of the ACL Access property shows a simple access control entry (ACE) for ServerB to delegate credentials to ServerC:

ActiveDirectoryRights : GenericAll
InheritanceType       : None
ObjectType            : 00000000-0000-0000-0000-000000000000
InheritedObjectType   : 00000000-0000-0000-0000-000000000000
ObjectFlags           : None
AccessControlType     : Allow
IdentityReference     : CONTOSO\ServerB$
IsInherited           : False
InheritanceFlags      : None
PropagationFlags      : None

Here is the one snag. The KDC has a 15 min SPN negative cache. If ServerB has already tried to talk to ServerC, then there is a negative cache entry. You need to clear the cache on ServerB using one of the following techniques:

1. klist purge -li 0x3e7 (preferred and fastest method)
2. Wait 15 minutes for the cache to clear automatically.
3. Reboot ServerB.

Invoke-Command -ComputerName $ServerB.Name -Credential $cred -ScriptBlock {
    klist purge -li 0x3e7
}

or

Restart-Computer $ServerB.Name -Force -Wait -For WinRM

Once that step is complete we can successfully run code like this from ServerA through ServerB to ServerC:

# Capture a credential
$cred = Get-Credential Contoso\Alice

# Test kerberos double hop
Invoke-Command -ComputerName $ServerB.Name -Credential $cred -ScriptBlock {
    Test-Path \\$($using:ServerC.Name)\C$
    Get-Process lsass -ComputerName $($using:ServerC.Name)
    Get-EventLog -LogName System -Newest 3 -ComputerName $($using:ServerC.Name)
}

Note that the $using variable prefix allows ServerB to reference the $ServerC variable that lives in memory on ServerA. This makes the code entirely flexible. Just modify the $ServerB and $ServerC variables above with the computer names you want to use. Read more about $using in about_Remote_Variables.

You may want to allow multiple servers to delegate credentials to ServerC. In that case, set the parameter to an array of computer or user objects like this:

# Set up variables for each server
$ServerB1 = Get-ADComputer -Identity ServerB1
$ServerB2 = Get-ADComputer -Identity ServerB2
$ServerB3 = Get-ADComputer -Identity ServerB3
$ServerC  = Get-ADComputer -Identity ServerC

# Grant resource-based Kerberos constrained delegation
Set-ADComputer -Identity $ServerC `
    -PrincipalsAllowedToDelegateToAccount @($ServerB1,$ServerB2,$ServerB3)

According to this whitepaper include the domain controller FQDN in the Server parameter of the Get-ADComputer command to make it work across domains:

# For ServerC in Contoso domain and ServerB in other domain
$ServerB = Get-ADComputer -Identity ServerB -Server dc1.alpineskihouse.com
$ServerC = Get-ADComputer -Identity ServerC
Set-ADComputer -Identity $ServerC -PrincipalsAllowedToDelegateToAccount $ServerB

To undo the configuration, simply reset ServerC’s attribute to null.

Set-ADComputer -Identity $ServerC -PrincipalsAllowedToDelegateToAccount $null

A Practical Example

You have a jump box server that you connect to for daily administration. From that server you access all the other servers in your environment. The jump box server would be ServerB, so all the other servers in your environment (ServerC) would need ServerB allowed. Here is a code sample to query servers from an OU and set them all for resource-based Kerberos constrained delegation:

$ServerB = Get-ADComputer -Identity JumpBox
$Servers = Get-ADComputer -Filter {Name -ne $ServerB.Name} `
    -SearchBase 'OU=Servers,OU=NA,DC=contoso,DC=com' -SearchScope Subtree
ForEach ($ServerC in $Servers) {
    Set-ADComputer -Identity $ServerC -PrincipalsAllowedToDelegateToAccount $ServerB
}

Summary

Kerberos double hop for PowerShell remoting can now be solved with one simple cmdlet:

$ServerB = Get-ADComputer -Identity ServerB
$ServerC = Get-ADComputer -Identity ServerC
Set-ADComputer -Identity $ServerC -PrincipalsAllowedToDelegateToAccount $ServerB
# Then on ServerB: KLIST PURGE -LI 0x3e7

The benefits are many:

• No PowerShell code modification.
• No more SPNs for constrained delegation!
• Credentials are not stored on ServerB.
• Multiple domains and forests supported across trusts.
• Easier setup and administration.
• ServerA can now talk to ServerC through ServerB.

Once again the world is a happy place. Now go try it for yourself. Use the comment area below for feedback. Let me know how it goes.

Resource-Based Kerberos Constrained Delegation Links

What’s New in Kerberos Authentication
Resource-based constrained delegation across domains and forest
https://technet.microsoft.com/en-us/library/hh831747.aspx

How Windows Server 2012 Eases the Pain of Kerberos Constrained Delegation, Part 1
http://windowsitpro.com/security/how-windows-server-2012-eases-pain-kerberos-constrained-delegation-part-1
“Constrained delegation in Server 2012 introduces the concept of controlling delegation of service tickets using a security descriptor rather than an allow list of SPNs. This change simplifies delegation by enabling the resource to determine which security principals are allowed to request tickets on behalf of another user.”
“Resource-based constrained delegation functions correctly regardless of domain functional level and number of domain controllers (DCs) running a version of Windows Server prior to Server 2012, provided you have at least one Server 2012 DC in the same domain as the front-end server and one Server 2012 DC in the domain hosting the back-end server.”

How Windows Server 2012 Eases the Pain of Kerberos Constrained Delegation, Part 2
http://windowsitpro.com/security/how-windows-server-2012-eases-pain-kerberos-constrained-delegation-part-2

Understanding Kerberos Constrained Delegation for Azure Active Directory Application Proxy Deployments with Integrated Windows Authentication
http://aka.ms/kcdpaper

[MS-ADA2]: Active Directory Schema Attributes M
2.210 Attribute msDS-AllowedToActOnBehalfOfOtherIdentity
https://msdn.microsoft.com/en-us/library/hh554126.aspx
“This attribute is used for access checks to determine if a requestor has permission to act on the behalf of other identities to services running as this account.”

[MS-SFU]: Kerberos Protocol Extensions: Service for User and Constrained Delegation Protocol
1.3.2 S4U2proxy
https://msdn.microsoft.com/en-us/library/cc246079.aspx

Resource Based Kerberos Constrained Delegation
https://blog.kloud.com.au/2013/07/11/kerberos-constrained-delegation/

Remote Administration Without Constrained Delegation Using PrincipalsAllowedToDelegateToAccount
https://blogs.msdn.microsoft.com/taylorb/2012/11/06/remote-administration-without-constrained-delegation-using-principalsallowedtodelegatetoaccount/

When All Else Fails… $using:cred

If you still have no Windows Server 2012 domain controllers in your environment then you can use this technique:

# This works without delegation, passing fresh creds
# Note $Using:Cred in nested request
$cred = Get-Credential Contoso\Administrator
Invoke-Command -ComputerName ServerB -Credential $cred -ScriptBlock {
    hostname
    Invoke-Command -ComputerName ServerC -Credential $Using:cred -ScriptBlock {hostname}
}

This works, because the $using:cred passes a fresh copy of the credential variable into the remoting session without storing it anywhere. You could also swap out the Invoke-Command cmdlets with Enter-PSSession. The only requirement is WMF 3.0 or above on your servers.

Edit 8/31/16: Thanks to fellow PFE Martin Schvartzman for the KLIST PURGE syntax!
Edit 9/26/16: Added a code sample for the $using:cred alternative.

Tired of hacking away at RegEx and string functions to parse text? This post is for you!

New toys

PowerShell 5.x includes a number of new features. One of the lesser-known and incredibly powerful is the string conversion set of cmdlets. The names are very similar. Check out how Get-Help describes them:

PS C:\> Get-Help Convert*-String | Format-Table Name,Synopsis -AutoSize

Name               Synopsis
----               --------
Convert-String     Formats a string to match examples.
ConvertFrom-String Extracts and parses structured objects from string content.

Convert-String

I will admit that doesn’t sound very interesting on first glance. But check out what they can do! Let’s start with a simple example of Convert-String:

'Alice Alpha <alice.alpha@contoso.com>;' |
Convert-String -Example 'Alice Alpha <alice.alpha@contoso.com>;=Alice,Alpha,alice.alpha@contoso.com'

The -Example parameter is a string set up as SampleData=DesiredFormat. The above example will parse Alice’s name and email into clean CSV data. It doesn’t always have to be CSV, but that is what I need for this use case.

Alice,Alpha,alice.alpha@contoso.com

1up

Now let’s take a list of email addresses pasted from Outlook. Notice that we pipe them to –Split, giving us an array for string conversion split on new lines:

$Emails = @"
Alice Alpha <alice.alpha@contoso.com>;
Bob Beta <bob.beta@contoso.com>;
Charlie Charles <charlie.charles@contoso.com>;
Doug Delta <doug.delta@contoso.com>
"@ -Split "`n"

$Emails |
Convert-String -Example 'Alice Alpha <alice.alpha@contoso.com>;=Alice,Alpha,alice.alpha@contoso.com'

This code turns a list of email addresses into nice CSV data. Because the addresses are all formatted similarly, the one example of Alice shows how to parse all the lines.

Alice,Alpha,alice.alpha@contoso.com
Bob,Beta,bob.beta@contoso.com
Charlie,Charles,charlie.charles@contoso.com
Doug,Delta,doug.delta@contoso.com

CSV baby!

Let’s add ConvertFrom-CSV to include some headers.

$Emails = @"
Alice Alpha <alice.alpha@contoso.com>;
Bob Beta <bob.beta@contoso.com>;
Charlie Charles <charlie.charles@contoso.com>;
Doug Delta <doug.delta@contoso.com>
"@ -Split "`n"

$Emails |
Convert-String -Example 'Alice Alpha <alice.alpha@contoso.com>;=Alice,Alpha,alice.alpha@contoso.com' |
ConvertFrom-Csv -Header 'FirstName','LastName','Email'

Now we have clean PowerShell object output for a list of email addresses pasted from Outlook. Cool!

FirstName LastName Email
--------- -------- -----
Alice     Alpha    alice.alpha@contoso.com
Bob       Beta     bob.beta@contoso.com
Charlie   Charles  charlie.charles@contoso.com
Doug      Delta    doug.delta@contoso.com

Convert-String works with a single parsing example. If you have a variety of complex patterns to parse, then use ConvertFrom-String instead. We will look at that in tomorrow’s post.

Your turn…

I decided to keep this post short and leave other things for you to discover with the cmdlet. I have shown you one way to use it. There is more! Get-Help is your friend.

Now take this and go parse some of your own flat text use cases. Use the comments below to share your challenges and victories. Enjoy!