Posts on this page:

Simple JSON lexical parser in PowerShell (academic version)

Hey guys! I was silent for a while due to a lack of good topics to discuss. Today I want to present another piece of my class work at university for “Compiler Development” course. The task is to write a manual lexical parser for a language of my choice. I decided to take JSON language, because its syntax is relatively simple and requires most common techniques to parse. In addition, it has well-looking BNF grammar for custom parser implementations.

The purpose

The purpose of lexical analysis is to read the source code and convert them to a sequence of tokens (lexemes) which are minimal parts of each language. It is important understand that lexical analysis doesn’t perform semantic (meaning) validation. That is, lexical analysis determines whether the source code can be written in a specific language’s alphabet. It doesn’t mean that the code will be executed successfully. Source code semantic is validated only after lexical analysis and uses its product (a set of tables, keywords, operators, literals, identifiers, etc.).

You can think that there is no need to write your own lexical parser, because there are LOTS of them. For example, PowerShell contains built-in JSON encoder and decoder via ConvertTo-JSON and ConvertFrom-JSON cmdlets. Though, these cmdlets completely hide parsing result and perform object conversion. You can’t access internal parser to look at exact results of the parsing. But results of lexical parsers are actively used in web. For example, JS-based syntax highlighters use lexical parser to split the source code into tokens and colorize or highlight them for better readability. And my website does it as well (though, not via JS). For example, all XML and PowerShell code snippets on my blog are colorized by using lexical parsers. For PowerShell code I’m using Tokenize method in System.Management.Automation.PSParser class. For XML strings I’m using custom XML tokenizer. And cororize them according to token types.

Read more →

Working with certificate revocation lists (CRL) in PowerShell (part 2)

Hello S-1-1-0!

In previous post we gave an introduction into techniques to work with certificate revocation lists in PowerShell. We explored common steps to read CRL’s basic information, CRL extensions and revoked certificate collection. Today I will discuss about CRL handy shortcuts and signature validation.

Get CRL next publication date and number

In some environments, it is impossible to automatically copy CRLs from CA server to CRL distribution points or there is a scenario when PKI administrators run custom scripts to monitor CRL health status at CRL distribution points and update them if they are about to expire. For such purposes I maintain two shortcut methods to quickly identify required values.

CRL validity is determined by a NextUpdate field. If the current time passes that timestamp, the CRL is considered expired. To provide better validity handling, Microsoft use their own Next CRL Publish CRL extension. This extension contains a date/time value at which CA will issue new CRL. This value (when present) is always set prior to value in NextUpdate field to provide a time window to replicate newly published CRL across all distribution points prior existing CRLs expire. I have a good article on this subject: How ThisUpdate, NextUpdate and NextCRLPublish are calculated (v2). However, Next CRL Publish extension is presented in CRLs issued by Microsoft CAs and is absent in 3rd party CAs, as the result, next CRL publication date is determined solely by Next Update field. Moreover, there might be a case when CA is in the decommission process and issues its last CRL which is supposed to be valid infinitely.

To simplify all these validations, I developed a GetNextPublish method. The logic of this method is as follows:

  1. If Next CRL Publish extension is presented, extension’s value is returned.
  2. If Next CRL Publish extension is absent: a value in the NextUpdate field is returned. If return value is null, the CRL is valid infinitely and no updated are expected.
PS C:\> $crl.GetNextPublish()

otrdiena, 2016. gada 25. oktobri 20:00:00

PS C:\>

For monitoring purposes you may want to compare CRL numbers (they are numbered sequentially) across all CRL distribution points with a base value (for example, retrieved from CA database). This will ensure that there are no stale CRLs especially when CRL was published manually. For such purposes, I maintain a GetCRLNumber method:

PS C:\> $crl.GetCRLNumber()

If the CRL Number extension is absent, then zero value is returned, because such CRLs doesn’t have CRL numbering capabilities. This is the case for Version 1 CRLs, or non-conformant CAs.

Signature Validation

Let’s take a look into our CRL (from DigiCert):

PS C:\> $crl

Version             : 2
Type                : Base CRL
IssuerName          : System.Security.Cryptography.X509Certificates.X500DistinguishedName
Issuer              : CN=DigiCert SHA2 Secure Server CA, O=DigiCert Inc, C=US
ThisUpdate          : 2016.10.18. 20:14:33
NextUpdate          : 2016.10.25. 20:00:00
SignatureAlgorithm  : 1.2.840.113549.1.1.11 (sha256RSA)
Extensions          : { (Authority Key Identifier), (CRL Number), (Issuing Distribution Po
RevokedCertificates : {Serial number: 0c587cfa9bf443daeab70526d4bc009f revoked at: 2015.11.06. 21:57:32, Serial number:
                       0b3ba5097ac6f59b551a1338357a0981 revoked at: 2015.11.09. 11:22:51, Serial number: 0e213e45ff44bd
                      975d6d22cbb8a40f2d revoked at: 2015.11.09. 19:21:01, Serial number: 08d64d9f888feee694b32d06bba9f
                      b83 revoked at: 2015.11.09. 19:22:03...}
RawData             : {48, 131, 4, 111...}
Handle              : 0

PS C:\>

Here we see a SignatureAlgorithm property that stores the information about signature algorithm (captain obvious is here too). This information is used to validate the signature against issuer. Let’s take proper and invalid issuer certificates:

PS C:\> $issuer = New-Object Security.Cryptography.X509Certificates.X509Certificate2 C:\Certs\DigiCertSHA2SecureServerCA
PS C:\> $nonissuer = New-Object Security.Cryptography.X509Certificates.X509Certificate2 C:\Certs\verisign.cer
PS C:\> $issuer

Thumbprint                                Subject
----------                                -------
1FB86B1168EC743154062E8C9CC5B171A4B7CCB4  CN=DigiCert SHA2 Secure Server CA, O=DigiCert Inc, C=US

PS C:\> $nonissuer

Thumbprint                                Subject
----------                                -------
A5EC73D48C34FCBEF1005AEB85843524BBFAB727  OU=Class 2 Public Primary Certification Authority, O="VeriSign, Inc.", C=US

PS C:\>

And try to use both certificates to validate signature:

PS C:\> $crl.VerifySignature($issuer,$true)
PS C:\> $crl.VerifySignature($nonissuer,$true)
PS C:\>

We can observe that DigiCert is the issuer of this CRL, while VeriSign is not. Plain and simple. VerifySignature method accepts two arguments: an instance of X509Certificate2 class and boolean value that indicates whether to perform CRL issuer name validation along with raw signature validation. In this case, SubjectName field of the issuer (or candidate at that point) is compared with IssuerName property of the CRL. Moreover, binary comparison is performed. This was made to be compatible with RFC3280 (although, obsolete, but still in use) which requires binary match. RFC5280 allows case-insensitive string match.

When using CRLs in your code to validate revocation status, do not forget to validate CRL signature as this will make your code more reliable and trustworthy.

Currently, VerifySignature supports RSA, DSA and ECDSA signatures. Alternate signature algorithms (RSASSA-PSS) are not yet supported, but I do have plans to add support of alternate signature formats.

Interoperability with CryptoAPI

In the past, I used CryptoAPI native functions (through p/invoke) to decode CRL objects. Since I have a reliable and very robust ASN.1 parser, I moved CRL decoder to a fully managed .NET code without using a piece of unmanaged code. As the result, normally you don’t need to dispose CRL object in order to release unmanaged resources and prevent memory leaks. However, there might be a need to get an unmanaged reference to CRL object. For such purposes I maintain a Handle property that contains a reference to a CRL object in unmanaged memory. Normally, it is zero:

PS C:\> $crl.Handle

If you need an unmanaged reference, call GetSafeContext method. This method will copy CRL object into unmanaged memory and retrieve a pointer to that memory block:

PS C:\> $crl.GetSafeContext()

                                                  IsInvalid                                                    IsClosed
                                                  ---------                                                    --------
                                                      False                                                       False

PS C:\> $crl.Handle

Now, you can use this handle to pass into unmanaged functions. For example, into CertAddCRLContextToStore function to add CRL into Windows Certificate Store. When CRL handle is populated, it must be freed after you finish working with CRL object by calling ReleaseContext method:

PS C:\> $crl.ReleaseContext()
PS C:\> $crl.Handle
PS C:\>

The difference between ReleaseContext and Reset methods is simple: ReleaseContext releases only unmanaged resources, while keeping managed object alive. If there are no unmanaged resources associated with the current object, the method does nothing (no errors are thrown). Reset method releases unmanaged resources (if necessary) and clears managed object by clearing all fields and setting them to default null/empty values.

That’s all for today. In next post, I’ll talk about CRL generation in PowerShell, so stay tuned!

Working with certificate revocation lists (CRL) in PowerShell (part 1)

Hello everyone!

Today I would like to summarize techniques on working with X.509 certificate revocation lists (CRL) in PowerShell. There are a lot of examples in my weblog, but most of this information is provided as context-specific addition to work in a given article’s context. Before talking about the subject, I’d like to put few words about the reason of this blog post and why it is written in that way.


As PowerShell evolves, it starts to cover more and new areas. And this process continues since PowerShell birth (in 2006). Systems administrators become more critical to script functionality. Previously, if something was not doable at all, we skipped that thing. With PowerShell we are able to do much more things. Maybe, not natively, maybe not in an elegant manner (say, through complex parsing), maybe very ugly, but we can do that right now. If community is interested in some area, they will develop a framework to make things easier and available to everyone.

One big area I’m interesting in is public key infrastructure, CryptoAPI, certificates and everything related to them. This happened several years ago (I recall it was in 2009) when I already was a passionate PowerShell enthusiast. Unfortunately, I realized that PowerShell has very-very basic support of certificate-related stuff. Mostly, this is due to poor support from underlying .NET platform. As the result, I attempted to start my big project to integrate cryptography into PowerShell through PS module and standalone scripts. Apparently, I was one of the first PS enthusiasts who started cryptography integration into PowerShell. I made huge work during these 7 years, went through tons of mistakes, bad choices, misunderstandings, trials and probes. It took 7 years and still continues. As of now, I have developed one of the best PowerShell module to work with PKI I’m aware of: PowerShell PKI. It is not just a module, it is a whole framework, which offers additional functionality through .NET-style objects and methods you can call from PS console. All this makes me confident in PowerShell PKI area.

Read more →

Efficient way to get AD user membership recursively with PowerShell

The other day, one customer asked for a solution to get full user membership in Active Directory for audit purposes. The solution should retrieve not only direct group membership, but indirect (through group nesting) too. Although, the question is plain and simple, solution is very interesting from various perspectives.

At first, let illustrate a sample user and group membership diagram:

Active Directory group membership graph

Quick diagram observation suggests us that we have a directed graph (it is not a tree), where users and groups are vertexes and membership relations are directed edges. Arrows identify relationship direction.

Our graph contains two users, User1 and User2 and eight groups: G1G8. In a given case, User1 is direct member of groups G1, G2 and G3, User2 is direct member of G8 only. Group G1 is member of G4, G2 is direct member of G4 and G5 and so on. For description purposes I labeled all edges. This should be clear.

Read more →

Convert data between binary, hex and Base64 in PowerShell – Managed Edition

Some time ago I wrote a blog post about converting data between Hex, Base64 and binary in PowerShell by using CryptoAPI functionality: Convert data between binary, hex and Base64 in PowerShell. I was impressed by those functions, because CryptStringToBinary function is magical and is able to convert “messy” hex string sequence (with or without address and ASCII columns which are not part of the data) to a pure byte array. I wish they have a bit more flexibility and extensibility. For example, when supporting certificate issues, I receive dumps from 3rd party tools (OpenSSL and similar) and browsers. They use different delimiters to separate hex octets. One tool use minus signs, other use colons to separate hex octets:

0000 - 20 ab 34 00 ff 87 50 1e-de fb c9 3d 10 2f 7b fd    .4...P....=./{.
0010 - 99 a1 61 e0 3d 5f 93 82-63 e9 0a 6f 1a 22 4f 04   ..a.=_..c..o."O.

or this:


Unfortunately, CryptStringToBinary supports only whitespace characters as delimiter and I have to manually remove them from dump before converting to a byte array. So I decided to get my own converter with blackjack and hookers in managed language.

Read more →