Some time ago I wrote a blog post about converting data between Hex, Base64 and binary in PowerShell by using CryptoAPI functionality: Convert data between binary, hex and Base64 in PowerShell. I was impressed by those functions, because CryptStringToBinary function is magical and is able to convert “messy” hex string sequence (with or without address and ASCII columns which are not part of the data) to a pure byte array. I wish they have a bit more flexibility and extensibility. For example, when supporting certificate issues, I receive dumps from 3rd party tools (OpenSSL and similar) and browsers. They use different delimiters to separate hex octets. One tool use minus signs, other use colons to separate hex octets:
0000 - 20 ab 34 00 ff 87 50 1e-de fb c9 3d 10 2f 7b fd .4...P....=./{. 0010 - 99 a1 61 e0 3d 5f 93 82-63 e9 0a 6f 1a 22 4f 04 ..a.=_..c..o."O.
or this:
26:C0:29:E9:8C:AB:C3:9E:95:38:74:8A:87:D3:86:8D 5C:5A:BA:47:44:83:7E:CB:48:BE:DD:E5:39:51:24:42:C6:C5:60:8B DA:26:B8:C8:F4:04:3E:62:F3:7F:3B:EC:1D:9F:85:66:28:00:45:55:66: 15:FF:BB:37:77:97:59:F0:EC:0B:B6
Unfortunately, CryptStringToBinary supports only whitespace characters as delimiter and I have to manually remove them from dump before converting to a byte array. So I decided to get my own converter with blackjack and hookers in managed language.
The work was hard and I literally reverse-engineered CryptStringToBinary and CryptBinaryToString functions with help from Windows Cryptography God – Vic Heller. I got the whole picture and spend even more time in implementation in C#. Eventually, I wrote 700+ lines of C# code and got pretty well AsnFormatted.cs static class. This version was shipped with PowerShell PKI module v3.2.5. After working with large data I noticed that the code is not very fast. For example, it takes about 8 seconds to convert 16MB byte array to hex table and 13 seconds to convert it back:
PS C:\> $bytes.Length 16196687 PS C:\> Measure-Command {$hex = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"hex")} Days : 0 Hours : 0 Minutes : 0 Seconds : 8 Milliseconds : 948 Ticks : 89480181 TotalDays : 0,000103565024305556 TotalHours : 0,00248556058333333 TotalMinutes : 0,149133635 TotalSeconds : 8,9480181 TotalMilliseconds : 8948,0181 PS C:\> Measure-Command {[SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($hex,"hex")} Days : 0 Hours : 0 Minutes : 0 Seconds : 13 Milliseconds : 889 Ticks : 138897717 TotalDays : 0,000160761246527778 TotalHours : 0,00385826991666667 TotalMinutes : 0,231496195 TotalSeconds : 13,8897717 TotalMilliseconds : 13889,7717
I spend some more time on StackOverflow boards to identify where I get performance penalty. I got them from two sources: arrays (searching) and converters. Ok, I removed these parts and replaced with more effective manual converters (but got 100+ lines of code). You can take a look at the most recent version of AsnFormatter.cs file and which is shipped with PowerShell PKI Module v3.2.6. And the numbers now are much better:
PS C:\> $bytes.Length 16196687 PS C:\> Measure-Command {$hex = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"hex")} Days : 0 Hours : 0 Minutes : 0 Seconds : 0 Milliseconds : 478 Ticks : 4782426 TotalDays : 5,53521527777778E-06 TotalHours : 0,000132845166666667 TotalMinutes : 0,00797071 TotalSeconds : 0,4782426 TotalMilliseconds : 478,2426 PS C:\> Measure-Command {[SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($hex,"hex")} Days : 0 Hours : 0 Minutes : 0 Seconds : 2 Milliseconds : 443 Ticks : 24438001 TotalDays : 2,82847233796296E-05 TotalHours : 0,000678833361111111 TotalMinutes : 0,0407300016666667 TotalSeconds : 2,4438001 TotalMilliseconds : 2443,8001
16MB data to hex in just half second? Not so bad, not so bad. This is about performance. Now I want to take a brief look into functionality. The functionality is almost the same in unmanaged CryptoAPI functions. Let’s format some data into various Base64 and hex formats:
PS C:\> $base64 = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"base64") PS C:\> $base64 MIG9MIGYMAkGBSsOAwIaBQAwRzETMBEGCgmSJomT8ixkARkWA2NvbTEXMBUGCgmS JomT8ixkARkWB2NvbnRvc28xFzAVBgNVBAMTDmNvbnRvc28tREMyLUNBFw0xMDAz MDYxMTEwMzFaFw0xNTAzMDUxMTEwMzFaMCQwIgIDEjRWFw0xMjA3MjQxNDI3MDVa MAwwCgYDVR0VBAMKAQMwCQYFKw4DAhoFAAMVAA4N+FY48JTRcNkFdE8EXRKH3/Pf PS C:\> $base64header = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"base64header") PS C:\> $base64header -----BEGIN CERTIFICATE----- MIG9MIGYMAkGBSsOAwIaBQAwRzETMBEGCgmSJomT8ixkARkWA2NvbTEXMBUGCgmS JomT8ixkARkWB2NvbnRvc28xFzAVBgNVBAMTDmNvbnRvc28tREMyLUNBFw0xMDAz MDYxMTEwMzFaFw0xNTAzMDUxMTEwMzFaMCQwIgIDEjRWFw0xMjA3MjQxNDI3MDVa MAwwCgYDVR0VBAMKAQMwCQYFKw4DAhoFAAMVAA4N+FY48JTRcNkFdE8EXRKH3/Pf -----END CERTIFICATE----- PS C:\> $base64crlheader = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"base64crlheader") PS C:\> $base64crlheader -----BEGIN X509 CRL----- MIG9MIGYMAkGBSsOAwIaBQAwRzETMBEGCgmSJomT8ixkARkWA2NvbTEXMBUGCgmS JomT8ixkARkWB2NvbnRvc28xFzAVBgNVBAMTDmNvbnRvc28tREMyLUNBFw0xMDAz MDYxMTEwMzFaFw0xNTAzMDUxMTEwMzFaMCQwIgIDEjRWFw0xMjA3MjQxNDI3MDVa MAwwCgYDVR0VBAMKAQMwCQYFKw4DAhoFAAMVAA4N+FY48JTRcNkFdE8EXRKH3/Pf -----END X509 CRL----- PS C:\> $base64reqheader = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"base64requestheader") PS C:\> $base64reqheader -----BEGIN NEW CERTIFICATE REQUEST----- MIG9MIGYMAkGBSsOAwIaBQAwRzETMBEGCgmSJomT8ixkARkWA2NvbTEXMBUGCgmS JomT8ixkARkWB2NvbnRvc28xFzAVBgNVBAMTDmNvbnRvc28tREMyLUNBFw0xMDAz MDYxMTEwMzFaFw0xNTAzMDUxMTEwMzFaMCQwIgIDEjRWFw0xMjA3MjQxNDI3MDVa MAwwCgYDVR0VBAMKAQMwCQYFKw4DAhoFAAMVAA4N+FY48JTRcNkFdE8EXRKH3/Pf -----END NEW CERTIFICATE REQUEST----- PS C:\> $hex = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"hex").trimend() PS C:\> $hex 30 81 bd 30 81 98 30 09 06 05 2b 0e 03 02 1a 05 00 30 47 31 13 30 11 06 0a 09 92 26 89 93 f2 2c 64 01 19 16 03 63 6f 6d 31 17 30 15 06 0a 09 92 26 89 93 f2 2c 64 01 19 16 07 63 6f 6e 74 6f 73 6f 31 17 30 15 06 03 55 04 03 13 0e 63 6f 6e 74 6f 73 6f 2d 44 43 32 2d 43 41 17 0d 31 30 30 33 30 36 31 31 31 30 33 31 5a 17 0d 31 35 30 33 30 35 31 31 31 30 33 31 5a 30 24 30 22 02 03 12 34 56 17 0d 31 32 30 37 32 34 31 34 32 37 30 35 5a 30 0c 30 0a 06 03 55 1d 15 04 03 0a 01 03 30 09 06 05 2b 0e 03 02 1a 05 00 03 15 00 0e 0d f8 56 38 f0 94 d1 70 d9 05 74 4f 04 5d 12 87 df f3 df PS C:\> $hexraw = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"hexraw") PS C:\> $hexraw 3081bd308198300906052b0e03021a0500304731133011060a0992268993f22c6401191603636f6d31173015060a0992268993f22c6401191607636f 6e746f736f311730150603550403130e636f6e746f736f2d4443322d4341170d3130303330363131313033315a170d3135303330353131313033315a 302430220203123456170d3132303732343134323730355a300c300a0603551d1504030a0103300906052b0e03021a05000315000e0df85638f094d1 70d905744f045d1287dff3df PS C:\> $hexaddress = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"hexaddress").trimend() PS C:\> $hexaddress 0000 30 81 bd 30 81 98 30 09 06 05 2b 0e 03 02 1a 05 0010 00 30 47 31 13 30 11 06 0a 09 92 26 89 93 f2 2c 0020 64 01 19 16 03 63 6f 6d 31 17 30 15 06 0a 09 92 0030 26 89 93 f2 2c 64 01 19 16 07 63 6f 6e 74 6f 73 0040 6f 31 17 30 15 06 03 55 04 03 13 0e 63 6f 6e 74 0050 6f 73 6f 2d 44 43 32 2d 43 41 17 0d 31 30 30 33 0060 30 36 31 31 31 30 33 31 5a 17 0d 31 35 30 33 30 0070 35 31 31 31 30 33 31 5a 30 24 30 22 02 03 12 34 0080 56 17 0d 31 32 30 37 32 34 31 34 32 37 30 35 5a 0090 30 0c 30 0a 06 03 55 1d 15 04 03 0a 01 03 30 09 00a0 06 05 2b 0e 03 02 1a 05 00 03 15 00 0e 0d f8 56 00b0 38 f0 94 d1 70 d9 05 74 4f 04 5d 12 87 df f3 df PS C:\> $hexascii = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"hexascii").trimend() PS C:\> $hexascii 30 81 bd 30 81 98 30 09 06 05 2b 0e 03 02 1a 05 0..0..0...+..... 00 30 47 31 13 30 11 06 0a 09 92 26 89 93 f2 2c .0G1.0.....&..., 64 01 19 16 03 63 6f 6d 31 17 30 15 06 0a 09 92 d....com1.0..... 26 89 93 f2 2c 64 01 19 16 07 63 6f 6e 74 6f 73 &...,d....contos 6f 31 17 30 15 06 03 55 04 03 13 0e 63 6f 6e 74 o1.0...U....cont 6f 73 6f 2d 44 43 32 2d 43 41 17 0d 31 30 30 33 oso-DC2-CA..1003 30 36 31 31 31 30 33 31 5a 17 0d 31 35 30 33 30 06111031Z..15030 35 31 31 31 30 33 31 5a 30 24 30 22 02 03 12 34 5111031Z0$0"...4 56 17 0d 31 32 30 37 32 34 31 34 32 37 30 35 5a V..120724142705Z 30 0c 30 0a 06 03 55 1d 15 04 03 0a 01 03 30 09 0.0...U.......0. 06 05 2b 0e 03 02 1a 05 00 03 15 00 0e 0d f8 56 ..+............V 38 f0 94 d1 70 d9 05 74 4f 04 5d 12 87 df f3 df 8...p..tO.]..... PS C:\> $hexaddrascii = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"hexasciiaddress").trimend() PS C:\> $hexaddrascii 0000 30 81 bd 30 81 98 30 09 06 05 2b 0e 03 02 1a 05 0..0..0...+..... 0010 00 30 47 31 13 30 11 06 0a 09 92 26 89 93 f2 2c .0G1.0.....&..., 0020 64 01 19 16 03 63 6f 6d 31 17 30 15 06 0a 09 92 d....com1.0..... 0030 26 89 93 f2 2c 64 01 19 16 07 63 6f 6e 74 6f 73 &...,d....contos 0040 6f 31 17 30 15 06 03 55 04 03 13 0e 63 6f 6e 74 o1.0...U....cont 0050 6f 73 6f 2d 44 43 32 2d 43 41 17 0d 31 30 30 33 oso-DC2-CA..1003 0060 30 36 31 31 31 30 33 31 5a 17 0d 31 35 30 33 30 06111031Z..15030 0070 35 31 31 31 30 33 31 5a 30 24 30 22 02 03 12 34 5111031Z0$0"...4 0080 56 17 0d 31 32 30 37 32 34 31 34 32 37 30 35 5a V..120724142705Z 0090 30 0c 30 0a 06 03 55 1d 15 04 03 0a 01 03 30 09 0.0...U.......0. 00a0 06 05 2b 0e 03 02 1a 05 00 03 15 00 0e 0d f8 56 ..+............V 00b0 38 f0 94 d1 70 d9 05 74 4f 04 5d 12 87 df f3 df 8...p..tO.]..... PS C:\>
And you can convert them back to original byte array:
PS C:\> [SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($base64,"base64").length 192 PS C:\> [SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($base64crlheader,"base64crlheader").length 192 PS C:\> [SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($base64reqheader,"base64requestheader").length 192 PS C:\> [SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($hex,"hex").length 192 PS C:\> [SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($hexraw,"hexraw").length 192 PS C:\> [SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($hexaddress,"hexaddress").length 192 PS C:\> [SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($hexascii,"hexascii").length 192 PS C:\> [SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($hexaddrascii,"hexasciiaddress").length 192
I just prove that the code successfully decodes all text-encoded strings back to original byte array. I just show that all they return exactly 192 bytes.
The following delimiters are supported (AsnFormatter.cs, ln18): ' '
, '-'
, ':'
, '\t'
, '\n'
, '\r'
. If necessary, this list can be easily extended to support other characters.
Here is online class documentation:
If you are a PowerShell or C# user and have to deal with Base64 and various hex formats, you can find my library very handy. And everything is for free :)
HTH
Post your comment:
Comments: