Convert data between binary, hex and Base64 in PowerShell

Convert data between binary, hex and Base64 in PowerShell – Managed Edition

Some time ago I wrote a blog post about converting data between Hex, Base64 and binary in PowerShell by using CryptoAPI functionality: Convert data between binary, hex and Base64 in PowerShell. I was impressed by those functions, because CryptStringToBinary function is magical and is able to convert “messy” hex string sequence (with or without address and ASCII columns which are not part of the data) to a pure byte array. I wish they have a bit more flexibility and extensibility. For example, when supporting certificate issues, I receive dumps from 3rd party tools (OpenSSL and similar) and browsers. They use different delimiters to separate hex octets. One tool use minus signs, other use colons to separate hex octets:

0000 - 20 ab 34 00 ff 87 50 1e-de fb c9 3d 10 2f 7b fd    .4...P....=./{.
0010 - 99 a1 61 e0 3d 5f 93 82-63 e9 0a 6f 1a 22 4f 04   ..a.=_..c..o."O.

or this:

26:C0:29:E9:8C:AB:C3:9E:95:38:74:8A:87:D3:86:8D
5C:5A:BA:47:44:83:7E:CB:48:BE:DD:E5:39:51:24:42:C6:C5:60:8B
DA:26:B8:C8:F4:04:3E:62:F3:7F:3B:EC:1D:9F:85:66:28:00:45:55:66:
15:FF:BB:37:77:97:59:F0:EC:0B:B6

Unfortunately, CryptStringToBinary supports only whitespace characters as delimiter and I have to manually remove them from dump before converting to a byte array. So I decided to get my own converter ~~with blackjack and hookers~~ in managed language.

The work was hard and I literally reverse-engineered CryptStringToBinary and CryptBinaryToString functions with help from Windows Cryptography God – Vic Heller. I got the whole picture and spend even more time in implementation in C#. Eventually, I wrote 700+ lines of C# code and got pretty well AsnFormatted.cs static class. This version was shipped with PowerShell PKI module v3.2.5. After working with large data I noticed that the code is not very fast. For example, it takes about 8 seconds to convert 16MB byte array to hex table and 13 seconds to convert it back:

PS C:\> $bytes.Length
16196687
PS C:\> Measure-Command {$hex = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"hex")}


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 8
Milliseconds      : 948
Ticks             : 89480181
TotalDays         : 0,000103565024305556
TotalHours        : 0,00248556058333333
TotalMinutes      : 0,149133635
TotalSeconds      : 8,9480181
TotalMilliseconds : 8948,0181



PS C:\> Measure-Command {[SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($hex,"hex")}


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 13
Milliseconds      : 889
Ticks             : 138897717
TotalDays         : 0,000160761246527778
TotalHours        : 0,00385826991666667
TotalMinutes      : 0,231496195
TotalSeconds      : 13,8897717
TotalMilliseconds : 13889,7717

I spend some more time on StackOverflow boards to identify where I get performance penalty. I got them from two sources: arrays (searching) and converters. Ok, I removed these parts and replaced with more effective manual converters (but got 100+ lines of code). You can take a look at the most recent version of AsnFormatter.cs file and which is shipped with PowerShell PKI Module v3.2.6. And the numbers now are much better:

PS C:\> $bytes.Length
16196687
PS C:\> Measure-Command {$hex = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"hex")}


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 478
Ticks             : 4782426
TotalDays         : 5,53521527777778E-06
TotalHours        : 0,000132845166666667
TotalMinutes      : 0,00797071
TotalSeconds      : 0,4782426
TotalMilliseconds : 478,2426



PS C:\> Measure-Command {[SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($hex,"hex")}


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 2
Milliseconds      : 443
Ticks             : 24438001
TotalDays         : 2,82847233796296E-05
TotalHours        : 0,000678833361111111
TotalMinutes      : 0,0407300016666667
TotalSeconds      : 2,4438001
TotalMilliseconds : 2443,8001

16MB data to hex in just half second? Not so bad, not so bad. This is about performance. Now I want to take a brief look into functionality. The functionality is almost the same in unmanaged CryptoAPI functions. Let’s format some data into various Base64 and hex formats:

PS C:\> $base64 = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"base64")
PS C:\> $base64
MIG9MIGYMAkGBSsOAwIaBQAwRzETMBEGCgmSJomT8ixkARkWA2NvbTEXMBUGCgmS
JomT8ixkARkWB2NvbnRvc28xFzAVBgNVBAMTDmNvbnRvc28tREMyLUNBFw0xMDAz
MDYxMTEwMzFaFw0xNTAzMDUxMTEwMzFaMCQwIgIDEjRWFw0xMjA3MjQxNDI3MDVa
MAwwCgYDVR0VBAMKAQMwCQYFKw4DAhoFAAMVAA4N+FY48JTRcNkFdE8EXRKH3/Pf

PS C:\> $base64header = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"base64header")
PS C:\> $base64header
-----BEGIN CERTIFICATE-----
MIG9MIGYMAkGBSsOAwIaBQAwRzETMBEGCgmSJomT8ixkARkWA2NvbTEXMBUGCgmS
JomT8ixkARkWB2NvbnRvc28xFzAVBgNVBAMTDmNvbnRvc28tREMyLUNBFw0xMDAz
MDYxMTEwMzFaFw0xNTAzMDUxMTEwMzFaMCQwIgIDEjRWFw0xMjA3MjQxNDI3MDVa
MAwwCgYDVR0VBAMKAQMwCQYFKw4DAhoFAAMVAA4N+FY48JTRcNkFdE8EXRKH3/Pf
-----END CERTIFICATE-----

PS C:\> $base64crlheader = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"base64crlheader")
PS C:\> $base64crlheader
-----BEGIN X509 CRL-----
MIG9MIGYMAkGBSsOAwIaBQAwRzETMBEGCgmSJomT8ixkARkWA2NvbTEXMBUGCgmS
JomT8ixkARkWB2NvbnRvc28xFzAVBgNVBAMTDmNvbnRvc28tREMyLUNBFw0xMDAz
MDYxMTEwMzFaFw0xNTAzMDUxMTEwMzFaMCQwIgIDEjRWFw0xMjA3MjQxNDI3MDVa
MAwwCgYDVR0VBAMKAQMwCQYFKw4DAhoFAAMVAA4N+FY48JTRcNkFdE8EXRKH3/Pf
-----END X509 CRL-----

PS C:\> $base64reqheader = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"base64requestheader")
PS C:\> $base64reqheader
-----BEGIN NEW CERTIFICATE REQUEST-----
MIG9MIGYMAkGBSsOAwIaBQAwRzETMBEGCgmSJomT8ixkARkWA2NvbTEXMBUGCgmS
JomT8ixkARkWB2NvbnRvc28xFzAVBgNVBAMTDmNvbnRvc28tREMyLUNBFw0xMDAz
MDYxMTEwMzFaFw0xNTAzMDUxMTEwMzFaMCQwIgIDEjRWFw0xMjA3MjQxNDI3MDVa
MAwwCgYDVR0VBAMKAQMwCQYFKw4DAhoFAAMVAA4N+FY48JTRcNkFdE8EXRKH3/Pf
-----END NEW CERTIFICATE REQUEST-----

PS C:\> $hex = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"hex").trimend()
PS C:\> $hex
30 81 bd 30 81 98 30 09  06 05 2b 0e 03 02 1a 05
00 30 47 31 13 30 11 06  0a 09 92 26 89 93 f2 2c
64 01 19 16 03 63 6f 6d  31 17 30 15 06 0a 09 92
26 89 93 f2 2c 64 01 19  16 07 63 6f 6e 74 6f 73
6f 31 17 30 15 06 03 55  04 03 13 0e 63 6f 6e 74
6f 73 6f 2d 44 43 32 2d  43 41 17 0d 31 30 30 33
30 36 31 31 31 30 33 31  5a 17 0d 31 35 30 33 30
35 31 31 31 30 33 31 5a  30 24 30 22 02 03 12 34
56 17 0d 31 32 30 37 32  34 31 34 32 37 30 35 5a
30 0c 30 0a 06 03 55 1d  15 04 03 0a 01 03 30 09
06 05 2b 0e 03 02 1a 05  00 03 15 00 0e 0d f8 56
38 f0 94 d1 70 d9 05 74  4f 04 5d 12 87 df f3 df
PS C:\> $hexraw = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"hexraw")
PS C:\> $hexraw
3081bd308198300906052b0e03021a0500304731133011060a0992268993f22c6401191603636f6d31173015060a0992268993f22c6401191607636f
6e746f736f311730150603550403130e636f6e746f736f2d4443322d4341170d3130303330363131313033315a170d3135303330353131313033315a
302430220203123456170d3132303732343134323730355a300c300a0603551d1504030a0103300906052b0e03021a05000315000e0df85638f094d1
70d905744f045d1287dff3df
PS C:\> $hexaddress = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"hexaddress").trimend()
PS C:\> $hexaddress
0000    30 81 bd 30 81 98 30 09  06 05 2b 0e 03 02 1a 05
0010    00 30 47 31 13 30 11 06  0a 09 92 26 89 93 f2 2c
0020    64 01 19 16 03 63 6f 6d  31 17 30 15 06 0a 09 92
0030    26 89 93 f2 2c 64 01 19  16 07 63 6f 6e 74 6f 73
0040    6f 31 17 30 15 06 03 55  04 03 13 0e 63 6f 6e 74
0050    6f 73 6f 2d 44 43 32 2d  43 41 17 0d 31 30 30 33
0060    30 36 31 31 31 30 33 31  5a 17 0d 31 35 30 33 30
0070    35 31 31 31 30 33 31 5a  30 24 30 22 02 03 12 34
0080    56 17 0d 31 32 30 37 32  34 31 34 32 37 30 35 5a
0090    30 0c 30 0a 06 03 55 1d  15 04 03 0a 01 03 30 09
00a0    06 05 2b 0e 03 02 1a 05  00 03 15 00 0e 0d f8 56
00b0    38 f0 94 d1 70 d9 05 74  4f 04 5d 12 87 df f3 df
PS C:\> $hexascii = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"hexascii").trimend()
PS C:\> $hexascii
30 81 bd 30 81 98 30 09  06 05 2b 0e 03 02 1a 05   0..0..0...+.....
00 30 47 31 13 30 11 06  0a 09 92 26 89 93 f2 2c   .0G1.0.....&...,
64 01 19 16 03 63 6f 6d  31 17 30 15 06 0a 09 92   d....com1.0.....
26 89 93 f2 2c 64 01 19  16 07 63 6f 6e 74 6f 73   &...,d....contos
6f 31 17 30 15 06 03 55  04 03 13 0e 63 6f 6e 74   o1.0...U....cont
6f 73 6f 2d 44 43 32 2d  43 41 17 0d 31 30 30 33   oso-DC2-CA..1003
30 36 31 31 31 30 33 31  5a 17 0d 31 35 30 33 30   06111031Z..15030
35 31 31 31 30 33 31 5a  30 24 30 22 02 03 12 34   5111031Z0$0"...4
56 17 0d 31 32 30 37 32  34 31 34 32 37 30 35 5a   V..120724142705Z
30 0c 30 0a 06 03 55 1d  15 04 03 0a 01 03 30 09   0.0...U.......0.
06 05 2b 0e 03 02 1a 05  00 03 15 00 0e 0d f8 56   ..+............V
38 f0 94 d1 70 d9 05 74  4f 04 5d 12 87 df f3 df   8...p..tO.].....
PS C:\> $hexaddrascii = [SysadminsLV.Asn1Parser.AsnFormatter]::BinaryToString($bytes,"hexasciiaddress").trimend()
PS C:\> $hexaddrascii
0000    30 81 bd 30 81 98 30 09  06 05 2b 0e 03 02 1a 05   0..0..0...+.....
0010    00 30 47 31 13 30 11 06  0a 09 92 26 89 93 f2 2c   .0G1.0.....&...,
0020    64 01 19 16 03 63 6f 6d  31 17 30 15 06 0a 09 92   d....com1.0.....
0030    26 89 93 f2 2c 64 01 19  16 07 63 6f 6e 74 6f 73   &...,d....contos
0040    6f 31 17 30 15 06 03 55  04 03 13 0e 63 6f 6e 74   o1.0...U....cont
0050    6f 73 6f 2d 44 43 32 2d  43 41 17 0d 31 30 30 33   oso-DC2-CA..1003
0060    30 36 31 31 31 30 33 31  5a 17 0d 31 35 30 33 30   06111031Z..15030
0070    35 31 31 31 30 33 31 5a  30 24 30 22 02 03 12 34   5111031Z0$0"...4
0080    56 17 0d 31 32 30 37 32  34 31 34 32 37 30 35 5a   V..120724142705Z
0090    30 0c 30 0a 06 03 55 1d  15 04 03 0a 01 03 30 09   0.0...U.......0.
00a0    06 05 2b 0e 03 02 1a 05  00 03 15 00 0e 0d f8 56   ..+............V
00b0    38 f0 94 d1 70 d9 05 74  4f 04 5d 12 87 df f3 df   8...p..tO.].....
PS C:\>

And you can convert them back to original byte array:

PS C:\> [SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($base64,"base64").length
192
PS C:\> [SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($base64crlheader,"base64crlheader").length
192
PS C:\> [SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($base64reqheader,"base64requestheader").length
192
PS C:\> [SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($hex,"hex").length
192
PS C:\> [SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($hexraw,"hexraw").length
192
PS C:\> [SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($hexaddress,"hexaddress").length
192
PS C:\> [SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($hexascii,"hexascii").length
192
PS C:\> [SysadminsLV.Asn1Parser.AsnFormatter]::StringToBinary($hexaddrascii,"hexasciiaddress").length
192

I just prove that the code successfully decodes all text-encoded strings back to original byte array. I just show that all they return exactly 192 bytes.

The following delimiters are supported (AsnFormatter.cs, ln18): ' ', '-', ':', '\t', '\n', '\r'. If necessary, this list can be easily extended to support other characters.

Here is online class documentation:

AsnFormatter class
EncodingFormat enumeration
EncodingType enumeration

If you are a PowerShell or C# user and have to deal with Base64 and various hex formats, you can find my library very handy. And everything is for free :)

HTH

Comments:

Post your comment: