Visualizing binary files with ImHex's DSL, the "pattern language"

I have a strong memory that AFL - american fuzzy lop the binary fuzzer had a feature similar to what this was doing based on the highlighted portions and screenshots. It wasn't the AFL status screen, it was (may have been a) third party app, and it would color code parts of the input files based on the outputs or whatever from afl's processing.

For example, there was a color key that explained that say, purple meant "magic bytes", like "0x4a46494600" for JFIF0, and if any part of the input file caused errors it meant it was probably a checksum and needed to be "fixed" so afl could properly fuzz all the functions in the source code.

I'm not super in to fuzzing or that realm anymore, so i doubt i could describe it better than i did, here. I clicked through to see if someone have leveraged the AFL stuff for use in another tool, which would be cool.

edit: i think it was afl-analyze - i had a go at the source code for aflplusplus:

> A nifty utility that grabs an input file and takes a stab at explaining its structure by observing how changes to it affect the execution path.

> Another tool in AFL++ is the afl-analyze tool. It takes an input file, attempts to sequentially flip bytes and observes the behavior of the tested program. It then color-codes the input based on which sections appear to be critical and which are not; while not bulletproof, it can often offer quick insights into complex file formats.

Other tools for parsing and analyzing binary data are listed here: https://github.com/dloss/binary-parsing

Great write up!

I looked at ImHex a good while back and I think I had some runtime issues or maybe even compilation issues and didn't dig deeper. Even though the definition language piqued my curiosity.

These days I tend to just use xxd, bless, ghex, or seldom wxHexEditor, depending on what I need. But ImHex looks really powerful, like it could replace all the GUI ones. I'm looking forward to giving it another go tomorrow.

Though these days I spend most of my time in wireshark, which is kind of a hex viewer in a way.

How does it manage with huge files? Does it try to load the entire thing into memory. I remember wxHexEditor being good for that, and even being able to open block devices directly and process memory IIRC. Might be getting mixed up with HxD.

The decompression and combining compressed with decompressed sections looks very cool. Is the decompression in memory or written to disk?

// TagRecord Tags[while(!std::mem::eof())];

This loop based length stuff is very cool too, though for large files I'd imagine it could be slow as it will need to iterate through all records to determine the offset for records at the end of the file.

To be fair, wireshark / pcap files have this problem too.

Kind of related, a tool that allows you to hand write ASCII-art-annotated hex dump files, while also able to generate the original binary file from such text file: https://github.com/netspooky/xx/blob/main/examples/elf.xx

I wasn’t aware that ImHex had this feature - perhaps I’ll try it!

I’ve been singing the praises of 010 Editor for years specifically because of its template and scripting features, the former of which is nearly identical to this DSL.

There's an ImHex WebAssembly build accessible online at: https://web.imhex.werwolv.net/.

Wow, I've never thought of it, but "syntax"-highlighting for binary files would be awesome.. e.g. "these bytes indicate the beginning of the next frame" (when talking about MP3/video files), maybe with mouseover support where it says e.g. "this value at this location indicates it's a $FOO variant of the file".

Anyone know of such a tool?

Whoa, a flashback! Many, many years ago I worked on online presentation software that dynamically assembled Flash clips from clip-arts / user-provided graphics and text. There was a lot of parsing and reassembling of SWF involved. There used to be Swfmill library, but it didn't have enough stuff in it to deal with animations / transformations. Sigh that was a fun project.

My approach at the time was to have Org-mode tables detailing the meaning and composition of chunks of binary data. That wasn't nearly as comfortable as this editor seems to be, but I was told that it looked very impressive to people looking over my shoulder. Just like in those "hackers movies" where the screen is filled with gibberish sequences of digits and other random characters :)

Wireshark has a similar feature where you can open an ELF, or PNG and look at the sections. The LUA interface isn't to shabby either to write such "dissectors".

Looks slightly more expressive than Kaitai's binary format DSL.