Moderator's introduction
You can't always tell, generally, who's your friend and who's your enemy. And friends, like enemies, can be quite unexpected. Our next speaker's friend became the decompiler, and his enemy — documentation. How all of that works, Maxim Sukhanov will tell us. Let's welcome him with applause.
[applause]
Maxim, here's your microphone. The clicker is lagging a bit, let me check. There, take it, please, go ahead.
Talk and Q&A
Hi everyone. I was asked to liven up the audience a bit, so I promise that during the talk I won't say the word "business" even once. When answering a question, I might say it. That's how I'll try to make the talk a bit less tedious.
The topic of the talk, well, you know it; there's a bit about me on the slide, that is, I do cybersecurity incident response, incident investigation, I write various software for incident response and forensics, an NTFS parser, registry, and so on. The problem. The problem is this: how can a person, a forensic expert, as the ideal example, justify some interpretation of some artifacts, traces, or, worse still, anomalies. That is, there's something a person observes when looking at computer data, and how do they explain what happened there. There are basically two approaches: we see some timestamp, and as to what that timestamp is, we refer to an article, a book, documentation, or someone else's results, that is, we say we interpret these bytes this way, and why do we do that, because there are other results out there, someone else's experience.
We can also do some kind of expert experiment and obtain that experience ourselves. That is, the interpretation of some timestamp can be found on our own. I underlined the word "documentation", because from here on we'll specifically look at the example with documentation. In our field there are a number of Telegram chats where people talk, and about a year ago there was a discussion on the topic of this very talk. And there was this small conclusion, which I've shortened further, that if a forensic expert says "I trust the documentation", then in principle that approach can be considered perfectly valid in some situations. Let's see why that's a problem, not a solution.
Vendors, that is, software manufacturers, that is, the very same Microsoft corporation, can deceive people. Deceive, in quotation marks, naturally. It's not deception done out of malice, but an untruth that's communicated for some other reasons. If we open the Microsoft website, the documentation, and look at how they describe the $STANDARD_INFORMATION attribute in NTFS, we'll see that there's no such concept there as timestamps. That is, we see the declaration of the corresponding structure in C, we see the text description, and we see that the places that are used for storing timestamps are reserved. Accordingly, the conclusion: if we blindly trust the documentation, there are no timestamps in the $STANDARD_INFORMATION attribute. Naturally, this is an extreme example. That is, we're trying to refute the original thesis by reducing it to absurdity, but a real absurdity. That is, it's clear here that this can't be the case.
We can also turn to the analogous documentation, the $FILE_NAME attribute, and see that apart from the file name, the file name length and flags there's nothing there, no timestamps at all. There's also a field that references the file record's parent directory. Again, none of this matches reality. It's all deception. But deception in quotation marks, because it has what you'd call a valid reason. The vendor is protecting its internal structures from so-called clumsy hands. That is, it deliberately hides part of the internals, the internals that exist in its operating system, so that people who have no business in there don't poke around. That is, programmer Vasya Pupkin has no business in there, let him get his information from other sources.
And because of that, and apparently only that, the documentation is scrubbed, so the fields you don't need are marked as reserved. This is an extreme example, this is, so to speak, the greatest absurdity one could come across. And I can't, unfortunately, give a comparable one. That is, this is the maximum degree of this kind of mismatch with reality. Also, a vendor that ships some piece of software may forget that its code changed long ago and no longer works that way. And because of that forgetfulness, the documentation was never updated. Here's, let's say, a sore example. There are shadow copies. Ransomware,
when it encrypts data, usually deletes all the shadow copies first. But in some cases, for instance if the encryption was launched over the network, via shares, the shadow copies aren't deleted. And they may survive. If we look inside such a shadow copy, we see that in many cases, starting from Windows 8, instead of user documents, some browser history files and the like, we see a mess of zero bytes. And this effect is seen quite often. That is, people who investigate ransomware attacks know this kind of thing happens.
If you go to the documentation, for the Volume Shadow Copy service, you'll see this shouldn't happen. That is, shadow copies capture all the allocated space of the file system, and you can exclude something user-related from them, but only by setting a special registry value. By default, everything should be included. On the left, an example in a hex editor: from the current state of the file system, a fragment of browser history, Chrome, and below it the same file extracted from the shadow copy. At the top there's data, meaning the file system's current state has data, but the shadow copy has none. That is, the whole space is filled with zero bytes.
And the answer to this riddle is simple. Starting from Windows 8, user files are excluded from the shadow copy. With an asterisk at the end. Since shadow copying is done in blocks with 16-kilobyte granularity, the real kind, based on 1024, while the default cluster size is 4 kilobytes, some overlap with user files can still occur, so part of the user data will still be found in shadow copies anyway. This behavior isn't documented anywhere, and what's, let's say, most wonderful is that Microsoft replied to at least two customers explaining this behavior. That is, they said, in one case, that your data that the ransomware encrypted can't be recovered, because user files aren't included in shadow copies. Thank you, goodbye.
Those replies were found online only once it was clear what to search for. That is, there's a keyword, it's called scoped, scoped shadow copy in full. So you can find a mention of someone citing the vendor's reply in some article.
Right. And this behavior comes as a complete, let's say, surprise to those who were hit. On the one hand, the shadow copies are there, per the documentation user files should be in them, that is, all kinds of Excel documents and so on, but in reality most of them most likely won't be there, if we're talking about Windows 8 and later.
How does this happen? First a full shadow copy is created, but since the copying works in copy-on-write mode, meaning the data isn't actually copied as some backup that constitutes the shadow copy, before it's modified, that is, copying happens as the data changes, that shadow copy doesn't immediately contain everything that will later be shown to the user.
And right after that, the srtasks.exe process starts a rather complex procedure, that walks the entire MFT file, looks for file records, reconstructs their paths and by extension, and not only that, excludes those files in the bitmap of what is subject to copying as it changes. So when user files are modified, the shadow copy won't contain the old version of the changed data. And this applies only to user files, system files, all the EXEs and DLLs. They'll be copied and will end up in the shadow copy in their old state, as required. This is done to reduce the size of shadow copies, since nowadays shadow copies are more a tool for restoring a working, healthy operating system than a backup tool.
On Windows Server versions it's different, and there everything works as it should. So this narrowing, so to speak, of the scope of data copied into the shadow copy doesn't happen there; instead everything works properly, just like Windows 7.
There's also the situation where a vendor documents a file system implementation other than the one it has. If we go back to the 90s and the early 2000s, we'll recall that besides Windows NT there was the 95, 98 and Millennium branch. These operating system branches are different families, different code. Different people, different developers, different code, and whatever overlap there was didn't extend to the file system drivers. And it was exactly then that Microsoft developed its FAT file system specification, which is now used for formatting EFI System Partition file systems. And that specification, unfortunately, is based precisely on the code that was in Windows 95.
Strange as it may seem. So the FAT driver in Windows NT, Windows 10, 11, Windows XP, doesn't follow that specification. And this creates all sorts of situations where developers of third-party FAT file system drivers, say, in Linux, suddenly realize that the requirements which are, as they say today, mandatory requirements, though supposedly requirements are always mandatory, that those requirements aren't followed by the vendor itself, and in particular, because of that, they have to modify the logic, say, of the file system error check, so as not to detect an error where, by the specification, there seems to be one, but in fact there's no error, if you trust the code.
But that's getting a bit nerdy. Another case is a deception that arose from header files being changed on the quiet. A low-level deception, that is. This is Prefetch. Tell me, who knows what is stored in files with names of this format? That is, OP, hyphen, process name, hyphen something, hyphen something else,.pf. A Prefetch file that sits in the Prefetch directory. Well, many probably don't know, because you don't run into it often, and especially not often when there's actually meaningful data in it. I tried, from all over the internet, including the English-language segment, to collect the hypotheses about what might be stored in this file.
Well, there are basically four. The most, let's say, exciting one is the third, because it stems from that, so to speak, unintentional deception, from stale data in header files, while all the others are, so to speak, made up, just hypotheses that people throw out. And the third one is, you could say, not just a hypothesis but a real theory. The problem is that all along, for a very long time, the right answer has been in a Microsoft blog about ASP.NET, but this artifact is only mentioned there in passing, and such a mention in a blog raises the question: is this a mistake? A passing mention, how relevant is it, or maybe it's all correct there, or not. Because the people who write these articles can still make mistakes.
From the blog, the correct answer looks like this. There are certain calls to the Windows API functions OperationStart, OperationEnd, which result in the generation of this file, highlighted in yellow. This is so-called Operation Based Prefetching. It doesn't follow from this, if you just read it, there's no emphasis on it being exactly a file with this name format that should be created. That's the sense that comes through in the text, but no attention is drawn to it being exactly that way and rock solid.
But let's try to take the hardest route, straight from the code. Let's look at where you generally start in such cases. We have a file name template, there's an extension, there's a prefix; we can search for it in the executables inside C:\Windows\System32.
Just those strings. Then filter out the noise. Since Windows ships with debug symbols, we'll know which functions use the strings we find. And the only occurrence left, so to speak, after filtering out everything unneeded, is in the kernel, the file ntoskrnl.exe. Well, or in other files, depending on how that kernel was compiled, but basically it's there.
Hm, the slides are advancing without me. Anyway, we find the name template there, and it's the name without the extension, and something is done with it. And a call to the function PfSnBeginScenario and outside this code branch, PfSnEndProcessTrace. What is this? Where is this function, whose code is shown in the decompiler, called from?
It's called from the function PfSnSetPrefetcherInformation, if a certain parameter with a value of 5 is passed as an argument. And this function is called, the one shown on the previous slide, PfSnOperationProcess. So we have a certain prefetcher, and it has a certain function. If we pass an argument equal to 5 into a call to this function, this argument is called, its field is called PrefetcherInformationClass, then this file gets created, and something is written into it. Question. What does this 5 mean? It's some constant whose meaning is unclear. And if you google all this, there's the Process Hacker project on GitHub. They merge header files from the Windows SDK into it. That is, everything published on the Microsoft developer site is merged in there, along with some community reverse-engineering results.
These particular constants were most likely taken from old header files from Microsoft; a number of signs point to that, including the manner, so to speak, of naming, but that SDK is most likely no longer available, meaning this information came from some old version of the SDK, the developer header file kit, that isn't published now. And the problem is that the constant equal to 5 is called PrefetcherBootControl. That's exactly where it came from: someone looked at where the function that creates a file with this name format is called, looked at what the constant 5 means, and saw that it's called PrefetcherBootControl. And because of that, in various documentation online, not from Microsoft, there's a mention that Prefetch files, OP, hyphen, something,.pf, are created during boot. That it's some kind of boot Prefetch that loads certain pages before they're read.
But the problem is that this name is out of date; in this case the header file doesn't match the code. So this half-documentation, half-something-else is no longer current. It was current once, but now it's all been reworked.
I've switched back a slide. If we move on to the black-box testing method, that is, we know which functions are responsible for creating these files, OperationStart, OperationEnd. If we throw together a Python wrapper for all of this, we'll see that this has nothing to do with booting, with the operating system boot process. That is, these Prefetch files that get created are created simply when the right functions are called, and the boot process doesn't come into it in any way at all. You can pick out, extract certain patterns from the results of black-box testing by looking at the code in a decompiler: first, this trace is created when this function is called, it ends when the OperationEnd function is called, and within a single process only one trace can be created at any one time, and if there are fewer than 32 I/O requests, the results aren't saved, it's considered there's nothing to prefetch.
But to recall how Prefetch works: it looks at which files a program accesses at startup, or after the OperationStart function is called, and preloads those files into memory, so that the data is already in RAM when it needs to be read, so there's no lag from the program reading files very slowly when we need the data much faster. So Windows, on the kernel side, loads that data in advance. So, 32 requests, so-called page faults: if they occur, the trace is saved, and on the next call to OperationStart the kernel loads that data into memory. Straight from these files.
A number of references were mentioned throughout the talk. I honestly cite all of them, so here, under number 4, is the article I managed to find about why shadow copies contain no user files, and it has a link to a response from Microsoft. So, following the investigation of a ransomware incident, people were puzzled by this very question: why is there no user data when shadow copies survived. And they published all of it, but it's hard to find on the internet. You must know what to look for. You have to know the process is called scoped, scoping. So. Various…
Well, that's about it. Anyway, I hope at least some of it made sense. Thank you for your attention. Happy to take questions. —
[applause]
— At least it was honest. He never once said the word "business". So, colleagues, your questions. I think you've broken my audience again. If there are no questions, Maxim, thank you very much.
That was great. Maxim is staying here, so you'll be able to ask him things.