# Automated government systems under the forensic computer expert's microscope: problems and solutions

Oleg Bezik · Digital Research Laboratory

MOSCOW FORENSICS DAY ’25 · Day 2 — Friday, 12 September 2025: information security day · Scheduled 16:45–17:15 · In the recording 08:13:25–08:50:23

Talk transcript · https://2025.moscow-forensics-day.workers.dev/en/transcript/19-bezik
Summary: https://2025.moscow-forensics-day.workers.dev/en/summary/19-bezik · Slides: https://2025.moscow-forensics-day.workers.dev/en/slides/18-bezik-avtomatizirovannye-sistemy · Watch from 08:13:25: https://youtu.be/4V7Wez3L_58?t=29605

---

## Moderator's introduction

So, our next speaker is a real, genuine forensic expert. And you'd be amazed at what ends up under a forensic expert's microscope. Please welcome Oleg Bezik, who's going to look at a completely new topic. A round of applause. Oleg is a regular guest of ours, a regular speaker. —

— A resident, you could say. —

— Okay, this is forward... no, that's back. There we go, great.

## Talk and Q&A

Greetings to everyone, ladies and gentlemen. I take it you're the survivors, right, who held out to the end of day two. Thank you very much. I hope I'll tell you something interesting and useful today.

I've been introduced, yes, I really am a forensic expert, I do forensic computer examinations, I do it at my own company, I'm the founder and CEO of a company called Digital Research Laboratory. Since 2016 we've been performing forensic computer examinations, and as of this year, 2025, we've also become an IT company: we're developing software for comparing source code in the course of forensic examinations.

A couple of words about me. I've been doing forensic examinations since 2014, I worked at Big Four firms, I hold several international certifications. Now I'm growing my own company. The company has existed since 2016. Over the last three or four years we've built up quite a lot of experience, including in performing forensic computer examinations of automated systems. Today I wanted to tell you about that, share a bit of our experience.

Let's start with what automated systems are in the first place. I hope everyone here uses Gosuslugi. That's one example of an automated system, one that gets developed and is used by a large number of people. There's also a category called state automated systems. What is that, essentially? It's a hardware and software system that automates some activity. State automated systems automate the activity of some specific government body, or some function. For example, GAS "Pravosudie" automates a function of the court system.

GAS "Legal Statistics" automates the work of prosecutors, and so on.

Why am I telling you about this at all? Because of how a state automated system gets developed. As a rule, it's a state contract, if it's done for the government, or else a contract between a customer and a contractor. The contractor is usually some legal entity that has a staff of IT people, developers, security people, who can write a program that meets the customer's technical requirements.

And very often relationships like that end badly. They develop the program, and it doesn't meet the spec requirements.

The value of such contracts is very high. Say, developing a system costs 150 million, 500 million, maybe even several billion rubles. And when the customer gets a system that doesn't work, while paying several billion rubles for it, well, that's not great, honestly, not much fun. And as a rule, situations like that always, well, most often end up in litigation, mostly it all happens in the commercial (arbitrazh) courts, legal entities suing each other, and sometimes it even gets to criminal cases. And in cases like that, 99% of the time, even 99.99%, a forensic computer examination gets appointed. Because judges understand nothing about how complex automated systems get developed.

And in order to figure out whether the program really fails to meet the spec requirements, they order a forensic examination. A forensic examination, I hope everyone here knows what that is, but just in case, let me remind you. Essentially a forensic examination is a way of bringing specialized knowledge into court proceedings, in our case IT knowledge, since it's a computer forensic examination.

On top of that, in this category of cases, where you need to check an automated system's compliance with the spec, they also bring in appraisers. What for? So that the appraiser can calculate how much the work is worth that was done correctly, or how much it would cost to finish the system, to bring it up to the condition stated in the spec. And then it turns into a complex, multi-discipline examination, that is, when the examination is done by experts in different specialties, with different areas of specialized knowledge. But we won't be talking about valuation examinations, that's a bit more probably for some other conference.

For now let's talk about computers. Here are a few examples of such cases, fairly serious cases. We worked on them, it's our experience. We did the examinations in these cases.

What is the purpose of such examinations, most of the time? First, to make sure that the program really does or does not meet the technical specification. Second, if it doesn't, to understand what specifically doesn't work and why it doesn't work. Because the reasons can always be different. It's not necessarily the developer's fault, that they're so incompetent. Especially when it comes to state systems, big complex systems, where the developer-customer relationship has to be practically like family, because the customer dictates the conditions, and the contractor has to react to those conditions very quickly. And sometimes there are simply communication problems, and because of that it all spills over into lawsuits. That happens too.

What other purpose? Again, the purpose is defined by the court's point of view. We think like a judge: what does the judge need in order to, say, rule on a claim.

So, the presence of critical defects; we'll talk more about what those are. And then cost, which I already mentioned: assessing the cost, how much was done correctly, how much needs to be reworked.

Roughly, these are the typical questions that get asked in this type of examination. They're actually more or less always the same, but it depends on the context. Of course, sometimes there are other questions, but mostly they look like this. That is, do the results comply with the requirements of the state contract, the agreement, the technical specification and the detailed specification. So, the contract has an annex, and as a rule that's the technical specification. And that specification describes in detail what the program is supposed to do. That is, what the automated system should do, what functions it performs, what results it should show, how it should show those results, and so on.

It's all described in great detail. But sometimes it's actually not that detailed. That's also one of the problems. We as experts get asked: does the program comply with the technical specification? We open the spec, and it's ten pages. Well, as they say, without a good spec, the result is anyone's guess. Well, that's roughly what this is about. But most often, when it comes to developing large automated systems, there, of course, the spec is very complex and well thought out, there are lots of items, the customer is very meticulous about drafting the spec and getting it approved. And as a rule, roughly speaking, here the customer is protecting itself. That is: we'd better ask for more, in more detail, than ask for too little.

But sometimes it's objectively different: the customer approves the spec and then makes demands that aren't in the specification. That happens too. And the developers often fulfill those demands. Well, to please the customer, it's big money, nobody wants to get into a fight. They fulfill them, but it still doesn't help, everyone still goes to court and gets examinations done. Right. So, next, the second category of questions is about cost. The cost of the work actually performed. For cost assessment there are certain methodologies, there's the Moscow DIT methodology for estimating the cost of software, there's the so-called COCOMO method, they're roughly similar, and they really do let you get more or less the same results, but that's more of a valuation matter.

I'm not an appraiser, so I won't go into it, I won't make things up for you. Right, so next is the category of questions about critical defects. This is also a very important point, because not all defects are equal. We often see this: there's some set of defects, and the customer of the system says, look, the contractor didn't finish this, and this, and this, and this. We start looking, and it turns out that this, this and this is about a week's worth of work, and basically the contractor is ready to do it, no problem at all. And sometimes it happens that the contractor says, no, everything works on our side, it's all fine, all good.

Yes, you sort of look through it, look through it, all the functions are implemented, but one function, the one and only function this software was created for, simply doesn't work. That's it. So everything looks nice, the buttons are blinking, like Nikita was telling us, everything's green and red, the filters can be configured, customized and so on. But the one report that matters most, the one that's used, say, for some important purposes, it doesn't get generated. Why? Nobody knows. And so what you get is a screw-up, a defect, that prevents the system from being used for its intended purpose. And it turns it, essentially, into a useless pretty toy that costs a lot of money but in the end leads to nothing good.

And it's made worse by the fact that people still have to work. System doesn't work, but people still have to work. Generate reports, deliver some result of their activity. And people end up having to do the very thing the program was created for, which is the whole reason it's built: to automate that activity. People are, in fact, doing it by hand. And people are also dying on that, dying from overwork. So critical defects are an important matter that is always checked in examinations like these. Well, almost always. And then there are the questions of whether a defect is remediable or not. That's also usually what the court is interested in, because it's one thing whether a defect can be fixed, and another thing when it seems fixable, but in order to fix it you need another contract just like this one, for another billion rubles.

And then it turns out the contract simply makes no sense anymore. So that's as far as remediable and irremediable go. That is, an irremediable defect, as opposed to a critical one, an irremediable defect is one that is simply impossible to fix, or not economically viable to fix.

Bless you. Next. And the last question: for what amount of the contract price was work of inadequate quality performed? Well, that's also more of a valuation question, really. Right.

Now I wanted to share the pain points we run into when doing examinations like these. Pain point number one is that methodologies for forensic examination of automated systems, you know, any kind of agreed, publicly available ones, simply don't exist, unfortunately. Well, we've already done a lot of these examinations, and for ourselves we've developed a methodology of certain steps that we take: roughly, we build a database of the objects submitted to us, check that all the deliverables are present, check that those deliverables meet the ToR requirements, and so on. So it's, let's say, just an algorithm of actions; for now, as a methodology, we haven't packaged it, but maybe someday we will.

It's further complicated by the fact that every system has something custom-made. It's like, you know, you can build lots of single-storey houses, but if someone wants to build themselves some unique palace, well, that's the story with automated systems. That is, there are no off-the-shelf automated systems sold in bulk, and for their requirements the customer always hires a developer to create something unique. By the way, there's another problem here: estimating the cost of all this work, because, say, one developer can do it for 100 thousand, another for a million, and a third for a billion.

And how to assess that is sometimes difficult. Problem number two: the huge volume of objects to examine. What are the deliverables in examinations like these? Documentation, technical documentation, meaning the ToR, the specific ToRs, the user manuals, all sorts of explanatory notes, and, in short, a whole million documents. By the way, the photo shows the object of examination from one of our cases. See, one box didn't even fit in the shot. That's just text documents on paper. A computer forensic examination. It's supposedly a computer forensic examination, but in the end it feels like it isn't a computer one at all.

That's one category of objects. What else is there? The program's source code. An automated system is, after all, programs. Programs are written by developers in some programming language. Source code is usually delivered on media, either on flash drives or on optical discs. Just blank discs that they burn the source code onto. Then they come to us for examination and we study them. And that source code is unbelievably huge. That's millions of lines of code.

What else is interesting here? The system itself, the automated system, as a rule, especially in government agencies, is designed for a huge number of people, for 5 thousand people, for 10 thousand people. I think the lighting just changed, or did I imagine it? Oh well. And accordingly, the functionality of these systems is complex. Imagine, we have an examination now, we did it, well, we've already finished, now we're writing the report. There were a thousand requirements, 1,041 requirements that had to be checked. So it's just an enormous amount of work, and specifically, when checking the functionality, how does it go: we sit down at a computer and, basically, open this automated system and start looking at what works and what doesn't.

We end up as testers of sorts, the role of testers. Not sure if there are any testers in the room. Anyway, we're similar in that respect. And what else? What other problems?

An examination can take a very long time. We have examinations that took us two years, a year, a year and a half. It's an unreal amount of work, and it's very difficult work. And, to be honest, it's expensive. Really expensive. Because, as a rule, examinations like these aren't done by one person. With us it's usually done by a panel of 3 or 4 people. We go, like to a job, to the office of the customer where the disputed information system is deployed, and sit there checking it. And the parties, the plaintiff and the defendant, come along with us. Our record, I think, was 9 people in the room. Besides the two forensic experts, seven more people, representatives of the defendants, sat with us, argued, quarrelled, told each other to go to hell, and we sat there watching all of it.

Quite a show, of course, but on the whole entertaining. You probably know that the parties may be present during a forensic examination. If you don't, here's a life hack for you: if anyone ever ends up in an examination, you can be present while it's being conducted.

Right. So those are the kinds of problems we run into when examining automated systems. And I wanted to give a couple of tips to those who do go into an examination, maybe someday it'll be useful to someone; I don't know how well this audience fits these tips, but if anything, pass them on to colleagues, lawyers or IT companies that develop automated systems, maybe it'll be useful to them someday. But on the whole, the main thing, of course, before entering an examination like this, is to understand in general what objects you currently have on hand. So if you're, say, the customer's representative, the contractor should have handed the deliverables over to you. Documentation is usually provided on paper and on media, on discs.

The program's source code and the deployed program itself. So those are the three main objects. You need to see if you have them or not, what state they're in, if... Oops, my... Ah, it works. Something flickered. Probably tired too.

And naturally, in such cases the objects of examination also include agreements. Agreements, contracts, there are a lot of curious clauses in there too, which we, as forensic experts, are also obliged to check.

The second point: it'd be a good idea to prepare the questions for the expert, yes, I gave a rough, approximate list yes, that you can rely on, but the devil is always in the details, and it all depends on the context, on the system, on the functionality that isn't working; that is, you can ask questions about individual modules, you can ask about the whole system, in short, there are many different options, but the basic concept for the questions, I've already given it. And third, of course, you need to choose the experts. The experts should still be... These systems are complex, automated, and as a rule the examination is long and difficult, so you need people who are professionals. I firmly believe that only professional forensic experts should do examinations.

I hope nobody takes offence if there are experts here who aren't forensic experts by education. And experience, of course, experience, because experience in such forensic research is very important, because once you've developed, well, a certain trained eye, yes, you can solve problems more precisely and more quickly. Right. Oops, where's the clicker? Here it is. That's it, colleagues, I'm done. Just one more thing I wanted to mention. We've made this checklist, a checklist-cum-memo on preparing for an expert report, for an examination of automated systems. You can go there and grab the PDF for yourselves.

Maybe it'll be useful to someone too. Right. That's all from me. Thanks a lot. Thank you, Oleg. Oh, so many hands.

You know, questions like these, about examining all sorts of systems, have been coming up for over 20 years now. When I worked at the Forensic Science Centre (EKC MVD), I tried to avoid this. What's the point? We had to examine an SAP system and banking systems, but never in our lives did we take on a question about testing. Let's look at it this way. There's economic forensic examination, accounting forensic examination. They fought to the death not to be made to do audits. That is, a full check of a warehouse, reconciling something else, that's not the expert's job. The expert's job is to check and confirm: doesn't add up here, doesn't add up there, why it doesn't, where it's recorded. And here you're describing it as if the system was built and nobody at all, there was no test programme and procedure, no acceptance of the work, and you start everything over from scratch. What is that? In principle, you should be checking specific questions that were identified: here's the discrepancy, right here.

A year ago we held, well, at Interpolitex we have our own event, where Mr Muzalevsky spoke, among others. We listened to six talks on the examination of automated systems. The first question that always comes up, and today Denis Aleksandrovich has left, he had his own problems to deal with, he talked about how the terms of reference should be prepared, what to pay attention to. And as a rule, the problem questions arise because the customer doesn't know how to formulate the tasks. Agreed. And then, you know, what simply astounds me. I can examine. Can you examine Microsoft Windows for ToR compliance?

That's too complex a system. So up to what system complexity do you go? Because, say, pilots are trained to fly on aircraft simulators. Can you assess the quality of that system without being a pilot? And you're unlikely to find the person who can help you with that from a professional standpoint. We're not specialists in, say, banking, so assessing how well that program has been written, is something you basically can't, or whether it'll work that way. Actually, as I say, one very correct, very good thing was said: when an examination like this is conducted, specialists from one side and the other should be present. They agree among themselves whether the actions we've taken are enough to check whether this program works or not. With SAP that's exactly how it was, because, well, a specialist who knows all of SAP just doesn't exist, and the consultants were configuring it, and actually this was the critical task: does the handheld terminal work?

I had a task like this too, where: we've been working for three years, you haven't finished it, and it doesn't work. And he goes: hold on, let's open it up, tick this box in this table, and over here, come on, check it: it works. And that's how it was. And so I don't understand when you answer qualitative questions. Quantitative ones, yes, but in a computer forensic examination, being responsible for the quality of all those actions, well, that's quite difficult. And then, look, they ordered a bicycle, and got a scooter. They built it, spent the money, but it's a scooter. —

— And so a valuation examination, damn it, what's it going to value? Sure, you can value the cost of the work, but nobody needs that. And in general, the market value of a public-services system, apart from the state itself, nobody needs it. Only the state can say, well, I'll pay that money, or I won't. Market value there is a really tricky thing. Unless you go, sort of, with the cost approach. There are many contentious issues here. Volokitin and I basically agreed that we'll work together on this and maybe come up with something. But for God's sake, don't take on all the testing. That's a road to nowhere.

Thank you very much for that comment. I wanted to respond. Can I, now? One second. Look, first of all, I wanted to say that, first, we conduct examinations on the questions the court puts to us. We don't define them. So the questions I listed are the questions the court puts to us. And the court needs to know whether the software meets the ToR requirements. What does the ToR say? The ToR says there must be such-and-such results. The results must comply with the detailed ToR. What does the detailed ToR say? We open it, and it says the program must do this. This isn't about SAP, this isn't about Word, this is about programs that are developed from scratch, custom-built for the client. That is, information systems of some kind.

For example, we did an examination of the State Automated System of Legal Statistics. It's a huge system that a large team developed over several years, from scratch. It's not off the shelf, and there are specific requirements for that system. And we can verify those requirements, as forensic experts, because they are set down in the detailed ToR.

And there's a test programme and procedure that we can use as a basis. —

— And how do you do that in two weeks? What's the question asked? It's whether it meets the contract requirements.

— The question needs changing, work with the parties.

— But we're not judges, we can't come to court and say, Your Honour, change the question.

— You can, why not? You can, and always could.

— I don't know, I've never come across that. Never.

— That's strange, really strange. All of that can be done.

— I just don't understand on what grounds we'd change the question. We come to court and say, court, you're thinking wrong?

— Yes, you file a motion saying that this isn't quite the right approach. Well, if you want to pay 5 million for an examination, go ahead, but you could pay 200 thousand, or 500 thousand.

And why would I want that? Ah, exactly, that's a different question. I don't get it, why would I want that? Then the approach is clear: don't deny yourself anything for the money paid. Of course, the court wants to know it all, and we'll tell it all. —

— Oleg, I wanted to say — my name is Barannikov, Sergey Nikolaevich, I'm also a forensic expert, and I've actually taken part in acceptance. So examinations like this arise when, under Federal Law 44-FZ, a state or municipal contract is concluded. And then the customer, to shift responsibility off themselves, so they don't go to prison for accepting God knows what, brings in computer forensic experts, among others, so that, step by step, the whole set of information documentation, the program code, with hashes, is checked for whether it complies or not. It's genuinely very painstaking work; I'll come show you later, I had mountains like that too. That was the state information system of Rosreestr. —

— And I really did sit there for several months, working on it, and then we reported that it complies, it complies. So such examinations are carried out, but not as part of the 44-FZ check, before it goes to court. This is what customers need to be told, so they cover themselves, because they really accept God knows what. The work is extremely interesting, it includes load testing and other tests, like that, to verify the data that would be... I've had to take part in that too.

— I understand you, it really is This work is very well paid, very complex and costly. —

— Congratulations on having taken part in work like that. It really is interesting work. Thanks a lot. Colleagues, any more questions? Yes, I see a hand, coming. —

— Andrey, please, don't knock me down again. —

— Oleg, thank you very much for the talk. I was looking at the questions that get put to the examination. Some of them concern cost issues, that is, valuation. So am I right that the examination format is usually multidisciplinary, that is, you bring in economists who deal with the valuation questions, the amounts, cost and so on? Or do you basically do it all yourselves within IT, with your special knowledge? Thank you. Thank you, Andrey. Well, as I said, such examinations are mostly multidisciplinary. That is, if there's a valuation question, you need an appraiser. Not an economist as such, but an appraiser.

Someone with an appraisal qualification who can apply valuation methods. I hope I've answered the question. —

— Another question. You were talking about valuing software development. Well, I don't know, I'll find out — people making software for 20 years, they can roughly estimate how many guys will be needed, well, development, how many designers, how much time, even consultants can. Honestly, I don't know what formulas you'd use to calculate, because developing some super-high-quality software solution that covers the whole ToR and about which no examination would ever say there are problems, right, if the customer for some reason decides to put the contractor in prison, then, well, I don't know how to calculate that. I mean, there's no formula; good software is — a huge contribution, especially if it's security, testing, deployment, well, that's many years, it's done over years. So, as I am, I just don't quite understand how you say you calculated it, right. Everything else I can see how to cost.

And the second question: for two solid years we... We're proving he stole the money, he must go to prison. And meanwhile, is anything being done so the system gets put into operation and can actually be used? Does the task get solved, or how does it go? That's also an excellent question. You're getting to the root of it. Look, on valuation, disclaimer first: I'm not an appraiser, so I can't give details. But I know we apply the Moscow IT Department (DIT) and COCOMO methodologies.

I suggest reading up, if you're interested. I just can't tell you the details, physically can't, I'm not an appraiser. But what's the idea? The idea is that these methodologies assume you can evaluate a program by some quantitative indicators. Number of lines of code, the number of, what's it called, not functionality, something like milestones or so. So, accordingly, those are the methodologies that are used. Apart from that, what other methodology did we use in our work? But mostly the classic method is like: it cost 100 million, 87 percent was done, so that's 87 million, right. Well, that's what we often come across, but we don't do it that way.

Well, we go by the number of requirements and how many are fulfilled and how many aren't.

— Just a second, but here, unfortunately,

— the weight can't be determined, — that's the whole point, so you're calculating the percentages all wrong.

— Why wrong? — Well, because every module has its own development complexity, and so, well, the contribution of that module.

— But in any case we somehow need to explain to the court how much is fulfilled, how much... — First, you're explaining it wrong. Second, I've repeatedly taken part, as the customer's representative, the functional customer, in commissioning and producing various R&D projects, including software systems and hardware-software systems. And I know perfectly well that the price announced for that work was, from the outset, lower than what companies would have spent that came in and started from scratch. Any company has certain prior developments. With prior developments in the field, you can do something for that price. If you've never been involved in it, you'll definitely, well, not so much lose the tender. You may even win it, but in the end won't do the work, because in a year, a year and a half, two years it's impossible. How do you account for prior developments, what the contractor must already have? They don't just go off for some price and do something.

It's always amazed me when people try to evaluate what they don't understand. Number of lines of code, quality of the program. We had it at one point: here we consider this programmer good, high quality, this one not so good, this one a bad programmer. How do you evaluate that? I consider myself an average programmer, and an expert tells me... I'd have written it better, so he's a low-quality programmer. And then I look: oh, how nicely written. That has always astounded me, really.

I'm sorry, one more question.

Fine, you rate yourself highly, you graduated from Bauman MVTU and so on, but there are two parties in the process, the customer and the contractor. The customer and the contractor came to an agreement at the preliminary stage, when the preliminary tests were done. Basically, 90% of the tasks are settled And they only have a dispute over the remaining 10% —

— But it's not always that way. So which of them will pay you those 5 million For re-confirming, right from the start, the 90% that nobody is disputing Colleagues, I think we're speaking different languages I'm talking about a forensic examination, one appointed by a court ruling. A request comes to us, we say it will cost such-and-such. The court pays us, not a party. That's one. Second, well, where the court gets the money from is not a question for me. We should probably hold the roast on day two. I don't know why every time legal questions come up, it nearly ends in a fight.

As if next time we'll just… It's just such a debatable, interesting topic, you see. Yes, basically, I think that's exactly how this legal business works. I have Valeria Mikhailovna, our lawyer too, and every time a question comes up, I just walk away, because I don't understand any of it. Thanks a lot, Oleg. Let's see Oleg off, a round of applause. Thank you, colleagues.