# From audio and video data to evidence: AI analytics, biometrics, and spoofing and deepfake detection Mikhail Inkin · STC, Speech Technology Center Slides · https://2025.moscow-forensics-day.workers.dev/en/slides/19-crt-ot-audio-i-videodannyh-k-dokazatelstvam An English version of the Russian deck shown in the recording: the text set by the authors is translated and redrawn in place, everything captured from a screen is left as it was. --- ## Page 1 — 08:51:41 \| ГРУППА (C) ИРТ КОМПАНИЙ MOSCOW FORENSICS DAY '25 12.09.25 From audio and video data to evidence AI analytics, biometrics, spoofing and deepfake detection Speaker: Mikhail Inkin Head of project group 16+ ## Page 2 — 08:52:09 About us (О) ЦРТ • ГРУППА КОМПАНИЙ The STC group of companies develops products and solutions based on intelligent speech technologies, generative artificial intelligence, and face and voice biometrics Saint Petersburg Moscow One of the largest players in the AI market in Russia 30+ 5000+ projects worldwide, including nationwide projects years in the AI market Winner of international scientific competitions in speech recognition and biometrics CHiME NIST ASVSPOOF CHALLENGE 750+ employees on staff 70+ R&D teams 20 PhDs and doctors of science Talent pool ViTMO Корпоративная лаборатория Технологии человеко-машинного взаимодействия www.speechpro.ru 2 ## Page 3 — 08:52:35 LLM solutions from Sber (С) ЦРТ КОМПАН- КОМПАНИЙ Business process transformation 8794 Knowledge base search Margin income calculation Predictive equipment diagnostics Procurement assistant GIGAIDE 8582 VERSE www.speechpro.ru ## Page 4 — 08:52:53 Problems Ever-growing volumes of unstructured data (С) ЦРТ ГРУППА КОМПАНИЙ Low efficiency of data search and processing High cost of scaling data processing systems Low speed of analysis and decision-making Loss of important and valuable information www.speechpro.ru ## Page 5 — 08:53:31 Types of biometric systems (C) ЦРТ ГРУППА КОМПАНИЙ System types Automatic Automated Expert (manual) Language-independent Language-dependent Text-independent Text-dependent GMM-UBM Xvector ResNet Wav2Vec (HuBERT) Microphone type: near-field mid-field far-field www.speechpro.ru 5 ## Page 6 — 08:54:17 Human speech synthesis (С) ЦРТ КРУПАН КОМПАНИЙ Every person's voice is unique! More than 70 body parts are involved in producing the voice Not what we say matters, but who speaks More than 30 years of use in forensic examination Hard Palate Soft Palate (Velum) Pharyngeal Cavity Larynx Esophagus Trachea Nasal Cavity - Nostril • Lip Tongue Teeth Oral Cavity Jaw • Lung Diaphragm www.speechpro.ru ## Page 7 — 08:54:35 Speaker model (С) ЦРТ КОМПАН- кОМПАнИЙ Feature vector, embedding V1, V1,2 ... Vi,m V2,1 V2,2... V2n V3,1 V3,2 ... V3,n Vn.л Vn,2 ••. Vn.л www.speechpro ru ## Page 8 — 08:54:56 Identifying characteristics of the voice (С) ЦРТ ІГРУППА КОМПАНИЙ Fundamental frequency (F0) (pitch, fundamental tone) — the rate of vocal fold vibration, is determined by the vocal tract characteristics and lies in the ranges (60-180 Hz M, 160-350 Hz F, and 200-650 Hz for children). Formants — frequency maxima arising from the resonance of the human vocal tract. Usually three formants (F1, F2, F3) are enough both to distinguish sounds and to recognise the speaker. Every formant is a function of all parts of the vocal tract, though the degree of their influence is not the same in each case. 8 ## Page 9 — 08:55:52 Challenges of the task (О) ЦРТ \| ГРУППА КОМПАНИЙ Causes of data variability Reading, spontaneous speech, monologue, dialogue, group conversation Inter-speaker variability Intra-speaker variability Noise, SNR Microphone Vocabulary Accent, dialect Speaker-related Technological Communicative Stress Emotions Lombard effect Language/Culture [1] - Speaker recognition by machines and humans [Hansen J.H.L., Hasan T., 2015] 9 ## Page 10 — 08:56:30 Approaches to speaker comparison (С) ЦРТ КОМПАНИЙ Auditory Acoustic-phonetic Spectrographic: comparison of formants, F0, F0 harmonics Automatic www.speechpro.ru ## Page 11 — 08:57:00 Solving a practical case ## Page 12 — 08:57:32 Product line and the stages of the forensic process (С) ЦРТ КРУПА КОМПАНИЙ Data collection Data analysis Evidence Decision Multipasport Nezabudka Gnom • AVIS IKAR Lab 3 • Nestor.Al www.speechpro.ru 13 ## Page 13 — 08:58:21 AVIS (С) ЦРТ ГРУППА КОМПАНИЙ Products for public safety Biometric search Diarization Full-text search Metadata search Gender, language Semantic search Auto-translation Cluster analysis Speech to text Statistical analysis Spoofing/deepfake detector Link analysis GOG Scattered data Practical result Video identification systems ON Public safety institutions Biometric data Person profile enrichment 14 ## Page 14 — 08:58:51 What happens to the data in AVIS (С) ЦРТ КОМПАН кОМПАНИЙ Рабочий стол www.speechpro.ru ## Page 15 — 08:58:59 What happens to the data in AVIS (С) ЦРТ кОМПАНИЙ Аналиа www.speechpro.ru ## Page 16 — 08:59:06 What happens to the data in AVIS (С) ЦРТ ГРУППА кОмПАНИЙ Анализ www.speechpro.ru ## Page 17 — 08:59:13 What happens to the data in AVIS • ГРУППА (С) ЦРТ КОМПАНИЙ Аналиа www.speechpro.ru 15 ## Page 18 — 08:59:27 What happens to the data in AVIS () ЦРТ КОМПАНИЙ ## Page 19 — 08:59:45 What happens to the data in AVIS (С) ЦРТ • ГРУППА КОМПАНИЙ www.speechpro.ru ## Page 20 — 08:51:16 Biometric link graph (С) ЦРТ КРУПА КОМПАНИЙ www.speechpro.ru ## Page 21 — 09:00:24 LLM in solving forensic tasks (С) ЦРТ КОМПЛА КОМПАНИЙ Text generation from instructions Question answering Text classification Personalisation of answers Creating coherent and grammatically correct text on a given topic Providing answers to questions based on information known to the LLM and on context Determining categories or labels for a given text (for example, sentiment analysis) Adapting answers and behaviour to the specific user Information extraction Conversation support Code generation and debugging Text summarisation Picking out key facts and data from text Holding dialogues with users, understanding and contextualising previous turns Writing code and fixing errors in it in various programming languages from a task description Condensed rendering of long texts, picking out the main ideas Text translation Translating text from one language into another Data interpretation Explaining data analysis results, data visualisation www.speechpro.ru 19 ## Page 22 — 09:00:43 Bulk information processing КОМПАН AVIS Поиск в архиве panta 0 0 00000 30а 01 33 34 www.speechpro.ru ## Page 23 — 09:00:55 Bulk information processing () ЦРТ \| ГРУППА КОМПАНИЙ AVIS Поиск в архиве MeTE www.speechpro.ru 20 ## Page 24 — 09:01:04 Bulk information processing (0) ЦРТ Т ГРУППА КОМПАНИЙ AVIS Анализ • 23000 00 10.54 DU1AT1 3018 10 Dances sou www.speechpro.ru 20 ## Page 25 — 09:01:11 Bulk information processing МПАН AVIS Анализ 2 santos Q. Беседы зази • XS090 301411 Dance 30 crocume 301211 www.speechpro.ru 20 ## Page 26 — 09:01:25 Bulk information processing (0) ЦРТ • ГРУППА КОМПАНИ AVIS Анализ Q Беселы рази ame Iligence Repor www speechpro.ru 20 ## Page 27 — 09:01:32 Bulk information processing (0) ЦРТ \| ГРУППА КОМПАНИЙ AVIS Анализ 9 беседы рази 0 230 20 ## Page 28 — 09:02:08 Auto-annotation of individual recordings (С)) ЦРТ КРУПАН КОМПАНИЙ Детали Результат работы Al-агента Call summary_ Eng О дати Отчет по Анализу Звонка 1. Название Звонка: Обсуж 2. Дата Звомка: Не 3. Ключевы зднике" (birthday party) 06 •(cake coming through this ероприятикх (petting the big lesques"). Be' ("propert) ительность мероприятия ениет Тейст Эрок / Время Дополнительная Информан Отсутствует точная дата звони альшого дня" ('big day) шего анализа www.speechpro.ru 22 ## Page 29 — 09:02:38 ГРУППА (C) ИРТ КОМПАНИЙ Exporting recordings to SIS Анализ 1 Рост уведомления Data KZ + KW Аудиозаписи Статистикл дунио записи + Дата события 2023-01-24 2135 Выгрузить аудио mato -tos 104101 Mena 2023-01-24 000251 ## Page 30 — 09:03:02 Preliminary analysis (О) ЦРТ ГРУППА КОМПАНИЙ www.speechpro.ru 24 ## Page 31 — 09:03:20 Feature extraction and identification (С) ЦРТ | КОМПАНИЯ Automatic comparison algorithms Speech and noise detection Editing and processing of recordings Identification wizard Expert comparison methods Signal characteristics calculation Tho bas ho bo ba Audio visualisation and analysis EdiTracker module HO DH Ti bà bà bà ba bie Authenticity diagnostics Expert report 41 4 “Super Loupe” module 25 ## Page 32 — 09:04:10 (С) ЦРТ КОМПАН КОМПАНИЙ Example 2 ## Page 33 — 09:06:03 Spoofing detector (С) ЦРТ ГРУППА КОМПАНИЙ The detection module* estimates the probability of spoofing (synthesis/conversion, replay, overall score) in the recording. - A.Ri wa - ElevenLatu, 2125-08-23710,34,44, Egor 12 Постовеная состиагающия An overall score and separate scores by attack type are calculated. • Mapod 13.09 секуто Three spoofogram graphs are produced, showing the sliding score values against the signal time. If the synthesis probability score is 50% or above, the extra function for identifying the synthesis vendor can be enabled. * Supplied under a separate licence 27 ## Page 34 — 09:07:09 Spoofing check or authenticity check (С) ЦРТ І ГРУППА КОМПАНИЙ Проверка аутентичности A spoofing check — identifying and detecting forged media files with algorithms that analyse inconsistencies and with deepfake detection technologies. Проверка спуфинт» Врализ по. An authenticity check — confirming the originality and integrity of digital content: metadata analysis; checking digital signatures and watermarks; comparison with trusted sources; deepfake detection. Анализ Deepfake detection aims to find forged media files, whereas an authenticity check confirms the originality and integrity of digital content. www.speechpro.ru ## Page 35 — 09:07:51 History of spoofing (С) ИРТ ГРУППА • КОМПАНИЙ "SH" whistle. Nostrils Speech sounds come out here..) Resonator of leather' 'SH" lever Reed cut-off "S"* lever Bellows, Auxiliary bellows "S" whistle Leather Nostril - Reed Compressed mina air chamber Mark J. Jones, Rachael-Anne Knight (Eds.) The Bloomsbury Companion to Phonetics. 2013 www.speechpro ru 29 ## Page 36 — 09:08:54 Some examples (О) ЦРТ КРУПА- КОМПАНИЙ 1920 1979 Fidelity Voice Chess Challenger (1979) the first talking chess computer https://www.ismenio.com/ chess fidelity vcc.html 2000 A simple speech generator https://ieeexplore.ieee.org /document/861820 2023 2024 Fake Joe Biden robocall urges New Hampshire voters not to vote in Tuesday's Democratic 00 Simple AI voice cloning Advanced voice cloning systems with emotions Google search: voice clone https://www.cnn.com/2024/01/ 22/politics/fake-joe-biden- robocall/index.html 31 ## Page 37 — 09:09:57 SIS capabilities (C)) ЦРТ ГРУППА КОМПАНИЙ Expert method for detecting spoofing Spoofing assessment The automatic detector produces an initial score for the recording The expert records spoofing features in an interactive artefact table* The result — a report saved into the project * Supplied under a separate licence www.speechpro.r ## Page 38 — 09:10:30 SIS capabilities ГРУППА ()) ЦРТ І КОМПАНИЙ EdiTracker module EdiTracker makes it possible to find signs of a breach of authenticity using the following methods: Examining the technical characteristics of the recording device Indicators of Dighal Processing Spr 6. 5.512. 8. 11.025. 16 KH2 the spectrum contains a fal off typicaly caured by and alas flering at lection rehabilly: medum Detecting traces of digital signal pre-processing Detecting stationary harmonics and scanning their phase Background noise scanning Auditory analysis www.speechpro.n ## Page 39 — 09:12:13 І ГРУППА (C) ИРТ КОМПАНИЙ LOVE FRIENDSHIP Example 3 ACCeSS ## Page 40 — 09:12:45 Deepfake analysis () ЦРТ КРУПАНИЙ 1 ГРУППА The video frames of the media file are analysed A high probability of a deepfake has been determined for the synthetic video Frame-by-frame analysis shows a score for every frame LOVE FRIENDSIIP After the analysis an averaged high probability that a deepfake is present is displayed www.speechpro.ru ## Page 41 — 09:12:52 Deepfake analysis (С) ЦРТ КРУПАНИ 1 ГРУПП The video frames of the media file are analysed A high probability of a deepfake has been determined for the synthetic video Frame-by-frame analysis shows a score for every frame LovE FRIENDSHIP Iml 380 конн After the analysis an averaged high probability that a deepfake is present is displayed www.speechpro.ru 38 ## Page 42 — 09:13:34 Deepfake analysis ()) ЦРТ \| ГРУППА І КОМПАНИЙ The video frames of the media file are analysed A high probability of a deepfake has been determined for the synthetic video Frame-by-frame analysis shows a score for every frame LOVE FRIENDSHE 98% -28.02 граку After the analysis an averaged high probability that a deepfake is present is displayed www speechpro.ru 38 ## Page 43 — 09:14:26 Deepfake analysis (С) ЦРТ \| ГРУППА КОМПАНИЙ The video frames of the media file are analysed A low probability of a deepfake has been determined for the original video Frame-by-frame analysis shows a score for every frame ASCeSS After the analysis an averaged low probability that a deepfake is present is displayed www.speechpro.ru 39 ## Page 44 — 09:14:57 SIS capabilities (О) ЦРТ І ГРУППА І КОМПАНИЙ Expert method for detecting deepfakes The automatic score can be supplemented by expert analysis of deepfake features* in the video frames The expert uses an interactive artefact table, marking the typical distortions not only on the face but across the whole frame The result is a report that is saved in the project * Supplied under a separate licence 40 ## Page 45 — 09:15:39 Nestor.Al (О) ЦРТ ГРУППА \| КОМПАНИЙ destor Al Admin e In court proceedings over an audio forgery the Nestor.AI system was used: it automatically turned the speech of the session into text, and the AI assistant prepared a short summary of the content and saved it into a strictly formatted document together with the transcript 0 beting Name Ites quine arman. 2106.2025,3415 21.05.2025.34.15 m 588R.audo.3025-02-12,00 sudio.2025-02-18.00-27-44 mar 12-37 wNd0.2023-02-19.00-22-37m0 22-33 SHER. Ando.1021-02-12.00 aс0 2020-02-300-22-28 ман 5869.sudio.2025-02-12,09 22-24 5809.0080.2025-02-18.09- adi, 2029-02-18.010-21-13 ma Rec 1 4110) Строк на стран www.speechpro * Supplied under a separate licence 41 ## Page 46 — 09:16:10 Nestor.Al (С) ЦРТ КОМПЛА \| КОМПАНИ! lentoc Al trasnes Admin 6 In court proceedings over an audio forgery the Nestor.AI system was used: it automatically turned the speech of the session into text, and the AI assistant prepared a short summary of the content and saved it into a strictly formatted document together with the transcript Ric 1 0000L.1.1 ma camel 2021-09-22 Pyccon 2025-08-20 2025-08-20 2025-00-54 2025-08-14 Русскии 2023-08-14 Русскин Русский 2025-08-13 2025-08-13 Prcosi 2025-08-07 2001-04-06 Pyccon 2001-04-08 2025-08-07 Pyccon 2023-08-07 2001-04-00 Arcan 20 - 1-7 Cooa tee * Supplied under a separate licence 41 ## Page 47 — 09:16:15 Nestor.Al \| ГРУППА (С) ЦРТ КОМПАНИЙ destoc Al Framan Tm Admin e In court proceedings over an audio forgery the Nestor.AI system was used: it automatically turned the speech of the session into text, and the AI assistant prepared a short summary of the content and saved it into a strictly formatted document together with the transcript 01 L0 2021-09-27 Русский 2025-08-21 2085-00 20 2025-08-20 Русски 2025-09-14 2025-08-54 EFFEEEFD Русски 2073-08-14 2025-08-14 Русски 61 m 2025-08-14 2025-08-14 Pycoond 2025-08-13 2025-09-13 2025-08-07 2001-04-06 Русски 2025-08-07 Pycoon on-3a 2001-04-00 2025-08-07 Орокнетіренн 20 - 1-20 * Supplied under a separate licence 41 ## Page 48 — 09:16:20 Nestor.Al (С) ЦРТ ГРУППА КОМПАНИЙ lentor Al Admis e In court proceedings over an audio forgery the Nestor.AI system was used: it automatically turned the speech of the session into text, and the AI assistant prepared a short summary of the content and saved it into a strictly formatted document together with the transcript TUPI OCEA 02.06.207100207 POCOETA MPO 2025-08-21 2025-08-20 2025-09-14 30 2025-08-14 2025-08-14 038 2025-08-13 200 2025-08-07 20 2025-08-07 www.speechpro * Supplied under a separate licence 41 ## Page 49 — 09:16:30 Nestor.Al (О) ЦРТ І ГРУППА \| КОМПАНИЙ destoc Al Franar Admic 6 In court proceedings over an audio forgery the Nestor.AI system was used: it automatically turned the speech of the session into text, and the AI assistant prepared a short summary of the content and saved it into a strictly formatted document together with the transcript OUCO 1 02.0.2071.90.202 Сірок не страния 2025-08-21 2025-08-20 2025-08-14 30 2025-08-14 2025-08-14 2025-09-13 www.speechpro * Supplied under a separate licence 41 ## Page 50 — 09:16:35 Nestor.Al (0) ЦРТ | І ГРУППА destoc Al Frasa In court proceedings over an audio forgery the Nestor.AI system was used: it automatically turned the speech of the session into text, and the AI assistant prepared a short summary of the content and saved it into a strictly formatted document together with the transcript * Supplied under a separate licence 41 ## Page 51 — 09:16:40 Nestor.Al (О) ЦРТ КРУПА- КОМПАНИЙ destoc Al Fase Admin e In court proceedings over an audio forgery the Nestor.AI system was used: it automatically turned the speech of the session into text, and the AI assistant prepared a short summary of the content and saved it into a strictly formatted document together with the transcript Экотирту Speaker зя организовать поездку в офи Отвентвенне Срок оручени Андерс- укачено і зеленые лина! рганизовить повзаку в орис мовый позлучный пари інтерст садут одушный шарик в форми накрос Спортие поженть Meeting Minutes www.speechpro * Supplied under a separate licence 41 ## Page 52 — 09:16:54 Conclusion ГРУППА (С) ЦРТ КОМПАНІ The products of the STC group of companies work at different stages of the forensic process and make it possible to use voice biometrics, face biometrics and intelligent speech technologies both to search for the material of interest and to obtain forensic reports www.speechpro.rul