I am a Senior R&D Researcher at the Technology Innovation Institute (TII) in Abu Dhabi, in the AI and Digital Science Research Center (AIDRC). I did my PhD at INSA Rennes, in the IETR laboratory, on hardware designs for the new coding tools of the VVC / H.266 standard, built for a real-time 4K encoder and decoder chip, and I defended it in May 2022.
Since then I have moved to AI, worked on 3D reconstruction with NeRF and 3D Gaussian splatting, edge AI and model compression for the Falcon LLM models, and now models and systems for robotic VLAs and agentic coding for robots.
News
2026Our patent "Learning based methods for real-time omnidirectional video streaming" (US 19/006,667, with M. K. Sharma and W. Hamidouche) has been accepted.
2026Q-Codec, reinforcement-learning-based adaptive encoding for immersive aerial video, accepted at IEEE WCNC 2026; ALRM, an agentic LLM for robotic manipulation, on arXiv.
2025The Falcon-H1 technical report is out; the results of our 360° video super-resolution challenge are published in Signal Processing: Image Communication.
2024ODVISTA, our omnidirectional video dataset for super-resolution and quality enhancement, accepted at IEEE ICIP 2024.
2024Our survey on real-time immersive aerial video streaming is published in the IEEE Open Journal of the Communications Society.
Main research interests
Edge AI and model quantization.
Vision-language-action models: pre-training, supervised fine-tuning and reinforcement learning.
M. K. Sharma, I. Farhat, W. Hamidouche, “Q-Codec: Reinforcement Learning based Adaptive Encoding for Real-Time Streaming of Immersive Aerial Videos”, IEEE Wireless Communications and Networking Conference (WCNC) 2026. [doi]
D. Scheinert, A. Acker, T. Wittkopp, S. Becker, H. Yous, K. Reddy, I. Farhat, H. Hacid, O. Kao, “Beyond Microservices: Testing Web-Scale RCA Methods on GPU-Driven LLM Workloads”, Companion of the 17th ACM/SPEC International Conference on Performance Engineering (ICPE 2026). [doi]
V. Gaboardi dos Santos, I. Khadraoui, I. Farhat, H. Yous, S. Teffahi, H. Hacid, “ALRM: Agentic LLM for Robotic Manipulation”, arXiv preprint 2601.19510. [arxiv]
2025
A. Telili, W. Hamidouche, I. Farhat, H. Amirpour, C. Timmerer, I. Khadraoui, J. Lu, T. Le, J. Baek, J. Lee, Y. Wei, X. Sun, Y. Gao, J. Huang, Y. Zhong, “360-degree video super resolution and quality enhancement challenge: Methods and results”, Signal Processing: Image Communication 138, 117376. [doi]
J. Flerlage, A. Acker, D. Scheinert, S. Becker, H. Yous, K. Reddy, I. Farhat, H. Hacid, O. Kao, “From Intentions to Actions: Workflow Generation with LLMs for Robots”, 3rd International Conference on Foundation and Large Language Models (FLLM 2025). [doi]
J. Zuo, M. Velikanov, I. Chahed, Y. Belkada, D. E. Rhayem, G. Kunsch, H. Hacid, H. Yous, B. Farhat, I. Khadraoui, et al. (27 authors), “Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance”, Technical report, arXiv preprint 2507.22448. [arxiv]
M. K. Sharma, B. Farhat, W. Hamidouche, “Learning based methods for real-time omnidirectional video streaming”, US patent application 19/006,667.
2024
M. K. Sharma, I. Farhat, C.-F. Liu, N. Sehad, W. Hamidouche, M. Debbah, “Real-Time Immersive Aerial Video Streaming: A Comprehensive Survey, Benchmarking, and Open Challenges”, IEEE Open Journal of the Communications Society 5, pp. 5680–5705. [doi] [arxiv]
A. Telili, I. Farhat, W. Hamidouche, H. Amirpour, “ODVISTA: An Omnidirectional Video Dataset for Super-Resolution and Quality Enhancement Tasks”, IEEE International Conference on Image Processing (ICIP) 2024, pp. 131–136. [doi]
I. Farhat, I. Khadraoui, W. Hamidouche, M. K. Sharma, “Energy Cost of Coding Omnidirectional Videos using ARM and x86 Platforms”, 2nd International ACM Green Multimedia Systems Workshop (GMSys 2024). [doi]
M. K. Sharma, I. Farhat, W. Hamidouche, “Error Concealment Capacity Analysis of Standard Video Encoders for Real-Time Immersive Video Streaming from UAVs”, IEEE Wireless Communications and Networking Conference (WCNC) 2024. [doi]
2023
I. Farhat, P.-L. Cabarat, W. Hamidouche, P. Angot, P. Gonon, D. Ménard, “Live and Low Energy VVC Video Decoding powered by the OpenVVC Decoder on ARM Platform”, IEEE ISCAS 2023. [doi] [hal]
M. Viitanen, J. Sainio, A. Mercat, G. Gautier, J. Vanne, I. Farhat, P.-L. Cabarat, W. Hamidouche, D. Ménard, “Open-Source Toolkit for Live End-to-End 4K VVC Intra Coding”, ACM MMSys 2023. [doi] [hal]
I. Farhat, P.-L. Cabarat, D. Ménard, W. Hamidouche, O. Déforges, “Energy Efficient VVC Decoding on Mobile Platform”, IEEE MMSP 2023. [doi] [hal]
A. Saha, W. Hamidouche, M. Chavarrías, F. Pescador, I. Farhat, “Performance analysis of optimized versatile video coding software decoders on embedded platforms”, Journal of Real-Time Image Processing 20, 120. [doi] [hal]
2022
I. Farhat, W. Hamidouche, A. Grill, D. Ménard, O. Déforges, “Adaptive Loop Filter Hardware Design for 4K ASIC VVC Decoders”, IEEE Transactions on Consumer Electronics 68(2), pp. 107–118. [doi] [hal]
I. Farhat, W. Hamidouche, A. Grill, D. Ménard, O. Déforges, “Efficient HW Design of Adaptive Loop Filter for 4K ASIC VVC Encoder”, Picture Coding Symposium (PCS) 2022. [doi] [hal]
2021
I. Farhat, W. Hamidouche, A. Grill, D. Ménard, O. Déforges, “Lightweight Hardware Transform Design for the Versatile Video Coding 4K ASIC Decoders”, IEEE Transactions on Consumer Electronics 67(4), pp. 329–340. [doi] [hal]
2020
I. Farhat, W. Hamidouche, A. Grill, D. Ménard, O. Déforges, “Lightweight Hardware Implementation of VVC Transform Block for ASIC Decoder”, IEEE ICASSP 2020, pp. 1663–1667. [doi] [arxiv]
My thesis
Efficient hardware designs of the new Versatile Video Coding (VVC) tools for ASIC platforms. INSA Rennes, defended 2022-05-31. [manuscript, open access] [theses.fr] [the explainer]
Abstract. The new MPEG/ITU video coding standard named Versatile Video Coding (VVC) was finalized in July 2020. VVC enables 40% better coding efficiency than its predecessor HEVC standard. This coding gain is brought by several coding tools. The MTS and LFNST are among the key coding tools introduced in VVC for the transform part. In addition, in order to enhance the perceptual quality of the coded frames, VVC adds a new tool to its in-loop filter process called the ALF. In this thesis, hardware implementations for all the above tools targeting ASIC platforms are proposed. They feature all optimisations in order to reduce their cost in terms of chip surface, memory and power consumption. All the proposed solutions were integrated in a real-time professional VVC encoder and decoder. The first contribution is an efficient hardware implementation of the MTS transform: a multi-standard architecture that supports the transform block of recent MPEG standards including AVC, HEVC and VVC, which removes unnecessary complexity found in other architectures by using regular multipliers instead of constant multipliers. Then, for the same context, the LFNST is added to the transform module, leveraging butterfly decomposition, coefficient zeroing and the linear relationship between the transform types. The second contribution is an efficient hardware implementation of the ALF tool. For the decoder, a novel scanning order between Luma and Chroma components significantly reduces the memory. For the encoder, where ALF has a filtering phase and a decision phase, the encoder activates ALF only for Luma samples and for CTUs of size 64, which proved efficient and was integrated in the real-time VVC encoder.
My PhD
PhD · INSA Rennes / IETR · 2019 – 2022
Efficient hardware designs of the new Versatile Video Coding (VVC) tools for ASIC platforms
For three years I worked on getting three of the new VVC tools to run in real time on a chip, for a company’s 4K encoder and decoder. This page explains the work in ten short parts, with more detail under each “Go deeper”.
Ibrahim Farhat · INSA Rennes / IETR · defended in Rennes on 31 May 2022
the blocks I designedthe rest of the chip, simplified
Bit rate at equal quality, relative to H.264/AVC, and encoder computation relative to HEVC. Thesis §2.2.
1 / 10
The starting point
Every few years a new video standard halves the bits needed for the same picture quality. VVC did that in 2020. The cost is computation: the VVC encoder is about ten times heavier than HEVC’s and the decoder about twice. In a product, that computation has to be done in hardware.
Go deeper
Standards: H.261 (1990) → MPEG-2 (1993) → H.264/AVC (2003) → HEVC/H.265 (2013) → VVC/H.266 (July 2020), each developed jointly by ITU-T VCEG and ISO/IEC MPEG.
VVC: about 40 % bit-rate reduction in objective terms (thesis abstract) and about 50 % in subjective tests (JVET), at the same quality.
Complexity estimated at about 10× HEVC for the encoder and about 2× for the decoder (thesis §2.2).
Video was forecast at 82 % of internet traffic by 2023 (Cisco, cited in the thesis).
Thirty years of standards. Thesis Fig. 3.3.
Sources: thesis §2.1–2.2 and §3.3; Bross et al., “Overview of the VVC standard”, IEEE TCSVT 2021.
An 8×8 block through the four steps. Darker means a larger value. The transform and the quantisation are computed, not drawn: the number of remaining coefficients is a real result.
2 / 10
What a codec does
A codec predicts each block of the picture from what it has already coded, keeps only the difference, and applies a transform that concentrates that difference into a few coefficients. The small coefficients are then rounded away. The decoder reverses these steps, and the encoder contains a full decoder so that both sides reconstruct the same picture.
Go deeper
Every modern codec is a hybrid coder: intra prediction (from neighbours in the picture) or inter prediction (from previous pictures, with motion), then transform, quantisation and entropy coding of the residual.
The encoder reconstructs exactly what the decoder will see, because future predictions must use the same pixels on both sides. In-loop filters clean that reconstruction before it becomes a reference.
Every encoder decision minimises a rate–distortion cost J = D + λ·R. Codecs are compared by Bjøntegaard delta rate under JVET common test conditions.
The figure runs a real 2-D DCT on its residual and quantises with a step of 16.
The VVC encoder with the decoder loop (grey box) inside it. Thesis Fig. 3.5.
Sources: thesis §3.2–3.4.
The three tools designed in the thesis, highlighted on both the encoder and the decoder side.
3 / 10
The three tools
VVC added two transforms, MTS and LFNST, and a new in-loop filter, ALF. Each one saves a few percent of bit rate, which is significant in this field. They are small additions in the standard but costly in hardware. My thesis covers these three tools, for the decoder and for the encoder.
Go deeper
MTS: the encoder picks DCT-2, DST-7 or DCT-8 per direction (5 combinations) for luma blocks; DCT-2 up to 64×64. Switching it off costs +1.3 % bit rate in intra coding (VTM 8).
LFNST: a second, non-separable transform on the low-frequency corner of DCT-2 blocks; 4 sets × 2 kernels. Off: +1.0 % (luma), +2 % (chroma) in intra coding.
ALF: a 7×7 luma diamond, a 5×5 chroma diamond and a cross-component filter. Off: +2.2 % (luma) and +12.8 % (chroma U) in intra coding; ALF is about a quarter of the software decoder’s time.
Hardware cost comes from more multiplications, more kernels to store (3 MTS, 8 LFNST, 64 fixed ALF filters) and, for ALF, the pixel neighbourhoods kept in memory.
The thesis map: which chapters cover which blocks. Thesis Fig. 11.1.
Sources: thesis §3.4, Tables 6.7 and 9.7, Fig. 9.12.
Energy for the same H.264 encoding, relative to a dedicated circuit (Hameed et al., ISCA 2010). Logarithmic scale.
4 / 10
Why a chip and not software
Live video has a deadline for every frame. The company I worked with makes fanless encoders for vehicles and portable use, so software was not an option: a processor spends most of its energy moving instructions and data rather than computing. A chip designed for one task avoids that. The drawback is that a chip cannot be changed after manufacture, so the design has to be correct before it is made.
Go deeper
The thesis cites a study by Xilinx: dedicated hardware (FPGA or ASIC) reaches higher quality at lower power than software for real-time video; and the rule of thumb of about 20× the performance of a CPU at about 20× lower power for an ASIC.
Hameed et al. measured a general-purpose processor at about 500× the energy of an ASIC for H.264 encoding; SIMD and VLIW extensions recovered about 10×; the remaining 50× needed custom datapaths.
Real-time encoders are judged on quality, latency, SWaP (size, weight, power) and feature set.
The partner’s previous (HEVC) generation ran on FPGAs; VVC moved to an ASIC once the standard was final (July 2020).
Flexibility against performance and power efficiency. Thesis Fig. 3.12.
Sources: thesis §3.5; Hameed et al., “Understanding sources of inefficiency in general-purpose chips”, ISCA 2010.
Four samples enter the transform chain at every clock cycle and move through its stages without stopping. In the chip a cycle lasts 2.2 nanoseconds; here it lasts half a second.
5 / 10
The throughput requirement
4K at 60 frames per second on a 450 MHz chip means the transform must process four samples every clock cycle, for every block size. Every block must also take the same number of cycles, so that the following blocks never wait. Within that requirement, the goal was to make each block as small as possible in area, memory and power.
Go deeper
Decoder: 4K120 over two cores → 4K60 per core at 450 MHz. 3840×2160×60 luma samples ×1.5 (4:2:0 chroma) ≈ 747 M samples/s ≈ 1.7 per cycle on average; the module is sized for the worst case (small blocks, no zeroing): 4 samples/cycle for the 1-D transform, 2 for the folded 2-D transform, 2 pixels/cycle for ALF.
Encoder: 4K60 over four cores → 1080p60 per core, with 9,000 cycles per 64×64 coding tree unit for all its transform candidates and loop filters.
Cost is reported after synthesis with Synopsys Design Compiler on TSMC 28 nm: cell area or gates, kilobits of RAM/ROM, milliwatts at 0.8 V and 85 °C.
The folded 2-D transform: one 1-D core used twice, with a transpose memory. Thesis Fig. 6.1.
Sources: thesis §5.1, §6.1, §7.1, §9.1, §10.1.
Each cycle, four samples are multiplied by coefficients read from the ROM, one pool of sixteen multipliers per sample, and four results leave.1-D inverse MTS at 2 samples/cycle, cell area after synthesis (28 nm, 450 MHz). Thesis Table 5.4.
6 / 10
The transform
The usual way to build a transform in hardware is to hard-wire its constants, and for HEVC that was the cheapest option. VVC has many more transform types and sizes, so I built both versions. The result was the opposite: 64 shared multipliers reading their coefficients from a small ROM took 63 % less silicon. The LFNST was added in the same way. We did not find an earlier hardware implementation of LFNST in the literature.
Go deeper
My first design (ICASSP 2020): 1-D inverse MTS at 2 samples/cycle. Constant multipliers (shift-and-add, one module per size and type): 266,315 cell area. Regular multipliers (32 shared, 1,872 coefficients in a 17.4 Kbit ROM): 96,849. The multi-standard engine also covers AVC and HEVC.
The second (IEEE TCE 2021): 4 samples/cycle. MTS scales to 64 multipliers (+55 % area for 2× throughput, 138.9 Kgates); inverse LFNST added with its own 64 multipliers and a 64 Kbit ROM of the 8 kernels (109.2 Kgates). Unified: 128 multipliers, fixed latency, bypass delay line for blocks without LFNST.
Techniques used: butterfly decomposition of DCT-2 (sizes nest), DCT-8 = DST-7 with reversed inputs and flipped signs, zeroing of high frequencies for 64-point DCT-2 and 32-point DST-7/DCT-8.
Cost in perspective: at equal throughput the VVC transform (MTS + LFNST, 163,672 gates) is about 4× the HEVC transform (41,500 gates); MTS alone about 2×.
Encoder side: three transform chains of 32 samples/cycle; the forward DCT-2 shares its core with the inverse DCT-2 (symmetric odd kernels), LFNST kept separate.
The unified inverse LFNST + MTS design and its two control units. Thesis Fig. 6.12.
The 7×7 luma diamond (darker = larger coefficient) and the 5×5 chroma diamond. The luma kernel is rotated or flipped according to the block’s class.
7 / 10
ALF, the adaptive loop filter
ALF is a 7×7 diamond-shaped filter whose coefficients the encoder computes and transmits. Each 4×4 block is classified into one of 25 classes, and the kernel is rotated or flipped accordingly. Because the filter is symmetric, one engine with 26 multipliers handles luma, chroma and the cross-component filter.
Go deeper
ALF is a Wiener filter: coefficients minimise the squared error between reconstruction and original; up to 25 luma sets and 8 chroma sets per adaptation parameter set, plus 64 fixed luma filters in the standard.
Point symmetry halves the multiplications: 12 unique luma coefficients, 6 chroma, 7 for the 3×4 cross-component filter.
Classification: sub-sampled Laplacian gradients in four directions give a directionality D (0–4) and an activity A (0–4); class = 5·D + A. The gradient ratios also choose rotation, vertical flip or diagonal flip of the kernel.
Filter engine: three pipeline stages (pixel selection → clip and add symmetric pairs → 26 multipliers), 2 pixels/cycle, one 4×4 block every 8 cycles, fixed latency for CTU sizes 16 to 128.
Luma 7×7 and chroma 5×5 diamonds; symmetric taps share a coefficient. Thesis Fig. 8.5.
Sources: thesis Ch. 8 and §9.5; Karczewicz et al., “VVC in-loop filters”, IEEE TCSVT 2021.
C · Cr, Cb, Y, Y per line (proposed)
memory0 Kbit
8 / 10
The memory problem
To filter one block, ALF needs the eight blocks around it, so blocks have to wait in memory. Feeding the filter in the order Cr, Cb, Y, Y keeps the luma needed by the chroma filter already in the pipeline, and only two lines of blocks have to be stored. The memory is reduced by three quarters. The three orders can be compared above: outlined blocks are the ones held in memory.
Go deeper
I compared three feeding orders for a 128×128 CTU (4:2:2, 10-bit): A, whole luma first, keeps the entire luma CTB for the cross-component filter: 193.28 Kbit. B, chroma first, parks the filtered chroma until luma arrives: 213.12 Kbit. C, interleaved Cr, Cb, Y, Y per CTU line: two lines and two columns of blocks per component, 50.56 Kbit (35.84 for 4:2:0).
Second problem: a 4×4 block at 2 pixels/cycle is an 8-cycle window, but 8 neighbour reads plus 1 write take 9 cycles. Consecutive blocks share 6 of their 9 neighbours, so the second block needs only 3 operations: 12 per pair in a 16-cycle window.
Two RAMs: one serving a 4×4 block per cycle (160-bit port), one holding a one-sample column for the cross-component filter (40-bit port). The memory module is 48 % of the ALF area.
The figure is a simplified model: the outlined blocks are the ones still held in memory, and the gauge rises to the value measured for each order.
The proposed scanning for a 32×32 CTU (4:2:0) and the block sequence over time. Thesis Fig. 9.6.
1-D inverse transform at 2 samples/cycle, gates after synthesis. Thesis Tables 6.10–6.11.
9 / 10
Results
Every block runs at 450 MHz on a 28 nm process with a fixed latency, and matches the reference software bit for bit: a hundred thousand random test vectors for the transforms, complete encoded sequences for the filter. The blocks were integrated in the company’s real-time encoder and decoder, and four papers came out of the work. One limitation: the encoder-side ALF handles only luma and 64×64 blocks, because the full version did not fit the cycle budget.
Go deeper
Decoder: inverse MTS 64 RM, 138.9 Kgates; inverse LFNST 64 RM, 109.2 Kgates, 64 Kbit ROM; ALF 26 RM, 66,997 µm², 50.56 Kbit RAM, 9.6 mW at 0.8 V (6.7 mW at 0.65 V for +12 % area).
Encoder: transform loop as three chains of 32 samples/cycle with a shared DCT/iDCT core; ALF luma filter 24 RM, 41,984 µm², 13.6 mW; ALF decision (4 filters + SSD) 48 RM, 58,628 µm², 101.6 Kbit, 17.9 mW. Encoder ALF limited to luma, CTU 64 and 4 of 16 filter sets.
Verification: 100,000 pseudo-random vectors per transform module against VTM procedures; ALF against trace files from the instrumented software encoder, sequences from 64×64 to 1080p in 8, 10 and 12 bits.
Possible next steps named in the thesis: one 64-multiplier core shared by MTS and LFNST; a hybrid transform with hard-wired constants for DCT-2 only; a learned shortcut for the encoder’s transform search; a chroma filter in the encoder ALF.
Sources: thesis §5.4, §6.5, §9.6, §10.4, Ch. 11; Appendix A.
Daniel Ménard
Thesis director
Professor, INSA Rennes / IETR (VAADER)
Olivier Déforges
Thesis co-director
Professor, INSA Rennes / IETR
Wassim Hamidouche
Supervisor
Associate Professor, INSA Rennes / IETR (VAADER)
10 / 10
The people
Daniel Ménard and Olivier Déforges directed the thesis, and Wassim Hamidouche supervised the work day to day, at the IETR laboratory of INSA Rennes. The industrial partner provided the constraints and the chip the designs were integrated into. I defended the thesis in Rennes on 31 May 2022.
Record
The record, for reference
Title
Efficient hardware designs of the new Versatile Video Coding (VVC) tools for ASIC platforms Conceptions matérielles efficaces pour des nouveaux outils du standard VVC ciblant les plates-formes ASIC
Author
Ibrahim (Brahim) Farhat
Defended
31 May 2022, Rennes · NNT 2022ISAR0009 · HAL tel-04477217 · 190 pages, in English
Institution
INSA Rennes · IETR, UMR CNRS 6164 (VAADER team) · doctoral school ED 601 MathSTIC · speciality Signal, Image et Vision
Setting
Industrial PhD: the designs were integrated in the partner’s real-time professional VVC encoder and decoder (AVC, HEVC and VVC on one chip)
Lightweight Hardware Implementation of VVC Transform Block for ASIC Decoder · IEEE ICASSP 2020 · doi · open access
2021
Lightweight Hardware Transform Design for the Versatile Video Coding 4K ASIC Decoders · IEEE Transactions on Consumer Electronics 67(4) · doi · open access
2022
Adaptive Loop Filter Hardware Design for 4K ASIC VVC Decoders · IEEE Transactions on Consumer Electronics 68(2) · doi · open access
2022
Efficient HW Design of Adaptive Loop Filter for 4K ASIC VVC Encoder · Picture Coding Symposium 2022 · doi · open access
