FastLanes: A Next-Gen F ile Format
Computer scientist Azim Afroozeh investigat ed how to redesign compression and storage methods so that data can b e processed much faster while using less space.
Afrooz
eh’s research focused on designing a new generation of data storage
formats that can keep up with modern computing hardware such as mult
i-core CPUs and GPUs. Today’s widely used formats, such as Parquet,
were created in an earlier era and no longer fully exploit the capab
ilities of modern processors. This mismatch wastes computing power an
d slows down data analysis.
In his work, Afroozeh investigated
how to redesign compression and storage methods so that data can be
processed much faster while using less space. He explored questions s
uch as: how do we reorganize data so it fits the parallel nature of t
oday’s hardware? How can we compress data in ways that remain extre
mely fast to decode?The resulting work, FastLanes, proposes a fundame
ntally new file format built for the hardware of today and tomorrow.<
/p>
He demonstrated that data files can be redesigned to be both mu
ch smaller and dramatically faster to read by aligning them with how
modern hardware actually works. The key insight is that data should b
e stored in a layout that allows thousands of values to be processed
in parallel, without bottlenecks.
The research shows that by r
eorganizing data and using new lightweight compression techniques, we
can decode billions of values per second - often faster than reading
uncompressed data. It also shows that these methods work not only on
CPUs but also on GPUs, which are increasingly used in analytics and
AI. In simple terms: computers can work far more efficiently when dat
a is stored in the “language” that modern processors prefer. Fast
Lanes proves that a file format designed with this principle in mind
can outperform current systems by a wide margin.
Afroozeh combi ned theoretical analysis with extensive practical experimentation. Fi rst, he studied how real-world datasets behave and how modern process ors - both CPUs and GPUs - handle data in parallel. Based on these in sights, he designed new compression layouts and algorithms tailored t o modern hardware. He then implemented all methods in high-performanc e C++ and evaluated them experimentally on many architectures, includ ing Intel, AMD, Apple, Amazon Graviton, and NVIDIA GPUs. This allowed him to measure speed, storage savings, and integration into real que ry engines. Finally, he developed a complete prototype file format an d validated it using real analytical workloads. All implementations w ere made open-source to ensure reproducibility and practical value.
More information on the thesis
DESCRIPTION: Computer scientist Azim Afroozeh investigated how to redesign compression and storage methods so that data can be proc essed much faster while using less space. Afroozeh’s resea rch focused on designing a new generation of data storage formats tha t can keep up with modern computing hardware such as multi-core CPUs and GPUs. Today’s widely used formats, such as Parquet, were create d in an earlier era and no longer fully exploit the capabilities of m odern processors. This mismatch wastes computing power and slows down data analysis.