FastLanes: A Next-Ge n File Format
Computer scientist Azim Afroozeh inv estigated how to redesign compression and storage methods so that dat a can be processed much faster while using less space.
Afroozeh’s research focused on designing a new generation of data
storage formats that can keep up with modern computing hardware such
as multi-core CPUs and GPUs. Today’s widely used formats, such as P
arquet, were created in an earlier era and no longer fully exploit th
e capabilities of modern processors. This mismatch wastes computing p
ower and slows down data analysis.
In his work, Afroozeh inves
tigated how to redesign compression and storage methods so that data
can be processed much faster while using less space. He explored ques
tions such as: how do we reorganize data so it fits the parallel natu
re of today’s hardware? How can we compress data in ways that remai
n extremely fast to decode?The resulting work, FastLanes, proposes a
fundamentally new file format built for the hardware of today and tom
orrow.
He demonstrated that data files can be redesigned to be
both much smaller and dramatically faster to read by aligning them wi
th how modern hardware actually works. The key insight is that data s
hould be stored in a layout that allows thousands of values to be pro
cessed in parallel, without bottlenecks.
The research shows th
at by reorganizing data and using new lightweight compression techniq
ues, we can decode billions of values per second - often faster than
reading uncompressed data. It also shows that these methods work not
only on CPUs but also on GPUs, which are increasingly used in analyti
cs and AI. In simple terms: computers can work far more efficiently w
hen data is stored in the “language” that modern processors prefe
r. FastLanes proves that a file format designed with this principle i
n mind can outperform current systems by a wide margin.
Afrooze h combined theoretical analysis with extensive practical experimentat ion. First, he studied how real-world datasets behave and how modern processors - both CPUs and GPUs - handle data in parallel. Based on t hese insights, he designed new compression layouts and algorithms tai lored to modern hardware. He then implemented all methods in high-per formance C++ and evaluated them experimentally on many architectures, including Intel, AMD, Apple, Amazon Graviton, and NVIDIA GPUs. This allowed him to measure speed, storage savings, and integration into r eal query engines. Finally, he developed a complete prototype file fo rmat and validated it using real analytical workloads. All implementa tions were made open-source to ensure reproducibility and practical v alue.
More information on the thesis
DESCRIPTION: FastLanes: A Next-Gen File Format Computer scient ist Azim Afroozeh investigated how to redesign compression and storag e methods so that data can be processed much faster while using less space.Afroozeh’s research focused on designing a new gener ation of data storage formats that can keep up with modern computing hardware such as multi-core CPUs and GPUs. Today’s widely used form ats, such as Parquet, were created in an earlier era and no longer fu lly exploit the capabilities of modern processors. This mismatch wast es computing power and slows down data analysis.