From the One Billion Row Challenge to a Multi-Threaded Parquet Library

An airhacks.fm conversation with Gunnar Morling about:

returning guest discussion four years after episode "#174 Kafka Connect CLI,

JFR Unit,

OSS Archetypes and JPMS",

the One Billion Row Challenge (1BRC) as a Java community coding challenge,

aggregating a 13 gigabyte semicolon separated file of one billion temperature measurements per weather station,

baseline single threaded idiomatic Java at five minutes versus a 1.5 second winning entry,

two orders of magnitude speedup,

parallel processing scaling near linearly across cores,

memory mapped files via the Foreign Function and Memory API for arbitrarily large files,

reading eight bytes at once into a long,

SIMD parsing via the incubating Vector API,