Pavilion™ Data Systems, a leading data analytics acceleration platform provider and a pioneer of NVMe-Over-Fabrics (NVMe-oF), announced a partnership with Los Alamos National Laboratories (LANL) to co-develop and evaluate acceleration of analytics by offloading analytics functions from storage servers to the storage array, minimizing data movement by enabling data reduction near the storage.
LANL is moving their I/O from file based to record or column based, which enables analytics to be done using tools from the big data/analytics community. LANL has shown 1000X speedups on analytics functions by leveraging data reduction near the storage devices via their DeltaFS technology. Shaping standards and commercialization of storage capabilities to assist in leveraging analytics near the storage is at the heart of the Pavilion/LANL partnership.
“For decades, our large-scale physics simulations used file-based I/O, as these simulations were tuned for disk-based parallel file systems,” stated Gary Grider, High-Performance Computing Division Leader at LANL. “The file-based I/O had hidden the inherent structure in the data in those files. Switching to record/column-based I/O exposes the structure in the data, thereby enabling analytics. Leveraging technology from the big data/analytics community is a natural method to accomplish this, and partnering with industry to explore this at extreme scale is the basis for this partnership with Pavilion.”
“We are very excited about partnering with LANL on this high-performance innovation, as together we explore the use of HyperOS™ functionality to enable offloaded analytics from extreme-scale scientific applications,” stated Dario Zamarian, Pavilion CEO.
The data processing algorithms of Pavilion HyperOS coupled with the performance density of the Pavilion HyperParallel™ Flash Array provides a highly performant computational storage array capability enabling analytics offloads at scale.
Los Alamos National Laboratory, a multidisciplinary research institution engaged in strategic science on behalf of national security, is managed by Triad, a public service-oriented, national security science organization equally owned by its three founding members: Battelle Memorial Institute (Battelle), the Texas A&M University System (TAMUS), and the Regents of the University of California (UC) for the Department of Energy’s National Nuclear Security Administration.