OpenKL Engine
High-throughput hardware-accelerated DataFrame engine for Python.
Abstract
OpenKL (openkl.DataFrame) is a high-throughput, hardware-accelerated tabular data processing engine written in C++20 for Python. Engineered to eliminate traditional DataFrame bottlenecks, OpenKL features 64-byte cache alignment, AST expression fusion, SIMD vectorization (AVX-512 & ARM NEON), and multi-GPU streaming dispatch—delivering up to 573× speedups over standard Python DataFrames.
Technical Specifications
64-byte cache-aligned columnar memory buffers for AVX-512 & ARM NEON
JIT expression compilation via AST fusion, eliminating temporary intermediate allocations
Multi-GPU & multi-stream asynchronous kernel dispatch for large tabular workloads
Seamless Pandas API interoperability with zero-copy Arrow memory buffers
Parallel chunked CSV/Parquet parsers with auto memory-mapped IO
Fast predicate pushdown, grouped aggregations, and rolling transforms
Audience Profile
- Data Engineers
- Quantitative Researchers
- High-Performance Computing Specialists
- ML Practitioners
Research Roadmap
Q4 2026: OpenKL v2.1 Apache Arrow zero-copy memory transport & Flight RPC
Q1 2027: Apple Silicon Metal Performance Shaders (MPS) unified memory backend
Q2 2027: Distributed out-of-core cluster execution with NVLink interconnects
Q3 2027: Native SQL query engine interface and integration with PyTekt pipelines