💡 Blosc2 tip: repeated text? use dictionary(): one int32 code per row, one copy of each value. On 1 Mrow with 100 distinct titles: group_by 22x faster, isin() 5.7x, 5.6x smaller on disk. blosc.org/python-blosc2/guides/optimization_tips.html#repeated-text-store-it-as-dictionary Enjoy data 🚀

📣 Python-Blosc2 4.11.0 continues aligning with Arrow conventions. New CTable nullability on a validity mask instead of a sentinel value ✨️ Also, wheels are now abi3 compliant w/ CPython 3.11+, including 3.15 (tested) 🚅 More info: github.com/Blosc/python... Compress Better, Compute Bigger 🚀

Releases · Blosc/python-blosc2

A high-performance library for compressed ND arrays and columnar tables, with compute and indexing engines - Blosc/python-blosc2

github.com

💡 Blosc2 tip: @blosc2.jit can compile complex control flow with loops and conditionals. A Mandelbrot escape loop written plainly: ~155x faster than plain Python, ~2.3x than hand-vectorized NumPy. blosc.org/python-blosc... Compress Better, Compute Bigger 🚀 #Python #Blosc2

💡 Blosc2 tip: for unbounded text, use utf8(). blosc2.field(blosc2.utf8()) string(200) costs 800 B/row on every read; utf8() rows cost what they weigh: ~4× less memory. blosc.org/python-blosc... Compress Better, Compute Bigger 🚀 #Python #DataScience #Blosc2

💡 Blosc2 tip: build big on-disk arrays from small ones. (rows + cols).compute(urlpath="big.b2nd", mode="w") Chunk by chunk: similar speed, ~26× less peak memory. blosc.org/python-blosc... Compress Better, Compute Bigger 🚀 #Python #Blosc2

Most compression libraries ask you to move in. We think that's backwards. Blosc2 4.9.1: DuckDB, Polars and pandas 3 read a CTable directly via Arrow's PyCapsule protocol — no conversion step. Cooperation, not completeness. blosc.org/posts/not-an... Compress Better, Compute Bigger 🚀 #DataScience

💡 Blosc2 tip: mmap read-only opens — skip the syscalls. >>> arr = blosc2.open(path, mmap_mode="r") Maps once, reads pages directly. ~15% faster for scattered reads, same memory. Free win, and scales with concurrent readers 🚀 blosc.org/python-blosc... #Python #DataScience #Blosc2

💡 Blosc2 tip: `where=` for filtered reductions. t["temperature"].sum(where=t.region == 3) ~1.9× faster, ~7× less memory than mask-then-index. blosc.org/python-blosc... Compress Better, Compute Bigger 🚀 #Python #DataScience #Blosc2

💡 Python-Blosc2 tip #7: don't slice a column to reduce it. t["val"].sum() reduces the compressed column chunk-wise, never materialising it: 1.7× faster, ~12× less peak memory on 50M rows. blosc.org/python-blosc... Compress Better, Compute Bigger 🚀

💡 Python-Blosc2 tip #6: generate arrays with DSL kernels, not NumPy A @blosc2.dsl_kernel handed to lazyudf() compiles to native code, filling an NDArray chunk by chunk, in parallel. ~3.6x faster, ~2x less peak memory on 200M elements. blosc.org/python-blosc... Compress Better, Compute Bigger 🚀

💡 Python-Blosc2 tip #5: sorted top-k? Stream it from a FULL index CTable.sort_by(view=True)[:k] and NDArray.iter_sorted(start=-k) read just that slice from the index sidecar instead of sorting everything: up to ~74× less time, ~193× less peak memory. blosc.org/python-blosc... Enjoy data! #Python

💡 Python-Blosc2 tip #4: let SUMMARY indexes answer min()/max() directly CTable auto-builds per-block min/max indexes for its scalar columns, so Column.min()/max() never decompress the column: ~4× faster, essentially no extra memory. blosc.org/python-blosc... Compress Better, Compute Bigger 🚀

💡 Python-Blosc2 tip #3: align your reads with the double partition A chunk-aligned read decompresses 1 chunk instead of 2 (~2.2× faster), and chunk-aligned slice() copies chunks as-is with no decompression at all (~4.9× faster). Same principle at block level. blosc.org/python-blosc... Enjoy data!

💡 Python-Blosc2 tip #2: many readers, one file? Open with mmap_mode="r" in every reader — all share one set of mapped pages instead of a syscall + copy per read. 8 concurrent readers: ~4.5x faster, half the CPU, still ~one copy of the file in RAM. blosc.org/python-blosc... Enjoy data! 🚀

💡 Python-Blosc2 tip #1: skip the NumPy detour. blosc2.linspace(0, 1, N) fills a compressed array chunk by chunk — at 200M float64, ~25x less peak memory than asarray(np.linspace(...)), comparable speed. Same for arange() and fromiter(). blosc.org/python-blosc... Compress Better, Compute Bigger 🚀

📢 Python-Blosc2 4.8.0: share on-disk compressed containers across processes! SWMR readers follow a writer's appends in near real-time, and opt-in locking makes updates atomic — no torn reads, no retries. 🎬 One writer appending, three readers chasing it live: www.blosc.org/python-blosc... #Python

Python-Blosc2 now has a 🔥 DSL-to-JavaScript transpiler! Newton-Raphson runs 2.7× faster on the browser! 💨 With #Pyodide, it’s not just Python in the browser: it’s the PyData stack 📊 too, #Blosc2 tacking compression 📦 + computing 🧮 duties. Demo 👇 blosc.org/demos/newton...

✨ Tabular data visualization, fast filtering, smart navigation, remote plotting, and more... with just a text terminal! This is the new b2view Text User Interface that comes with latest Python-Blosc2 release. github.com/Blosc/python... Enjoy!

🚀 New Blosc post: one selective query, 24.3M taxi trips, five tools. ⚡ Cold: CTable/.b2z ~1.9x faster than DuckDB, 9.4x vs pandas 🎯 Block indexes skip ~90% of the data 🤝 Warm: dead heat with DuckDB 💾 Only +2% disk vs Parquet blosc.org/posts/ctable... #Python #DuckDB

📣 Python-Blosc2 4.4.x is out! 🖥️ b2view: interactive TUI browser 🎯 SUMMARY indexes → fast WHERE 🧮 DSL kernels as CTable columns ⚡ Faster chunk-by-chunk writes 🔀 where() via miniexpr Compressed, NumPy-native arrays & tables. 📝 github.com/Blosc/python... Enjoy data! #Python #NumPy

🚀 C-Blosc2 3.1.0 is here! New in this release: sparse coordinate getters for faster random access, header-only b2nd metalayer access for plugins (no libblosc2 link needed), and new codec IDs for J2K/HTJ2K support. 🧩⚡ Release notes: github.com/Blosc/c-blos... Compress better, compute bigger! 💥

Reaching 5 milions monthly downloads in PyPI is something to celebrate! 🥳 🥳 Thanks to all our users! This makes us happy for continuing working tirelessly for providing high quality compression for storage, computation for arrays, and now, for tabular data too! 🙏 #DataCompression #Computation

Did you know that in recent Python-Blosc2 4.2.0 we released extremely efficient indexing engines that can store data in (guess what) ...compressed state? As a result, much larger tables can be indexed. Look at how this fares against other good indexing engines in plots below. Enjoy! #TabularData

Python-Blosc2 4.2.0 is out 🎉 Meet `CTable`: a new compressed columnar container with null support, fast queries, and easy Arrow/Parquet interoperability. Compress better, analyze faster. Release notes: github.com/Blosc/python...

🚀 C-Blosc2 3.0.0 is here! With variable length support, better compression, streamlined threading and more safety protections, C-Blosc2 3.0.0 makes compressed, persistent binary data storage more scalable, flexible, and production-ready. Release notes: github.com/Blosc/c-blos... Enjoy!

Releases · Blosc/c-blosc2

A fast, compressed, persistent binary data store library for C. - Blosc/c-blosc2

github.com

📣 C-Blosc2 3.0.0-rc2 is out 🚀 Big highlight: a new shared managed thread pool for parallel execution. ✅ With this and previous work, C-Blosc2 will become specially useful for storing variable length data, using extremely low resources. github.com/Blosc/c-blos... #CBlosc2 #Compression #OpenSource

🚀 C-Blosc2 3.0.0 RC1 is out! What's new? 📦 VL-blocks for irregular data (strings, JSON) 🗜️ LZ4/LZ4HC dictionary compression 🛡️ Important safety fixes Check out the full release notes here: github.com/Blosc/c-blos... Give it a spin and provide feedback! #Blosc2 #OpenSource #C

Look at the bump in performance that we will see with the next Python-Blosc2 release. Matrix multiplication has been speeded up by using blocks and Blosc2 prefilters and its own efficient and multithreaded engine. Expect between 5x and 6x better speed for matrices with no padding.

We designed DSL kernels in Blosc2 so that they can be used in the main platforms; nowadays this necessarily includes Web Assembly (WASM) in the browser. We also implemented a new JIT compiler for WASM specially meant for Blosc2 DSL kernels. Play with it at: cat2.cloud/demo/roots/@... Have fun!

New DSL kernels in Python-Blosc2, being parallelizable and JIT-compiling capable, can accelerate code quite a bit 🚀 For example, linspace() has been rewritten to use DSL kernels, and performance boost is well beyond expectations.