Sweet! Smart caches for remote arrays.

Blosc Development Team@blosc.org · 4d ago

📢 Python-Blosc2 4.12.0 is out! Remote arrays work over S3/GCS/HTTP via fsspec. lazy=True fetches only the blocks needed and reuses a validated cache — 5–17x faster in S3 benchmarks. Also: concurrent Caterva2 writers + UTF-8 indexes. github.com/Blosc/python... Compress Better, Compute Bigger 🚀

Dicts are nice data structures, and super-useful for storing repeated data.

Blosc Development Team@blosc.org · 2w ago

💡 Blosc2 tip: repeated text? use dictionary(): one int32 code per row, one copy of each value. On 1 Mrow with 100 distinct titles: group_by 22x faster, isin() 5.7x, 5.6x smaller on disk. blosc.org/python-blosc2/guides/optimization_tips.html#repeated-text-store-it-as-dictionary Enjoy data 🚀

Cities aren't LOUD CARS ARE LOUD #Urbanism #BikeTooter #Cycling #TheWarOnCars Great video by u/MiserNYC- on Reddit

📣 Python-Blosc2 4.11.0 continues aligning with Arrow conventions. New CTable nullability on a validity mask instead of a sentinel value ✨️ Also, wheels are now abi3 compliant w/ CPython 3.11+, including 3.15 (tested) 🚅 More info: github.com/Blosc/python... Compress Better, Compute Bigger 🚀

Releases · Blosc/python-blosc2

A high-performance library for compressed ND arrays and columnar tables, with compute and indexing engines - Blosc/python-blosc2

github.com

The JIT in Blosc2 is quite capable, but requires some love to squeeze the last drop of performance out of it.

Blosc Development Team@blosc.org · 4w ago

💡 Blosc2 tip: @blosc2.jit can compile complex control flow with loops and conditionals. A Mandelbrot escape loop written plainly: ~155x faster than plain Python, ~2.3x than hand-vectorized NumPy. blosc.org/python-blosc... Compress Better, Compute Bigger 🚀 #Python #Blosc2

💡 Blosc2 tip: for unbounded text, use utf8(). blosc2.field(blosc2.utf8()) string(200) costs 800 B/row on every read; utf8() rows cost what they weigh: ~4× less memory. blosc.org/python-blosc... Compress Better, Compute Bigger 🚀 #Python #DataScience #Blosc2

Hey Austin!! I’m very proud to announce the grand opening of the Austin STEM Center, a unique space for hands-on science learning! Come to our grand opening, bring your kids, and see all that we have to offer! Play with robots, check out our fab and maker spaces, and help us Keep Austin Curious!!

Austin STEM Center@austinstemcenter.org · 4w ago

✨ Our new sign is officially up! 🤩 The countdown is on! Join us for the Grand Opening of Austin STEM Center on Saturday, August 22 | 1:00–6:00 PM. 🎟️ Register here: buff.ly/0BROdzc We can't wait to welcome you! 🚀 #AustinSTEMCenter #GrandOpening #STEMEducation #AustinFamilies #ComingSoon

On-disk broadcasting can save lots of memory.

Blosc Development Team@blosc.org · last mo.

💡 Blosc2 tip: build big on-disk arrays from small ones. (rows + cols).compute(urlpath="big.b2nd", mode="w") Chunk by chunk: similar speed, ~26× less peak memory. blosc.org/python-blosc... Compress Better, Compute Bigger 🚀 #Python #Blosc2

Most compression libraries ask you to move in. We think that's backwards. Blosc2 4.9.1: DuckDB, Polars and pandas 3 read a CTable directly via Arrow's PyCapsule protocol — no conversion step. Cooperation, not completeness. blosc.org/posts/not-an... Compress Better, Compute Bigger 🚀 #DataScience

💡 Blosc2 tip: mmap read-only opens — skip the syscalls. >>> arr = blosc2.open(path, mmap_mode="r") Maps once, reads pages directly. ~15% faster for scattered reads, same memory. Free win, and scales with concurrent readers 🚀 blosc.org/python-blosc... #Python #DataScience #Blosc2

💡 Blosc2 tip: `where=` for filtered reductions. t["temperature"].sum(where=t.region == 3) ~1.9× faster, ~7× less memory than mask-then-index. blosc.org/python-blosc... Compress Better, Compute Bigger 🚀 #Python #DataScience #Blosc2

💡 Python-Blosc2 tip #7: don't slice a column to reduce it. t["val"].sum() reduces the compressed column chunk-wise, never materialising it: 1.7× faster, ~12× less peak memory on 50M rows. blosc.org/python-blosc... Compress Better, Compute Bigger 🚀

💡 Python-Blosc2 tip #6: generate arrays with DSL kernels, not NumPy A @blosc2.dsl_kernel handed to lazyudf() compiles to native code, filling an NDArray chunk by chunk, in parallel. ~3.6x faster, ~2x less peak memory on 200M elements. blosc.org/python-blosc... Compress Better, Compute Bigger 🚀

💡 Python-Blosc2 tip #4: let SUMMARY indexes answer min()/max() directly CTable auto-builds per-block min/max indexes for its scalar columns, so Column.min()/max() never decompress the column: ~4× faster, essentially no extra memory. blosc.org/python-blosc... Compress Better, Compute Bigger 🚀

💡 Python-Blosc2 tip #3: align your reads with the double partition A chunk-aligned read decompresses 1 chunk instead of 2 (~2.2× faster), and chunk-aligned slice() copies chunks as-is with no decompression at all (~4.9× faster). Same principle at block level. blosc.org/python-blosc... Enjoy data!

💡 Python-Blosc2 tip #2: many readers, one file? Open with mmap_mode="r" in every reader — all share one set of mapped pages instead of a syscall + copy per read. 8 concurrent readers: ~4.5x faster, half the CPU, still ~one copy of the file in RAM. blosc.org/python-blosc... Enjoy data! 🚀

💡 Python-Blosc2 tip #1: skip the NumPy detour. blosc2.linspace(0, 1, N) fills a compressed array chunk by chunk — at 200M float64, ~25x less peak memory than asarray(np.linspace(...)), comparable speed. Same for arange() and fromiter(). blosc.org/python-blosc... Compress Better, Compute Bigger 🚀

This is what I have been using Fable 5 for during this week. I should say that I am quite impressed on the good taste that the beast is showing for API design. Really excited on the perspective that open models would reach this level soon!

Blosc Development Team@blosc.org · 2mo ago

📢 Python-Blosc2 4.8.0: share on-disk compressed containers across processes! SWMR readers follow a writer's appends in near real-time, and opt-in locking makes updates atomic — no torn reads, no retries. 🎬 One writer appending, three readers chasing it live: www.blosc.org/python-blosc... #Python

Python-Blosc2 now has a 🔥 DSL-to-JavaScript transpiler! Newton-Raphson runs 2.7× faster on the browser! 💨 With #Pyodide, it’s not just Python in the browser: it’s the PyData stack 📊 too, #Blosc2 tacking compression 📦 + computing 🧮 duties. Demo 👇 blosc.org/demos/newton...

Yo no puedo ser tan optimista 👇🏽 science.nasa.gov/earth/explor...

Carbon Dioxide - Earth Indicator - NASA Science

Carbon dioxide (CO2) is an important greenhouse gas. Greenhouse gases trap the heat from sunlight, warming the planet. Without any greenhouse gases, Earth

science.nasa.gov

Both amazed and humbled of how much you can build with modern LLMs, #Textual and #Blosc2 . Seriously, if there is something that a LLM can boost creativity, like a lot, this is TUI making.

Blosc Development Team@blosc.org · 3mo ago

✨ Tabular data visualization, fast filtering, smart navigation, remote plotting, and more... with just a text terminal! This is the new b2view Text User Interface that comes with latest Python-Blosc2 release. github.com/Blosc/python... Enjoy!

🚀 New Blosc post: one selective query, 24.3M taxi trips, five tools. ⚡ Cold: CTable/.b2z ~1.9x faster than DuckDB, 9.4x vs pandas 🎯 Block indexes skip ~90% of the data 🤝 Warm: dead heat with DuckDB 💾 Only +2% disk vs Parquet blosc.org/posts/ctable... #Python #DuckDB

Pyodide 314.0.0 is out! 🎉 This marks a significant milestone for the Python-in-the-browser ecosystem! PEP 783 has been accepted, so Pyodide wheels can now be published directly to PyPI 🐍📦 blog.pyodide.org/posts/314-re... Thanks to @pydantic.dev for their guide: pydantic.dev/articles/ems...

This is our most playful release so far. Under the hood we did amazing improvements, making toying with very large tabular data a pleasant, NumPy-centric experience. Have fun!

GitHub - Blosc/python-blosc2: A high-performance library for compressed ND arrays and columnar tables, with compute and indexing engines

A high-performance library for compressed ND arrays and columnar tables, with compute and indexing engines - Blosc/python-blosc2

github.com

Blosc Development Team@blosc.org · 3mo ago

📣 Python-Blosc2 4.4.x is out! 🖥️ b2view: interactive TUI browser 🎯 SUMMARY indexes → fast WHERE 🧮 DSL kernels as CTable columns ⚡ Faster chunk-by-chunk writes 🔀 where() via miniexpr Compressed, NumPy-native arrays & tables. 📝 github.com/Blosc/python... Enjoy data! #Python #NumPy

This opens the door to exciting new developments in the Python-Blosc2 wrapper. Stay tuned!

Blosc Development Team@blosc.org · 3mo ago

🚀 C-Blosc2 3.1.0 is here! New in this release: sparse coordinate getters for faster random access, header-only b2nd metalayer access for plugins (no libblosc2 link needed), and new codec IDs for J2K/HTJ2K support. 🧩⚡ Release notes: github.com/Blosc/c-blos... Compress better, compute bigger! 💥

Always honored to see our work being broadly used! 🚀

Blosc Development Team@blosc.org · 4mo ago

Reaching 5 milions monthly downloads in PyPI is something to celebrate! 🥳 🥳 Thanks to all our users! This makes us happy for continuing working tirelessly for providing high quality compression for storage, computation for arrays, and now, for tabular data too! 🙏 #DataCompression #Computation

Under the new indexing engine is code that I figured out and implemented about 20 years ago... But it still serves pretty well (with a touch of modernization indeed).

Blosc Development Team@blosc.org · 4mo ago

Did you know that in recent Python-Blosc2 4.2.0 we released extremely efficient indexing engines that can store data in (guess what) ...compressed state? As a result, much larger tables can be indexed. Look at how this fares against other good indexing engines in plots below. Enjoy! #TabularData

Have CTable a try and tell us how it goes! We distilled all our tabular skills there 👇

Blosc Development Team@blosc.org · 4mo ago

Python-Blosc2 4.2.0 is out 🎉 Meet `CTable`: a new compressed columnar container with null support, fast queries, and easy Arrow/Parquet interoperability. Compress better, analyze faster. Release notes: github.com/Blosc/python...

The beast is out of the cage! 🧸

Blosc Development Team@blosc.org · 4mo ago

🚀 C-Blosc2 3.0.0 is here! With variable length support, better compression, streamlined threading and more safety protections, C-Blosc2 3.0.0 makes compressed, persistent binary data storage more scalable, flexible, and production-ready. Release notes: github.com/Blosc/c-blos... Enjoy!

📣 C-Blosc2 3.0.0-rc2 is out 🚀 Big highlight: a new shared managed thread pool for parallel execution. ✅ With this and previous work, C-Blosc2 will become specially useful for storing variable length data, using extremely low resources. github.com/Blosc/c-blos... #CBlosc2 #Compression #OpenSource