DuckDB Secure Remote Storage community extension! - Now with session resumption! #duckdb #opaque #pake #boilstream github.com/dforsber/boi...

GitHub - dforsber/boilstream-extension: Secure Remote Secrets Storage for DuckDB

Secure Remote Secrets Storage for DuckDB. Contribute to dforsber/boilstream-extension development by creating an account on GitHub.

github.com

🚀 Scalable Browser click/event streams to BoilStream! github.com/boilingdata/...

GitHub - boilingdata/boilstream: Diskless high-throughput data ingestion (Kafka, Flight) server with local DuckDB caching. Writes optimised Parquet files onto Object Storages.

Diskless high-throughput data ingestion (Kafka, Flight) server with local DuckDB caching. Writes optimised Parquet files onto Object Storages. - boilingdata/boilstream

github.com

🏠 BoilStream Tech Architecture! ⁉️ On my post at LinkedIn: www.linkedin.com/posts/danfor...

#duckdb #kafka #arrow #flightsql #postgres #rust | Dan Forsberg

🏠 BoilStream Technical Architecture! ⁉️ 📋 🧠 Here is technical architecture of BoilStream - A DuckDB Server with Steroids! 🤗 We have been playing with Candle AI/ML pipelines as well, but it's not y...

linkedin.com

💯 👉🏻 Power BI now can read BoilStream v0.7.10 types! github.com/boilingdata/...

GitHub - boilingdata/boilstream: Diskless high-throughput data ingestion (Kafka, Flight) server with local DuckDB caching. Writes optimised Parquet files onto Object Storages.

Diskless high-throughput data ingestion (Kafka, Flight) server with local DuckDB caching. Writes optimised Parquet files onto Object Storages. - boilingdata/boilstream

github.com

⁉️ First in the world? Kafka interface to DuckDB! 🦆 www.linkedin.com/posts/danfor...

BoilStream - A Streaming LakeHouse | Dan Forsberg

🎉 🧠 Kafka interface to DuckDB! What do you say Stanislav Kozlovski, why not Kafka directly into DWH? 😄 🚀 BoilStream v0.7.7 introduces initial version of high performance Kafka interface! 🚕 It runs...

linkedin.com

🚀🎉 BoilStream v0.7.5 out! Power BI now works with all data types! See the demo_database.sql test suite on GitHub. NOTE: Power BI uses old version of npgsql, so we had to downgrade some temporal (TEXT) and list types (JSON). But only for Power BI clients. github.com/boilingdata/...

GitHub - boilingdata/boilstream: Diskless high-throughput data ingestion server with local DuckDB caching. Writes optimised Parquet files onto Object Storages.

Diskless high-throughput data ingestion server with local DuckDB caching. Writes optimised Parquet files onto Object Storages. - boilingdata/boilstream

github.com

🚀 Run MS Power BI withour DuckDB Server! 🦀 We added extensive Postgres compatible interface and put it around DuckDB as a server with Rust. You can use psql, bgbench, DBeaver, Grafana, Superset, ..and Power BI. #powerbi #duckdb #analytics Download for free! boilstream.com

🚀 BoilStream is a DuckDB server with Postgres interface for BI Tools. For data ingestion, just use DuckDB Airport community extension and you get GBs/s data ingestion speed to DuckDB database while running realtime analytics over the postgres interface. github.com/boilingdata/...

🦆 How fast can @DuckDB ingest data? 🚄 It's much more than I thought. You can read about it on BoilStream website and Documentation. 🚀 We have new BoilStream release which lands data on local DuckDB db AND on S3 Parquet at the same time. www.boilstream.com

💡 You can now have multiple storage backends with BoilStream, like S3 and Filesystem. Or two different S3 buckets, or Minio S3 and AWS S3, or.. www.boilstream.com

🚀 Skip the Pipeline. Stream to Gold. 🏆 You can write CREATE VIEW from DuckDB client against our Stream Processor server (DuckDB Airport extension) and it forks the base topic into derived topic(s) for you while it runs zero-copy SQL over the incoming data stream in real-time. www.boilstream.com

⫚ SQL stream processing with zero-copy over Arrow IPC data with DuckDB 🦆 🏹 github.com/Query-farm/a...

Add support for Airport CREATE VIEW statement by dforsber · Pull Request #20 · Query-farm/airport

Support for CREATE VIEW This PR adds support for CREATE VIEW statements on the Airport extension. Motivation I would like to add support for materialised views on my FlightRPC Data Ingestion Server...

github.com

🔐 BoilStream now supports all major SSO providers 🚀 The paid version includes support for TLS and Bearer token authentication for all major SSO providers: Azure AD (Entra ID), AWS Cognito, GCP, Auth0, and Okta. And includes 3-level authorization (groups, scopes, id). www.boilstream.com

🚀 Launching boilstream.com - Making streaming data dead simple. Directly from DuckDB into Parquet files on S3 from thousands of concurrent writers and GBs/s with single node. 🔥 See DEMO on the web page.

70k RPS with 1KB packets with single thread on arm64 macbook pro M3 (Nov 2023). Not bad.

Update README! Example MCP server. Connect Claude Desktop to your MCP compatible Server. github.com/boilingdata/...

GitHub - boilingdata/mcp-server-and-gw: An MCP stdio to HTTP SSE transport gateway with example server and MCP client

An MCP stdio to HTTP SSE transport gateway with example server and MCP client - boilingdata/mcp-server-and-gw

github.com

The total time to perform a cold run of the TPC-H queries on SF300 shows an almost 3× speedup between DuckDB 1.0.0 and DuckDB 1.2.0. The benchmark was run on a MacBook Pro (M1 Max / 64 GB RAM).

We have implemented the basic data analytics requirement 😍. We can reliably consume Postgres CDC stream with ephemeral node, including DR without full resync needs. The system is transaction safe with Icebreg Table on S3 target and can even detect if Postgres was restored from a backup. Interested?

Have been running Claude Code (agent based) and like it a lot. You still need to keep control and verify direction and quality.

❄️ Snowflake integration into Boiling Insights. Copy tables efficiently with optional sampling to local DuckDB! Copied tables are automatically attached and detached on delete. Easy to run queries.. #snowflake #duckdb #boilinginsights youtu.be/GisDleU8PDE

2025 02 13 Snowflake Integration

YouTube video by BoilingData

youtu.be

🚀 New video coming tomorrow about ❄️ Snowflake integration into Boiling Insights, and table sync to DuckDB db.. the table jobs run on bg using an optimal DAG executor and the GUI shows progress.

Small Remote Backends demo. youtu.be/m3DV3GlDu-M

Introducing Boiling Insights Remote Backends

YouTube video by BoilingData

youtu.be

Zig is amazing as is DuckDB!! Created for fun and as an example fixed column text file scanner extension for DuckDB (DanskeBank KTON file). By no means production proof and it actually misses couple of columns in the end. But this was less than an evening work. github.com/dforsber/duc...

GitHub - dforsber/duckdb-kton-extension: DuckDB KTON extension

DuckDB KTON extension. Contribute to dforsber/duckdb-kton-extension development by creating an account on GitHub.

github.com

We are happy to announce DuckDB v1.2.0 “Histrionicus”! The new release has several usability, security and performance improvements. It has new features for the CSV and Parquet formats, as well new (opt-in) options for DuckDB's own format. Read more at duckdb.org/2025/02/05/a...

🎉 New Boiling Insights version available along with support for Remote Backends! From single-user local to multi-user distributed/centralised! Beefed up DuckDB hosted experience with integrations to AWS DynamoDB and Glue Catalog. Application and Backend container. Simple. www.boilinginsights.com

What is the current best practice lib(s) to establish secure datagram connection from browser to nodejs based server?

⁉️ Help me out here: How to manage multidimensional configuration? We have 3 dimensions in Boiling Insights app: - Users (with roles), - Projects (e.g. GitHub), and - Backends (compute instances) ==> What "rules" would you implement between them? Like only Projects can define Backends?

Zig's concurrent/cached builds are sooo fast that I changed the default behavior of zig build to compile the extension for *all* supported DuckDB versions & platforms. Simplifies and speeds up CI. Also, the output dir (zig-out) is now a ready-to-use extension repository! github.com/mlafeldt/qua...

Multi-version builds by mlafeldt · Pull Request #1 · mlafeldt/quack-zig

Build and test the extension for all DuckDB versions by default, leveraging Zig's concurrent/cached builds. This simplifies CI and turns zig-out into a ready-to-use extension repository. Uncach...

github.com

Created SSH backend connector module. Stores ssh keys encrypted with password on the disk and implements additional security features. - AI helped to generated tests - AI helped to do security audits - 100% test coverage github.com/dforsber/ssh...

GitHub - dforsber/ssh-backend-connector: Establish and maintain SSH port forwarded secure tunnel to backends

Establish and maintain SSH port forwarded secure tunnel to backends - dforsber/ssh-backend-connector

github.com

💡 Pure DuckDB SQL example for reading Iceberg Tables github.com/dforsber/glu... #duckdb #iceberg

Add support for Iceberg Glue Tables · Issue #2 · dforsber/glue-table-cache

Apache Iceberg AWS Glue Catalog may have Iceberg Tables too. Since we already do and cache S3 listings, we can find out the Iceberg metadata.json files, but also .avro files listing the actual file...

github.com