Tessera
Folgen

Henry Pinkard

@henrypinkard.bsky.social

Exploring frontiers of AI and its applications to science/engineering Formerly: PhD/postdoc in Berkeley AI Research Lab + UC Berkeley Computational Imaging Lab https://henrypinkard.github.io/

52 Posts1219 Follower2158 Folgt
Lucien Hinderling@lhinderling.bsky.social · 5mo ago

PAPER OUT ✨ How can we make smart microscopy more interoperable? What are the technical and cultural challenges? 30+ people from academia and industry propose a roadmap: doi.org/10.1515/mim-... Also a review of applications and repo of implementations. Join the discussion! smartmicroscopy.github.io

Screenshot of the abstract: Smart microscopy is transforming life sciences by automating experimental imaging workflows and enabling real-time adaptation based on feedback from images and other data streams. This shift increases throughput, improves reproducibility, and expands the functional capabilities of microscopes. However, the current landscape is highly fragmented. Academic researchers often develop custom solutions for specific scientific needs, while industry offerings are typically proprietary and tied to specific hardware. This diversity, while fostering innovation, also creates major challenges in interoperability, reproducibility, and standardization, which slows progress and adaption. This article presents a collaborative effort between academic and industry leaders to survey the current state of smart microscopy, highlight representative implementations, and identify common technical and organizational barriers. We propose a framework for greater interoperability based on shared standards, modular software design, and community-driven development. Our goal is to support collaboration across the field and lay the groundwork for a more connected, reusable, and accessible smart microscopy ecosystem. We conclude with a call to action for researchers, hardware developers, and institutions to join in building an open, interoperable foundation that will unlock the full potential of smart microscopy in life science research.Screenshot of Figure 5: Interoperable smart microscopy ecosystem. Concept of a modular architecture for smart microscopy, where standardized experiment descriptions (e.g. useq-schema) and open data formats (e.g. OME-Zarr) allow integration of diverse microscopes, analysis tools, and user interfaces. Core components such as segmentation [76], [77], tracking [79], [80], and experiment logic are decoupled from specific hardware, enabling reuse across platforms. The system supports multiple input modalities (code, GUI, or natural language) and can be extended with additional devices like fluidics or environmental control modules. This structure enables flexible, feedback-driven acquisition strategies and cross-platform reproducibility.Screenshot of figure 4: Strategies for hardware abstraction that allow smart microscopy workflows to run across different microscope systems, illustrated with example implementations collected on the SMWG website. (A) Software communication layers: Image analysis and experiment logic are implemented in a platform-independent manner, while platform-specific adaptors control acquisition through proprietary microscope software via macros, APIs, or other interfaces. Custom GUIs allow users to configure analysis, while the vendor-provided software manages hardware and acquisition settings. By developing additional adaptors, these workflows can be extended to support microscope systems from other vendors. Example implementation: AutoMicTools , Supplementary Information S3. (B) Device-level standardization (e.g. μManager-based workflows) bypasses proprietary GUIs and provides a unified API for direct device control across manufacturers. This API abstracts vendor-specific differences, enabling consistent control of a growing collection of supported hardware. Example implementation: UU_smart_microscopy , Supplementary Information S2. (C) Event-based standardization decouples experimental design from hardware by describing acquisition events (e.g. acquire frame at x, y, t with channel c) in a structured format (e.g. useq-schema). Control software interprets these definitions and translates them into device-specific commands, enabling the use of vendor-specific features and optimizations during execution. Example implementation: rtm-pymmcore, Supplementary Information S1.
eurobioimaging.bsky.social@eurobioimaging.bsky.social · 5mo ago

🚨Publication! New review on #SmartMicroscopy: current implementations and a roadmap for interoperability by @lhinderling.bsky.social & colleagues from #EuroBioImaging's #SmartMicroscopy Working Group! 🔗 www.eurobioimaging.eu/news/new-pub... 📸 Hinderling et al, 2026, (DOI: 10.1515/mim-2025-0029)

Nature Methods@natmethods.nature.com · 7mo ago

A Comment discusses how commands, data, and metadata currently discarded by scientific instruments could be used to train AI systems to learn to conduct experiments. @henrypinkard.bsky.social @nilsnorlin.bsky.social www.nature.com/articles/s41...

The missing data for intelligent scientific instruments - Nature Methods

Most scientific instruments currently discard rich streams of commands, data and metadata from which AI systems could learn to conduct experiments with expert-level decision-making and troubleshooting...

nature.com

Henry Pinkard@henrypinkard.bsky.social · 9mo ago

Imaging systems have traditionally been designed to produce pictures for human eyes. But increasingly, measurements get processed by AI. What if we designed them to maximize information rather than visual appeal? Could we see the universe, diagnose diseases, and capture photos better? #NeurIPS2025

Henry Pinkard@henrypinkard.bsky.social · last yr.

While AI excels at tasks requiring specialized expertise, it often struggles with simple problems humans solve effortlessly. Can OpenAI's new "reasoning models" change this? 🤖🧠 And with some many variants to choose from, how do you select the right model for the right task? 🧵

Example of the ARC-AGI benchmark