Carmel Eve

@carmeleve.bsky.social

She/her. Software engineer @endjin.com, LinkedIn instructor and STEM ambassador! Here for tech, feminism, mental health & social change #WomenInTech

There's something wrong with the #pandas API on #Spark: 1. Things happen much incredibly slowly 2. Data isn't reduced as it should be I've been conducting an experiment to investigate these issues over the past few weeks! endjin.com/blog/2024/11...

There's something wrong with the Pandas API on Spark | endjin

Fix the following issues: Errors converting large datasets to pandas, pandas for Spark is very slow, and pandas for Spark column reduction doesn't reduce data.

endjin.com