Tessera
Folgen

Takashi Ishida

@tksiia.bsky.social

🤖🗼 https://takashiishida.github.io

18 Posts29 Follower9 Folgt
Takashi Ishida@tksiia.bsky.social · 3mo ago

I'm in Seoul for ICML 2026!🇰🇷 If you’re interested in evals, coding and long-horizon agents, or issues such as contamination, reward hacking, and model cheating, please stop by our posters (details below) or say hi if you see me around, I'd love to chat!

skythanawat.bsky.social@skythanawat.bsky.social · 3mo ago

Coding agents are evaluated with unit tests: more tests passed = better model. But if tests or feedback are accessible, models may learn to game them. We introduce CapCode to detect suspiciously high scores, and CapReward to discourage them during RL. 🧵1/10

Bild
Takashi Ishida@tksiia.bsky.social · 5mo ago

I will be at ICLR 2026 this week! @iclr-conf.bsky.social I would love to chat with people with similar research interests. Please check out our posters at Session 6, Pavilion 3 & Pavilion 4, on Saturday, April 25. See you there!

Bild
Takashi Ishida@tksiia.bsky.social · 8mo ago

Happy to share that our papers were accepted to ICLR 2026!🇧🇷 Big thanks to my co-authors! Scalable oversight: arxiv.org/abs/2510.22500 EDINET-Bench: arxiv.org/abs/2506.08762 Optimal classification error estimation: arxiv.org/abs/2505.20761

Takashi Ishida@tksiia.bsky.social · 5mo ago

If you are at ICLR 2026 this week and are interested in model evaluation and financial benchmarks, please stop by our poster at Session 6, Pavilion 3, on Saturday, April 25! 👋🇧🇷

Sakana AI@sakanaai.bsky.social · 5mo ago

Sakana AIが昨年公開した日本語金融ベンチマーク「EDINET-Bench」が、国際会議 #ICLR2026 に採択されました。 ブログ： sakana.ai/edinet-bench/ EDINET-Benchは、金融庁EDINETの有価証券報告書約41,000件をもとに、会計不正検知・業績予想・業種予測の3タスクでLLMを評価するベンチマークです。 公開以降、日本の金融分野における複数の研究で引用されています。Sakana AIでは、今後も金融領域におけるAIの実装を進めつつ、その基盤となる研究開発を推進します。 論文： arxiv.org/abs/2506.08762

hardmaru@hardmaru.bsky.social · 5mo ago

I am very proud of our team for releasing EDINET-Bench, and it is fantastic to see a Japanese financial dataset recognized at #ICLR2026 this week. We need more diverse, non-English datasets to evaluate models in the real world. Paper: openreview.net/forum?id=Dxn...

back arrowGo to ICLR 2026 Conference homepage EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements Large Language Models (LLMs) have made remarkable progress, surpassing human performance on several benchmarks in domains such as mathematics and coding. A key driver of this progress has been the development of benchmark datasets. In contrast, the financial domain poses higher entry barriers due to its demand for specialized expertise, and benchmarks remain relatively scarce compared to those in mathematics or coding. We introduce EDINET-Bench, an open-source Japanese financial benchmark designed to evaluate LLMs on challenging tasks such as accounting fraud detection, earnings forecasting, and industry classification. EDINET-Bench is constructed from ten years of annual reports filed by Japanese companies. These tasks require models to process entire annual reports and integrate information across multiple tables and textual sections, demanding expert-level reasoning that is challenging even for human professionals. Our experiments show that even state-of-the-art LLMs struggle in this domain, performing only marginally better than logistic regression in binary classification tasks such as fraud detection and earnings forecasting. Our results show that simply providing reports to LLMs in a straightforward setting is not enough. This highlights the need for benchmark frameworks that better reflect the environments in which financial professionals operate, with richer scaffolding such as realistic simulations and task-specific reasoning support to enable more effective problem solving. We make our dataset and code publicly available to support future research.
Sakana AI@sakanaai.bsky.social · 5mo ago

Sakana AIが昨年公開した日本語金融ベンチマーク「EDINET-Bench」が、国際会議 #ICLR2026 に採択されました。 ブログ： sakana.ai/edinet-bench/ EDINET-Benchは、金融庁EDINETの有価証券報告書約41,000件をもとに、会計不正検知・業績予想・業種予測の3タスクでLLMを評価するベンチマークです。 公開以降、日本の金融分野における複数の研究で引用されています。Sakana AIでは、今後も金融領域におけるAIの実装を進めつつ、その基盤となる研究開発を推進します。 論文： arxiv.org/abs/2506.08762

Takashi Ishida@tksiia.bsky.social · 7mo ago

📣We are excited to launch the CapBencher toolkit today! CapBencher caps the best achievable accuracy on purpose. If an LLM scores above the cap, it’s a red flag for leakage, contamination, or leaderboard gaming🚩 If you are creating a new benchmark, you might find it useful👉

Takashi Ishida@tksiia.bsky.social · 8mo ago

Happy to share that our papers were accepted to ICLR 2026!🇧🇷 Big thanks to my co-authors! Scalable oversight: arxiv.org/abs/2510.22500 EDINET-Bench: arxiv.org/abs/2506.08762 Optimal classification error estimation: arxiv.org/abs/2505.20761

Scalable Oversight via Partitioned Human Supervision

As artificial intelligence (AI) systems approach and surpass expert human performance across a broad range of tasks, obtaining high-quality human supervision for evaluation and training becomes increa...

arxiv.org

Johannes Ackermann@johannesack.bsky.social · last yr.

Reward models do not have the capacity to fully capture human preferences. If they can't represent human preferences, how can we hope to use them to align a language model? In our #COLM2025 "Off-Policy Corrected Reward Modeling for RLHF", we investigate this issue 🧵

Bild
Takashi Ishida@tksiia.bsky.social · 12mo ago

Released bibfixer 🎉 A tiny AI tool that cleans & standardizes your BibTeX files using LLMs + web search. No more tedious edits like fixing capitalization (ai -> AI), swapping arXiv for the conference version, or expanding "and others" into full author lists. Let bibfixer do the grunt work for you!

GitHub - takashiishida/bibfixer: A Python tool that automatically cleans, completes, and standardizes BibTeX entries using LLMs and web search.

A Python tool that automatically cleans, completes, and standardizes BibTeX entries using LLMs and web search. - takashiishida/bibfixer

github.com

hardmaru@hardmaru.bsky.social · last yr.

EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements Paper: pub.sakana.ai/edinet-bench/ We just released a Japanese financial benchmark designed to evaluate the performance of AI Agents on challenging financial tasks like accounting fraud detection.

GitHub - SakanaAI/EDINET-Bench: Evaluating the performance of LLMs on Japanese challenging financial tasks.

Evaluating the performance of LLMs on Japanese challenging financial tasks. - SakanaAI/EDINET-Bench

github.com

Sakana AI@sakanaai.bsky.social · last yr.

日本語金融ベンチマーク「EDINET-Bench」を公開 ブログ: sakana.ai/edinet-bench/ 論文: pub.sakana.ai/edinet-bench/ 金融庁の電子開示システムであるEDINETの有価証券報告書を活用し、高度な金融タスクにてAIがどの程度対応できるかを測るための日本語金融ベンチマークを構築しました。 EDINET-Bench での評価の結果、現状のLLMを単純に適用するだけでは、会計不正検知等において実用的な性能は出ないという課題が確認された一方、入力情報を工夫することによる性能向上の可能性も示唆されました。

Takashi Ishida@tksiia.bsky.social · last yr.

Excited to announce EDINET-Bench, a financial LLM benchmark built from 40k annual reports in Japan! It features accounting fraud detection, earnings forecasting, industry classification, and includes our tool edinet2dataset as a foundation for designing new tasks. Hope researchers find it useful!

Sakana AI@sakanaai.bsky.social · last yr.

日本語金融ベンチマーク「EDINET-Bench」を公開 ブログ: sakana.ai/edinet-bench/ 論文: pub.sakana.ai/edinet-bench/ 金融庁の電子開示システムであるEDINETの有価証券報告書を活用し、高度な金融タスクにてAIがどの程度対応できるかを測るための日本語金融ベンチマークを構築しました。 EDINET-Bench での評価の結果、現状のLLMを単純に適用するだけでは、会計不正検知等において実用的な性能は出ないという課題が確認された一方、入力情報を工夫することによる性能向上の可能性も示唆されました。