While we're on the topic, quick survey: Are there any services I should add to the model? Do you get much value from labels other than X, Bluesky and/or Truth Social?

XBlock Screenshot Labeller@xblock.aendra.dev · 3mo ago

Apologies, there was some downtime with the auto inference engine the last few days, just realised it this afternoon. Hoping to overhaul a bunch of the tech stack sometime in the next few weeks, including retraining the model. 🤞🏼

Apologies, there was some downtime with the auto inference engine the last few days, just realised it this afternoon. Hoping to overhaul a bunch of the tech stack sometime in the next few weeks, including retraining the model. 🤞🏼

Tried recreating the service record again, still feel like the service is missing a lot of screenshots. I think model retraining time draws nigh, RIP my free evenings

The AppView disconnected from the labeller so I had to recreate the service record, which means that anything labelled since that happened and now wasn't properly ingested. I think I've fixed it now though.

Please report posts that the labeller misses: "..." menu › Report post › Other › Xblock I'm retraining the model soon and reports are extremely helpful for improving accuracy!

There's a lot of screenshots happening today so apologies if processing is a bit slower than usual

We have a localised power outage at the moment so services are down for now

We're experiencing some inference downtime due to the relay switchover, hopefully back soon!

bryan newbold (⛱️ sabbatical mode)@bnewbold.net · 6mo ago

we are still investigating some issues with some jetstream servers. the us-west instances seem to be doing fine, but us-east instances are struggling

Status: Localised network outage, hopefully back soon

Back up

XBlock Screenshot Labeller@xblock.aendra.dev · 8mo ago

Autolabelling service outage while I figure out why Ozone keeps dropping 502s

Autolabelling service outage while I figure out why Ozone keeps dropping 502s

Okay I think we're back

Argh it's not PC maintenance and repair without at least 2 aggravating moments

Service update: Inference is going down while we continue to upgrade our systems

Service update: system upgrade complete (tho I still wanna upgrade CUDA to 13 at some point)

XBlock Screenshot Labeller@xblock.aendra.dev · 8mo ago

New GPU has been installed and we're just updating drivers now. Hold onta yer butts

New GPU has been installed and we're just updating drivers now. Hold onta yer butts

XBlock Screenshot Labeller@xblock.aendra.dev · 8mo ago

Service update: scheduled downtime for system upgrade about to commence, will update here once complete.

Service update: scheduled downtime for system upgrade about to commence, will update here once complete.

Pretty relieved to realise my AWS credit doesn't expire until September next year (and that I've only used £2k of the £5k from the ATProto grant), that gives me a good few months to get XBlock onto stable local infra. I think if I can get Redis and Ozone back locally it'll be sustainable.

Finally upgrading some infra 🎉

ændra.@aendra.com · 9mo ago

I did it. I finally bought a new graphics card.

Service update: more network fun.

Continuing to have intermittent connectivity issues with our London datacentre (Aendra's flat)

XBlock Screenshot Labeller@xblock.aendra.dev · 9mo ago

Service update: we had a localised network outage earlier so it might take awhile for the inference server to catch up with backlog

Service update: we had a localised network outage earlier so it might take awhile for the inference server to catch up with backlog

Service update: I've figured out why Ozone isn't accepting appeals, apparently there was a lexicon change or something which meant we had to explicitly configure it to accept appeals and not just reports. Appealing a label should work now, sorry that took so long to fix!

There is an issue with appeals not submitting properly, I'm investigating that today. I hoped it would be as simple as restarting Ozone but alas nothing in life is that easy.

Open to feedback: our moderation efforts have really slowed down so it's taking longer than it should to action appeals (apologies!). I'm thinking, should the appeals process be semi-automated? I've refrained because it's somewhat open to abuse but I suspect it would help more than hinder.

The Ozone instance appears to be struggling, please bear with me as I figure out what's up

📌 Info for devs: Repo: github.com/aendra-rinin... Current model (xblock-m, TIMM version): huggingface.co/howdyaendra/... Please feel free to inspect, interrogate, adapt and/or reuse XBlock code/models for your own projects, we welcome and encourage it!

GitHub - aendra-rininsland/xblock-m

Contribute to aendra-rininsland/xblock-m development by creating an account on GitHub.

github.com

The queue was indeed the issue, there's a 4M post backlog (likely from that week it was down) and newer posts were being put at the very end so by the time a post is processed it's super old and everyone's seen it already. 🙈 I've switched it to process newer posts first so hopefully that'll help! 🤞🏼

XBlock Screenshot Labeller@xblock.aendra.dev · last yr.

Restarted the auto-labelling process and am running into issues with the Redis queue, it seems stable now but will investigate further later.

Restarted the auto-labelling process and am running into issues with the Redis queue, it seems stable now but will investigate further later.

I think there's a connectivity issue with the Ozone instance at the moment and labels aren't correctly propagating, or the model's needing retraining, or something. Please bear with me as I troubleshoot this. 🙏🏼