Wop

Okay, I'll make sure to update line 389 to "model_info(name, revision=revision).sha". Noted.

I really like your perspective on ownership. It makes a lot of sense, and I agree with it. Looking at the Git history as the ownership ledger feels like a much more natural way for an open-source project to grow than assigning titles up front.

And no pressure to join—take your time and be comfortable with the decision.

As for the one problem I'd wish someone else already owned, I think it depends entirely on a few things: your skills, your interests, and what motivates you.

The people who've joined so far usually started by asking, "What if BenchLabs had this?" My answer has almost always been, "Go for it." They ended up naturally becoming the people who owned those areas because they genuinely wanted to build them. I mostly just helped them get started.

So to answer your question, I'd probably enjoy having both a generalist and someone who likes digging deeply into a specific problem. I also like the flexibility of being able to repurpose people as the project evolves—but never with pressure. If someone finds something they genuinely enjoy working on, I'd much rather they stick with that.

I'm actually curious: what kinds of problems or work would you enjoy owning?

That's a fair question.

To be transparent, we're still very early, so there aren't rigid day-to-day roles yet. People generally gravitate toward the areas they're most interested in and end up owning those over time as the project evolves, rather than us assigning fixed responsibilities from day one.

I'd rather not go into detail here about some of the things we're currently working on since a few are still unannounced, but if you join the Discord I'd be happy to give you a fuller picture there.

There's also no mandatory activity or minimum contribution expected if you join. We're an open-source project first, so people contribute when they have the time and interest.

On the ownership side, we're still figuring out what the long-term structure should look like ourselves. Everything we've released so far is under MIT or Apache-2.0, and we'd actually be interested in hearing your thoughts on how ownership and responsibilities should evolve as the project grows.

For the technical problem: happy to fix it tomorrow, or wait for your PR (let me know)

This may be out of context.. but here it is because I got curious:

Do you want to join us into BenchLabs?

there is an update, probably irrelevant to script.py- it now defaults to latest benchmarks only (default)

@dipankarsarkar right now its out

There will be huge update today-

--revision flag will be there.

https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard/discussions/2

I have found a problem where CoT models arent properly handled. I am patching both issues right now (since you haven't replied yet.

I'll let you do this one

Moving branch today --model takes a plain name/path, no --revision flag, so it resolves to whatever main is at run time.

I already went ahead and patched this: right after the model loads, it calls HfApi().model_info(name).sha to capture the actual resolved commit (falls back to config._commit_hash for edge cases, None for local checkpoints). So it's automatic, not something the user has to pass in — no extra flag to remember, no way to forget it.

One gap that leaves open: if the repo gets updated between the tool's model load and your re-run, "automatic" only pins to whatever main pointed to at run time, not a chosen commit. Possibly worth adding an optional --revision flag later so a maintainer can force a specific SHA for a guaranteed byte-equal re-run, rather than trusting that main didn't move. Not blocking for this PR though — happy to open that as a follow-up issue if you think it's worth doing.

Also added the sha256 field as you described — swapped the hardcoded v1.0 string for a runtime hash of the script's own bytes.

Tested it: editing the scoring function changes the hash, confirmed the leaderboard.json and results.json both carry model_revision + script_sha256 now instead of the old static fields.

Let me know if you found any further issue, if it my code has failure cases.
PR is at https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard/discussions/1 let me know if I did right.

Good catch there. Thanks for sharing.. If you think you can propose a PR that fixes the issue, it would be great.
I am not an expert, so I would deeply enjoy if we can have a discussion and solve this together
Eitherway, I've noted this. And I'm submitted to resolve it as soon as possible.