Some history

[sampsyo]

Recently, @luharder emailed me out-of-band to ask about some of beets' history as a project. I thought this would be fun to write down, so here's a little writeup. Let me know if there's anything in this story that anybody is curious about—I'd be happy to expand!

Origin

I started working on beets in college (2008, shockingly). I'm not sure if this is obvious, but my main motivation was to build a better iTunes. I had used many great open-source music players, and while I liked actually playing music with them, none of them seemed quite right for the organization part. For that, I thought iTunes had gotten one core philosophy right. I'd summarize that as:

1. The database is the source of truth. Music files also have ID3 tags (or whatever), but those are secondary. We build the database from the tags, and it's possible to restore the tags from the database. But you search for music by querying the database, not by combing through the filesystem.
2. Generate filenames based on rules. As a kind of corollary, we want to have an algorithmic way to generate every single filename (and therefore the entire directory structure) from the information in the database. If you want your hierarchy to look like <artist>/<album>/<number> <track>.mp3, then all that information should come from the database. And if you change the metadata, the files should automatically move.

Of course, the idea was to build an iTunes for nerds. That meant exposing a command-line interface, of course, but it also meant extensibility and hackability. Regular expression queries are, perhaps, the quintessential example: I craved the ability to search for weird problems like trailing whitespace in tags or whatever, and regular expressions are the obvious nerd way to do that. I started to imagine a basic foundation on which we could build a cornucopia of different tools for managing music metadata—automating away the drudgery of manual editing in the iTunes GUI.

While I originally thought of the database as the "essence" of beets, I think maybe many people see its importer (the MusicBrainz auto-tagger) as its central feature. It's just as important to me, of course—the first thing I wanted to do with the database was use MB to get my tags right once and for all. I had tried Picard many times over the years, but I could never figure out exactly how it wanted me to use it. The business of "clustering" and explicit commands to "look up" were always really confusing to me. I thought I could make something more straightforward if I stuck with the command line—and I tried to build what I imagined Picard should do. This of course turned into a huge project that is just as complex as the core database abstractions.

A long-forgotten bit of history is that I originally named the project "Asshole." The idea was that the tool would be uncompromising to the point of cruelty about getting your metadata right. The CLI executable name was ahl. I thought better of this quickly and asked my friends what to call it; my friend Will suggested the punny name beets.

I love the name. The only downside is that people are occasionally confused by the similarity to Beats (i.e., by Dre). But the first Beats headphones hit the market on July 25, 2008, two months after I made the first commit to my Subversion repository. So we were here first!

Growth

It looks like I released the first version on June 17, 2010. I was "dogfooding" beets all along and liked it, and my college-age ego really liked the idea of getting other people to use it. This was before the "social coding" aspect of GitHub made publicizing open-source projects somewhat easier (we had Google Code at the time, and we liked it!). So I spent a lot of time in those days trying to get the word out. I posted to lots of music forums I was part of whenever a new version came out, and I religiously posted updates to Freshmeat (another casualty of time).

The forum threads and such were an avenue to start building a community. I would try to explain what the "big idea" was and provide tech support right there on the forum. In retrospect, I can't believe how much time I spent carefully walking users through their first time installing a Python package, for example, or SSHing into someone's machine to reproduce a tricky bug. I guess I had a lot of free time in college, and I really liked the feeling of the project's slowly growing popularity. I kept up this pace of personal tech support for a surprisingly long time before it became infeasible after the project got more popular.

A huge part of the project's growth came from supporting plugins. I added those in version 1.0b3. Plugins are not only useful for the obvious reasons—they factor off complexity from the core and let third parties build their own features without mucking around with beets itself—they also made it easier to onboard other developers. Contributors very often got started by writing a new plugin, or hacking on an existing plugin without understanding the whole infrastructure. This was an easy "onramp" that made it simpler to get acquainted with the rest of the codebase.

Sometime after the migration to GitHub from Google Code, I developed a de facto "policy" where I would add people to the project after they made 2--3 useful contributions. I think this gave people a sense of buy-in, and I have never had a single problem with someone doing something bad with that permission. So I continue to be very liberal in adding people to our GitHub org to expand the "hive mind."

Recently

My life changed a lot in 2016, when I started teaching at Cornell. The professor life can be all-consuming to the point of absurdity—I am permanently behind on 100 things! The result is that I had (and still have) a lot less free time than ever before, even in grad school. And I feel much less motivated to do more of two things that are parts of my day job: programming and community management.

The result is that beets is still ticking along, and I usually do something with it every day (responding to a bug report, reviewing a PR, etc.). But I cannot be the indefatigable undergrad I once was. While I think the project is overall pretty healthy, I think there are two big problems that are a direct result of my being overwhelmed:

1. We have a backlog of PR reviews. While the community is great at answering questions and even handling bug reports, code reviews tend to fall through the cracks. Even when a code contribution looks great, it is hard to find the time to look at it carefully to certify that it's up to our standards of quality and maintainability. Even a small PR requires a lot of brainpower for me to review, so I often let them go too long. I think this is a problem because getting prompt feedback on your contribution is critical for feeling included. And the more people feel included, the more they can step up and help maintain the project overall.
2. I don't release new versions often enough. I'm the only person who currently has all the context and the keys to do new releases, so that's a serious SPOF. Some time ago, we said we'd switch to time-based release—one every three months to hopefully ease the burden of deciding when to do a release. I have seriously blown that deadline with the most recent version. Cleaning up a pull request and doing a last certification is also something that requires a lot of focus from me, so I have to admit that it too slips through the cracks.

I tried to reckon with this leadership problem by trying to spread the responsibility a bit farther via the idea of a "core team". That has been only a middling success. Some really great community members stepped up to be part of this core team, but we have not been very organized. It clearly needs more hands-on leadership than I've been able to give it. So I am still interested in how to forge a model for organization that is more sustainable.

The main issue, I think, is that beets is in a strange "middle point" of open-source project popularity. Small hobby projects don't have a lot of pressure and are not hard to maintain. Huge projects are often developer-focused (think Node.js, for example!) and have a critical mass that allows for more serious organization. (And people are often paid to do this work!) Beets is somewhere in the middle: it's still quite popular, but no one's job depends on it. And following the rise of music streaming services like Spotify, you have to be a special kind of person to want to organize a hard drive full of MP3s.

With all that in mind, I still think beets has a bright feature. I would be really excited to explore a slate of big new ideas around streaming and discovery. RollingStar recently posted some ideas about making it easier to contribute to beets. All of these things are stuff I would love to spend all day working on---if only I could find the time and mental energy.

So on that exciting note, I'll close by saying I continue to seek advice about how to manage this kind of mid-sized open-source project. I'm sure people have great ideas and I'd love to help implement them.

[JOJ0]

Hi @sampsyo, dear project members,
I recently came across https://discourse.beets.io/t/forming-a-beets-core-team/639 again and want to suggest it to readers. Some things in there are still current. I wanted to answer to that very old post but now think this is the better place and since you ask for ideas here, I'll go for it.

Appologies beforehand for being direct and telling what I think. It might not always be superpolite. Also appologies for probably pointing out issues that you are already aware of, even stated above and working on solutions already. I just want to add my 2 cents from my point of view as a user and a contributor. Shortly myself: Using beets since Dec 2021, contributed some features and fixes already, got some PR's in the queue, been workin as a Linux Sysadmin/DevOps Engineer/Technical Consultant for years, not a programmer per se but quite a decent automator.

Responsibilities

The main thought that still comes up when contributing to beets, browsing around PR's and issues, reading forum posts: "It is very few project members answering things and reviewing PRs". Nameley it's @sampsyo and wisp3rwind. Occasionally other members pop up, no offence to all of you (Hi @jackwildson and @RollingStar for example :-) and I appreciate the efforts of each and everyone but just reporting what I see is happening (during the last year, and certainly also what I see in older threads). When we look at the members list of the beetbox organization we see that there are 14 members: https://github.com/orgs/beetbox/people

In this post and on the Discourse thread mentionend above @sampsyo you are talking about consistently adding frequent contributors to the gh-organization, creating a core team, delegating things and so on.

I'd like to share what I've learnend on my dayjob and from being part of two very tiny Open Source projects. One project consisting of only 2 persons, the other currently 4. Certainly I'm aware that by no means these little projects can be compared to a larger or mid-sized project like beets, still I often think about the current beets project situation and it reminds me on things we were struggling as well.

One thing I've learnend is to not only rely on the fact that members will grab "any" open PR and review it, but to split responsibilities. There is so much things around beets, there is the core importer, autotagger, the library, the querying code, and a lot of other very "corey" things but on the other hand there are quite some plugins that are shipped with beets that are rather "non-core" and from my point of view, don't touch much core things and can't ruin too much when not being reviewed by a "core specialist". Certainly and more often than not, the reviewer needs to be aware of dependencies to the core things.

What I'm trying to say: There could be members that are specialist in "core beets things" and there could be others being responsible to certain plugins. Certainly this comes with the existing knowledge and the interest in a topic of the member. As one of my former teamleads always said: "You are the specialists, my job is just to know for which topics you are." For a contributor to plugin X, it would be interesting to know which of the members is the best person to talk to and ask for a review of a PR. It could be stated in the contribution docs which member feels most responsible for a specific plugin and should be bothered. My personal experience is that at some point I felt the urge to ask for you two guys @sampsyo and @wisp3rwind and I was already feeling bad about it and thought: Why don't I know another person to ask, those guys can't do everything on their own, they are going to burn out at some point.

My experience with split-up-responsibilities like that is that members will automatically grab certain PRs and review them because they feel the responsibility, because it's their topic. If they feel they need a review from another member, let's say a "core specialist", they would ask for it specifically, summarizing what they need from the specialist. They would take the responsibility of "taking care of beets as a working whole" away from the contributor and at the same time releaving the "core specialist" from having to deal with basic contribution things in, for example, a tiny plugin's change.

A documentation expert could be asked by the main reviewer for a final wording check. Again moving away responsibility from themselves and the core specialist. Let the expert do it and call it a day...

Long story short: Not everone can do everything in a project and not everyone should do everything. There can be easy things that everyone should be aware of, yes certainly, but not everyone is a specialist for the whole project. A contributor should have the possibility to inform themselves about those responsibilities.

Time

Everybody is struggling with time. Maintaining Open Source is time consuming.

It must be ok that members are not there to take their responsibilites. It must be ok for contributors if they'd have to wait for weeks, even months.

It is appreciated and will be understood by contributors if at least they get a heads up. Even if it's something like: We are not interested in merging new features at the moment, just bug fixes right now because we are working on a release / because we just don't have enough members with spare time right now".

Draft PR's

Another thing that would help to see the trees in the forest again would be to change PR'
s state to "Draft". There is a lot of them named WIP or something or others where open bullet points state that it is just not finished and not ready for a final review.

To achieve that goal, contributors could kindly be asked to change their PR's to Draft if they think it is not ready for a final review yet. If there is no anwer within a month or a specific and told time in that question, maintainers would just change it back to Draft state. It doesn't hurt but helps finding out what's important to a member/maintainer.

If a contributor wants an early review on a Draft PR they should specifically ask for it.

Probably a lot of things I'm saying are nothing new to all of you dear project members, you've been probably discussing in your internal chats already and I'm already afraid of getting to be the person who tries to be smart and telling you what to do. Still I hope there is something usable in here and last but not least: I'd like to help.

[RollingStar]

Thanks for the shoutout. Adrian, my biggest request right now is that we shift to one forum instead of 2. The best thing you could do as product owner is lay this foundation for future growth.

Beets as a project has a few interrelated issues which are perfectly understandable with the context in this history thread.

1. Lack of figurehead and roadmap. I think if there was an Adrian clone who wasn't programming in his day job, real-Adrian would gladly hand over the project to him and step back a bit from leadership. It's hard to want to take a project into phase 2 when you got it through phase 1 and it's already meeting your needs well.
2. College CS students aren't as good at UX or usability as experts.
3. Beets is awesome about what it was designed to do in 2008, but the way people interact with music has radically changed since then. Processing power and especially ML/data science has exploded and enabled tons of possibilities. Musicbrainz is much more robust in number of data fields. High res covers are easy to find. Lyrics are still a pain but even they are somewhat obtainable.
4. No one in the beets inner circle is an expert in the peculiarities of healthily running a healthy open-source project. I'm not. I wonder if other projects could provide advice here. I also wonder if sponsorships or partnerships could be pursued. Here's what Musicbrainz has. I think Bandcamp would be a logical and potentially really good partnership. They care a lot about community relations and my hunch is that a $5,000-$25,000 donation could go a long way. But grant-writing is a job in itself, and nobody involved in Beets has expertise.

IMO, beets is okay for now. I see it continuing to provide the current functionality for the foreseeable future. But I believe we are nearly in a contributor death spiral where we are not bringing on new people fast enough to replace old contributors. The beets contribution funnel is brutal. Code is interdependent, there are a lot of unique terms (Library, Item, Pipeline), and it's non trivial to get any of your code to show up in Beets; even Hello World. Skimming this doc, there's an implicit reliance on OOP which makes perfect sense but is also the deep end for non-programmers. The way functions go from this() to Subcommand('this') to this.that is not clear in the way print("Hello World!") is. I'm not saying this complexity could be reduced, but it's something to know about and possibly document better. Maybe link to an OOP / OOP in Python primer.

I work in software and Python so it's not too tough for me, but back when I was just a bright eyed tech-adjacent noob, it was hopeless. I feel like other projects have a shorter path for hello world for tech-curious people. Here and there I'll read comments for other projects like "by day I'm a SQL Analyst or Excel Macro Wizard but I figured out how to do [basic thing], here's a PR".

What are the benefits of installing a non-editable Beets? I wonder if it makes sense to make editable the default/only way to get it (outside of distro maintainers ex. Debian).

My personal wishlist for expanding the beets appeal is taking the manual tagging and throwing it into the sun. The return on time investment is not there unless you are really particular about knowing whether you have the 1984 or 1992 master of The Wall. We should focus on matching release groups, not releases. The false matches happen more on popular tracks, which is what normal people are matching most of the time. Let's design for a future where we save all possible data about an imported track, and a user can fine-tune the match later. This goes hand in hand with ever-wilder ML. Beets is ripe for someone with <2 years ML experience to come in and blow our minds. As Adrian said above, the database is the source of truth. We can update it at any time. And a user should only do that when they have a use case that requires it.

Pretty much every popular streamer only has one or two releases of an album. They can change name or mastering silently in the backend and the user might not even notice. Clearly a large % of streaming users don't care about the release, only the release group (and arguably, they just care about the track, not even the release group). I'd call myself a music enthusiast and I only actively own, maintain, and care about having multiple masterings of a very small number of albums. These are my all-time favorite albums/artists, or ones with notably different mastering. Things like Thriller, The Beatles, and Floral Shoppe with its 3 releases to cover every track on it. Even if I was a "prosumer" level of music owner with 200 albums, it would come out to like 20=10% of my library. Yet Beets is implicitly designed around serving this 10% use case at the expense of confusion and complexity for the other 90%.

---

Changes we should make now:

• enable fromfilename and acoustID by default, and adjust those plugins as needed to accommodate.
• No files left behind. Beets should get everything on import, and pare it down later upon user request. Plugins for this, are any of them still working?
• No data left behind. Preserve the original filename and folder path on import. There's already dirfields which does this.
• cover autodownload should gracefully interact with existing cover.jpg files the user might have. Perhaps rename the original file to original-cover.jpg. I believe that right now fetchart is eager to overwrite local files with web sourced files.
• The manual importer prompts should be completely avoided without opt-in. Possibly need to make the re-import process more user friendly to compensate.

---

Some easy to implement use cases we could serve more aggressively, trying to bring in new and casual users:

1. Conversion for phones. I use foobar2000 currently and my genres are hot garbage. (My non-beets-ified tags are like "jazz", "smooth jazz", "island jazz", "jazz fusion"...). But my integration with Syncthing (open source Dropbox) is impressive. I can easily edit which files to include on my PC, and my Android phone wirelessly picks up the changes and adds/removes files to match. Calliope integration could have some fun use cases with minimal effort.
2. Conversion for one-off file sending (sometimes I send my friends music)
3. Conversion for youtube/video, possibly with integrated graphics. It's possible to do this with FFmpeg but a bit janky. Beets could tie together music, cover, lyrics (possibly exporting to SRT for youtube) in a reliable way and assure the user there is no / minimal quality loss. Have a few export targets like youtube, twitter, facebook. I could swear the YouTube "topic" videos used to look more like this, with the cover to the side and some basic info on the right. Either option would be swell, though spicing things up with metadata could be a use case that differentiates beets.
4. Download covers as a first-class, dead-simple, default(?) operation.
5. Download lyrics as a first-class, dead-simple, default(?) operation.
6. Basic data vis. I could take a crack at this in June-Dec 2023. Spotify Wrapped, /r/dataisbeautiful etc. show that basic viz helps people feel more connected to their library and certain products. A large chunk of the population thinks a bar or line graph is wizardry.

Some open source projects have good walkthroughs like for my Odroid HC4. I see there are a few videos on Beets and I wonder if there's a relationship to foster there.

---

The reason I write long posts like this rather than contribute to beets is my beets/music project is tied up in my NAS project. Which has itself taken a backseat to major life changes for me. I will return to beets however; there are too many use cases I have that aren't met yet.

[arsaboo]

I also feel that we could prioritize things.

Several PRs that don't affect the core and are limited to a specific plugin/feature are low-risk ones and could be merged faster. Worst case scenario, there may be a few bugs introduced (which can happen even after an exhaustive review). But I have noticed that those who are motivated to get a feature introduced are also motivated to fix the bugs. I see @JOJ0 being very familiar with several aspects of Beets and being involved in several related projects. I feel that s/he could have merge rights (if they are up for it) so that we can get a few more PRs out of our way. That would leave more time for other reviewers to focus on the remaining PRs.

[sampsyo]

I would love that. Let's get the venerable @JOJ0 on board as a reviewer & committer!!!

[JOJ0]

I'm very happy to have joined the project and want to thank @sampsyo here and also @arsaboo. Fun fact is that I thought about reaching out to @sampsyo and "applying for a maintainers job" ;-) for quite some time now already but was hesitating for a lot of reasons. Probably your mentioning here @arsaboo was the last kick in the butt I needed to finally do it haha ;-)))

[arsaboo]

Super excited about you joining the team. I will also try to review some PRs, but I am still learning the ropes here.

Now I know who to give a "kick in the butt" to get my Plex PR merged....haha 🤣

[JOJ0]

Regarding your idea @arsaboo

Several PRs that don't affect the core and are limited to a specific plugin/feature are low-risk ones and could be merged faster. Worst case scenario, there may be a few bugs introduced (which can happen even after an exhaustive review). But I have noticed that those who are motivated to get a feature introduced are also motivated to fix the bugs.

So basically what you are saying is that you would like to follow a "low hanging fruits" methodology. In my opinion not only "non-core-touching-things" could be prioritized but even smaller changes to core things. If they tidy up things, are small enough to be well understood for a reviewer and would help readability of the codebase or remove unnecessary things I would even consider them to be more "low hanging fruit" than a big new feature in a plugin.

And yes I agree that most of the time if new bugs would be thrown at contributors that introduced a new feature they would usually be happy to continue improvement on their precious work.

I'll try to do my best and start reviewing some "fruit" ;-)

[arsaboo]

Absolutely! In any case, this is a work in progress and the code is never "perfect". Most of the contributors are more than happy to fix the feature they contributed to. That is the beauty of beets....people (at least some of them) take their music very seriously and will be happy to spend time learning and contributing to beets.

So, non-core or minor changes to the core (like simple refactoring or adding an additional feature to the core that does not interfere with other functions) could be merged faster. The worst thing for a contributor is for the PR to be pending review/merge for days...let us try to avoid that if we can.

[RollingStar]

My roadmap ideas:

https://docs.google.com/spreadsheets/d/1L_n013wLNBpUGiTQQiuDnHSVN_ErX1Lw/edit?usp=sharing&ouid=103614650926151820429&rtpof=true&sd=true

[sampsyo]

Really nice work getting this organized!! Would it make sense to set something up in GitHub's new Roadmaps feature so we can iterate on the planning?

[RollingStar]

Sure. I don't think I have privileges to do that. We would need a "project" and then we can make a roadmap within it.
https://github.com/orgs/beetbox/projects?query=is%3Aopen

[RollingStar]

Do you have any links? I can't find Aaron Golub writing on metadata.

[wisp3rwind]

TBH, this looks like LLM-generated spam.