wp_list_categories walker without links on categories that have subcategories (to make a nested dropdown menu)

I’d like to make a custom walker that works with wp_list_categories to generate a menu of nested uls using my category, sub-category and sub-sub-category(?) structure. Any category that has sub-categories can’t be a link so it can be clicked in order to reveal the sub-categories (or sub-sub-categories).

Here’s what I want (only items with a 🔗 should be links):

Tops
  Shirts
    T-shirts 🔗
    Longsleeves 🔗
  Jackets
    Hoodies 🔗
    Raincoats 🔗
Shoes
  Boots 🔗
  Formal 🔗
Trousers
  Shorts 🔗
  Longs 🔗
Bags 🔗

See also this CodePen demo with the hard-coded HTML I’d like to generate, ideally with a custom walker for wp_list_categories.

1. I need to show a list of categories and subcategories in a series of nested uls.
2. If a given category has subcategories, that category can’t display as a link (a div or similar instead).
3. If a given category does not have any subcategories, that category will display as a link.

My menu system is at most three levels deep, and categories without descendants could be anywhere. The reason I’d really like to use wp_list_categories is so my editors have the flexibility to reorder categories. If I hard-coded in each top level category they can’t reorder them freely using WP’s built-in category editor.

Goal: Right now, wp_list_categories does output my categories properly. I just need to figure out how to override the code that is making category titles with subcategories be links. I need them to just be plain text.

I’m trying to make a custom walker myself but can’t seem to figure out where it decides if a category is a sub (or a sub-sub) in order to omit links where they’re not wanted.