I don't think this is a good idea - the chat is good enough for what the community needs, but if you find others feel free to create a private Discord space for WorldBuilding.SE and announce it in the Factory Floor, WorldBuilding.SE's main chat room, to see who else wants to join.
As you have already mentioned StackExchange provides a chat and each site has their own rooms. There is one main room and users with more than 100 reputation can create their own chat rooms.
These chat rooms are already integrated into the existing reputation and privilege system, allowing people with a network wide rep of 10k to handle flags that highlight rude/abusive behaviour and mods have their mod powers to kick/ban people if they need to. It works well with the features of StackExchange, such as bots that feed the important messages from Main, Meta or our Blog. There are room owners with special privileges that allow them to handle people that stray too far from Be nice., too.
Voice chat can be nice, but I am not sure if there is so much demand in voice chatting with other members of the community. Some users have a podcast where, but things like that are generally outside of the StackExchange network and not actively supported by SE. I don't see why SE should support a second chat or switch from their current system.
SE is not built around private messages. This has been asked and answered lots of time throughout the network. Have a look at Any way to send a personal message to another user?. From this answer:
The consistent response to this is that it's a bad idea, because:
It could hide information from the community: useful information transmitted privately is unavailable to other readers, subverting the core purpose of the site.
It could be used to harass other users ("Answer my question!", "Accept my answer!", "Yer momma [sic!] so fat she overflows the stack!", etc...)
That answer also explains that mods have the special power to be able to send private messages to users - but this is reserved for critical messages, not for day-to-day social interactions. SE is built around answers and questions and the chat is their way to compromise to give people a chance to socialise - and I don't think that this should change.
If you want to chat with people on Discord feel free to drop into the Factory Floor and ask if people want to join you. Just to mention a few projects outside of the normal network activities, we have the Blog and there is the Podcast. The Universe Factory is the chatroom that is mainly used for these things, but it could be a private Discord server if that's what you and others prefer.
There is also the story-go-round, a collaborative story whose main points can be read here. If you want to start a chat room for something like a game, similar to The Back Room: Live Tabletop games on RPG.SE or like the story-go-round room for collaborative story creation or for a story you are working on and some more elaborate brainstorming dedicated to the topic - by all means, create your own chat room and post the link to the Factory Floor so that people know your room exists. The chat can be used in a lot of different ways, all of which are integrated into how the whole network is supposed to work. It's a very versatile tool.