Besides Worldbuilding, I moderate two other sites, History of Science & Mathematics and Mythology. These are sites that get maybe one or two questions a day, at best, and these are the sites where we've had this sort of discussion before - how to grow the sites.
Worldbuilding is, by comparison, gigantic. We get 20 questions per day and almost 10,000 visitors daily. We have an enormous userbase that consistently contributes questions, answers, and comments, raises flags, votes by the truckload, and in general forms a very vibrant community - which is more important than anything else. This is why we graduated early this year. The Community team was confident that we have enough activity to be a self-sustaining site for a long time to come, and so we left beta and became a full-fledged Stack Exchange site. And, might I add, our activity has skyrocketed since then. So we're well above the threshold for a very healthy site.
I'm guessing your previous Stack Exchange experience is on the really big sites, judging from your profile - Stack Overflow, Mathematics, etc. I come from the opposite - the small sites, many of which are still in beta and may never "graduate". Some of these sites might get half a dozen questions a day, while others, as I've said, get only a couple. I guess my perspective, then, is different from most people here. Worldbuilding seems enormous to me (because it is!), and frankly, I'm really happy with how it's grown over the past two years.
Another thing to consider is that worldbuilding is a relatively niche topic (in comparison to certain other sites). Very few people worldbuild, in comparison with the millions (?) of programmers out there. I'm pretty certain we'll never match Mathematics, Physics, Super User, or Stack Overflow in traffic. There simply aren't enough people interested. But there are enough, and we're getting more every day.
I guess the point of this spiel is that we're a healthy community, in a number of ways, and compared to many sites, we're already quite large. I think focusing our efforts on growth isn't very important at this stage in the site's life. We always love to have more users, and more content, but we also need to think about a lot of other issues the community faces: question quality, fleshing out the scope, dealing with site policies, and a lot more. After a lot of work, we've grown a large community. But we still have other challenges ahead.
I'm not saying site growth is unimportant, and growing the site now requires different thinking from growing it when it was a wee beta site, in the early months. But we're at a good place already.
And if I can say one more thing, it's this: Thank you so much for asking this question. You've been here for two months, and I have to say, not many relatively new users care about how a site's doing. You're one of the exceptions, and I'm glad to see that this is your favorite site.
I, for the record, concur.