I'd like to ask how to set up a fully-customizable MediaWiki-based website (not by Wikia, but one where I can install any extension I want) for free, but I'm afraid this would be like a website recommendation-type question as it would involve people recommending which website to do this on.
By "Wiki", do you mean the main Pro Webmasters site, or this Meta site?
Assuming that you meant the main site, then unfortunately I don't think this would be considered on-topic: it sounds like it would fall under the too broad category, and like you suspected, it would probably result in recommendations for sites and resources, which as you seem to already know, is off-topic here.
I'm not sure how many webmasters there are here using MediaWiki type websites, but you could ask your questions in our chat room. Also, if you're looking for recommendations for software, you can try asking this over at Software Recommendations, but it's wise to review their Help Center first to see what's considered on-topic for their site.
Thanks for asking though.