I think an important facet that needs to be addressed with instances such as this:
Is what I am producing meant to be a business model?
If the answer to this (or something similar along this line of thinking) is yes, then an approach to marketing should be entertained. One way could be going in at it in the way that the product is worth the money since it provides such a service.
With keeping it open source, the marketing angle is to market it as something that the consumer (whichever audience this pertains to) gains for the amount it’s worth. Obviously, a certain apprehensiveness has to be understood by the maintainer as both sides of the argument would hold true. One side, the product is open source which can be used freely; whereas, on the other side, the product is worth it’s weight to the audience which takes advantage of it’s use.
Create the brand and market it with the trust built behind it. Linux has branched endlessly (seemingly) with products branched which are profitable (ex. RHEL). Take the product core and market it as the trusted branded product which has the track record to back it up. It can be “open source” by allowing the base to be open source, but the product which is marketed can be closed. Close it to paying customers which can trust the maintainers behind it to provide the timely updates, security, customer service, and reliablility.
Then it becomes closed source, right? That product of the branch does. And that’s the product which will get a return on investment whilst keeping it open source for those which want to maintain it outside of the branded product. Slap a name on it which differentiates it from the rest and market it as the product the paying audience can depend on.
This is obviously an oversimplification, but it made sense in my head!