You are wrong. VSS is not able to take application consistent backups. What VSS can do? Do you know what VSS does at all? It flushes disks, takes point-in-time snapshot with open-files read-write with VSS writer, quiescenes windows operating system databases, MS SQL and Exchange.
Yes, which is why you are wrong, and you've written it here. Quiescence tells applications to write in-memory changes down to disk prior to a snapshot taking place. VSS is a Microsoft technology that enables this on Windows, but there is a sync driver for Linux that does the same thing. It works the same way and clearly you need to go read about it.
I have instead some brand server software there. Its even Linux based embedded VM, all-in-one or on-the-box.
Well, that's a problem you must take up with the vendor of this software and what they recommend as far as backup procedure. Contact them and ask what their recommendations are for backing up. They may offer pre-freeze and post-thaw scripts used for backing up.
It even dont know nothing about VMWare tools - there is no vmware tools at all. Or the tools are outdated and there is no any possibility to insert them, because all stuff in on-the-box and any external action is strongly prohibited. But VSS it dont have anyway, because its not windows.
Again, that's a problem with whatever this software is and the vendor who provides it. You're conflating how quiescence works with not being able to quiesce on this application. Just because this application doesn't seem to have it out-of-the box doesn't mean it doesn't work for other applications.