What's the correct order of installing updates - first Windows, and
then SQL Server ? vice versa ? Or the order does not matter ?
There is unlikely to be any prerequisite for a Windows update that requires specific SQL Server updates to be applied because not all Windows Servers run SQL Server, however, it is possible that you may need a specific Windows update for a given SQL Server update (none spring to mind but it is conceivable). So given this, applying Windows Updates first is probably your best bet.
NOTE: Windows updates often set the restart pending flags and this can cause SQL Server update installers to fail the prerequisite checks. You need to orchestrate your patching process to handle restarts between applying patches if required.
How do I know if my Windows or SQL Server update failed? Is it
possible that any of updates can fail and bring server to an unusable
While it is possible for a SQL Server update to cause a fault that makes SQL Server unusable, it isn't too common. There are examples of it occurring though, check out Aaron Bertrand's blog article on updates for older SQL Server versions to support TLS 1.2 - SQL 2008 and 2008 R2 had a number of early issues causing service interruptions once the TLS 1.2 patch was applied.
For confirming the outcome of your SQL update installation, you need to review the setup log files which will provide a summary status that should confirm if the installation was successful. Generally, you want to check the summary.txt file and confirm the outcome is reported as "Passed" or "Passed, but requires reboot".
For Windows updates, you want to review the CBS.log file for Windows Update installations.
How to rollback (revert) Windows or SQL Server update if it was
unsuccessful (or even broke the system)?
Typically, updates can simply be uninstalled from the server to revert to the previous state. This can be done via Programs and Features (for SQL Updates, for SQL 2008+) and Windows Update Center (for Windows updates)
In the case of fatal errors preventing the server from booting, you can take a snapshot of the VM prior to applying updates that you can use for rollback.