QubesOS is a client OS (=desktop, phone, tablet, … currently desktop) that implements security through compartmentalization. It uses a hypervisor (Xen) to make that happen. The client must be secure, because if that is compromised there is no security, the client can see the keyboard and screen. Present client systems are really insecure. Attacks come through apps (e.g. browser), from malicious applications, from USB devices, through the networking stack. through filesystem metadata. Once attacked, the lack of GUI isolation makes it possible for the malware to see the sensitive information of another application that is secure. Note that these are the security challenges of desktop systems, which is in many ways different from the challenges on servers.
Just trying to find all the bugs is not going to work – there always will be bugs. A monolithic kernel is bad for security, because it all runs on the same TCB (Trusted Computing Base) so there is no isolation between a compromised Wifi stack and the rest of the kernel. Same for Xorg, network-manager, … . And making them run as non-root doesn’t really help, because it’s the user stuff that is important.
Qubes runs several parallel OSes on the same desktop using virtualization (Xen) to isolate domains, e.g. Secure, Home, Work, Random stuff. Why does virtualization help? Because it reduces the interfaces, which makes the attack surface a lot smaller. Still, because it is virtualized, it preserves compatibility with legacy apps and drivers. However, not just the VM – hypervisor is critical. The VMs still communicate with each other, e.g. because you do file sharing. This creates another leak path between the compartments. Essentially, virtualization doesn’t do much more than what the MMU allows you to do for inter-process isolation. BTW you also need an IOMMU so the driver domains can also be isolated. Bottom line, the inter-VM communication framework is also essential.
Of course there is a trade-off between security and usability. If the compartments are really isolated from each other, you can’t do much with the system. QubesOS tries to find a good balance.