Watch out: Your problem arises because this second-quantized Hamiltonian contains terms that do not conserve the particle number (e.g. db in the "G" term). As a result, when "diagonalizing" such a Hamiltonian, you actually need a Bogoliubov transformation (instead of a simple unitary transformation) to obtain new normal modes. The requirement for this transformation, which mixes creation and annihilation operators, is that the new operators d′ and b′ fulfill the same bosonic commutation relations as the old ones (e.g. [d′,d′†]=1).
While it is formally possible to write everything in terms of a matrix M (as you have done), finding the Bogoliubov transformation is not equivalent to diagonalizing this matrix in the usual way. You need to fulfill the condition [Xj,X†k]=δjkσk, where σk=+1 for the k=1,2 and σk=−1 for k=3,4 in your example (the minus sign comes about because X†3=X1 and so on). Setting your Bogoliubov transformation to be Xj=SjnYn (summation over repeated indices), this implies SΣS†=Σ, where Σ would be the diagonal matrix with entries given by σk (1,1,−1,−1). This is different from the usual unitary transformation, and indeed is known as a symplectic transformation.