Some details of the third twist can be found in section 6 of this thesis http://arXiv.org/pdf/hep-th/9907123v2. The BPS equations correspond to a non-abelian version of the monopole equations considered by Witten in http://arXiv.org/pdf/hep-th/9411102v1. Some aspects of this topological field theory were considered in http://arXiv.org/pdf/hep-th/9504010v1, generalising the analysis in http://arXiv.org/pdf/hep-th/9411102v1 to the non-abelian case.
In each of the three topological twists of $N=4$ supersymmetric Yang--Mills in 4d, the set of bosonic fields contains a gauge field and two real scalars (just as in the twist of $N=2$ supersymmetric Yang--Mills that gives Donaldson--Witten theory). In the respective twists, the remaining four bosonic degrees of freedom in the $N=4$ supermultiplet assemble into either (i) a scalar and a self-dual two-form, (ii) a one-form, (iii) two chiral spinors. (Of course, all fields are valued in the adjoint representation of the gauge group.) Twist (i) gives the Vafa--Witten theory of http://arXiv.org/pdf/hep-th/9408074. Twist (ii) is the one first noted by Yamron in Phys. Lett. B213 (1988) 325-330, considered by Marcus in http://arxiv.org/pdf/hep-th/9506002v1, and more recently by Kapustin and Witten in the context of geometric Langlands. Twist (iii) is the one mentioned in the paragraph above.
On a compact Kähler four-manifold $X$ with $b_2^+ (X) > 1$, I believe that the close analogy between twists (i), (iii) and Donaldson--Witten theory relies on a vanishing theorem similar to that used in section 3 of http://arxiv.org/pdf/hep-th/9411102 in the abelian case of twist (iii). The implication being that all solutions of the BPS equations resulting from twists (i) and (iii) correspond to instantons on $X$ (with the four twisted scalars equal to zero).
Twist (ii) is a bit more subtle in the sense that it actually gives rise to a family of topological field theories, with each member labelled by a point on ${\mathbb{CP}}^1$. This is so because, up to an irrelevant overall scale, one can define a topological BRST operator from any complex linear combination of the two scalar supercharges which survive this twist. To quote Witten; "there are no trivial equivalences among this family of topological field theories, only interesting equivalences that come from dualities". In certain special cases, solutions of the BPS equations can be thought of as flat complexified connections of the gauge bundle (e.g. as in http://arxiv.org/pdf/hep-th/9506002v1) rather than instantons.
This post has been migrated from (A51.SE)