Disclaimer: I haven't studied in college philosophy, nor linguistics, nor literature. I've studied psychology and sociology. I'm familiar with how academic discussions over texts and long period of time work, but I'm not an expert.

Ontology is in philosophy the study of being, as in what makes that a thing is this thing and not another. And old ontological question is that of whether universals such as "red" exist in the world, or if they're just a name we put onto collections of properties of things (in this case, being red). Another might be if only mater exists (materialism), or if only ideas exist (idealism), or both (dualism), or something else (eg. neutral monism). Yet another type of question would be about how beings are made in the abstract: bundles of universals, bundles of universals attached to a bare particular, bundles of tropes, a core of "nuclear" tropes onto which further tropes can be attached, etc. Those types of questions are highly abstract, but they can have consequences in surprising domains, such as in expert systems.

Propositions about the world always assume an ontology. For example, if I say "God doesn't exist", then I assume that something called "God" can even be defined well enough so that it can be said not to exist, which can be debated. Or if I say "men oppress women", then I have an ontology that assumes "men" and "women", which is an ontology that can also be criticized, like Butler does in the first chapter of Gender Trouble. It's also connected to fundamental problems of epistemology, like Quine put forth in a famous article.

Now, about McHale. I'm really not knowledgeable about postmodern literature nor theories of it. The furthest I've gone is reading Pale Fire, and articles about unnatural narratives. I'm more interested in symbolism, surrealism, and I'm better read in russian formalism, structuralism, and semiotics, and all for my own writing project, not with an academic goal. You see the sisyphean grain of salt I'm pushing. Anyway, first let's see in McHale's Postmodernist Fiction how he defines the "dominant" (pp. 6-7):

Jurij Tynjanov probably deserves the credit for this concept, but it is best known to us through a lecture of Roman Jakobson’s, dating from 1935. I quote from the 1971 English translation:

The dominant may be defined as the focusing component of a work of art: it rules, determines, and transforms the remaining components. It is the dominant which guarantees the integrity of the structure...a poetic work [is] a structured system, a regularly ordered hierarchical set of artistic devices. Poetic evolution is a shift in this hierarchy... The image of... literary history substantially changes; it becomes incomparably richer and at the same time more monolithic, more synthetic and ordered, than were the membra disjecta of previous literary scholarship.

“Hierarchical”? “Monolithic”? To pre-empt the deconstruction that such deterministic and imperialistic language, with its overtones of power and coercion, seems to call for, let me try to salvage Jakobson’s dominant for my own uses by deconstructing it a bit myself. Or rather, let me observe that Jakobson has in effect already deconstructed it somewhat himself.

Despite his claim about the monolithic character of a literary history organized in terms of a series of dominants, Jakobson’s concept of the dominant is in fact plural. In this brief but typically multifaceted lecture, Jakobson applies his concept of the dominant not only to the structure of the individual literary text and the synchronic and diachronic organization of the literary system, but also to the analysis of the verse medium in general (where rhyme, meter, and intonation are dominant at different historical periods), of verbal art in general (where the aesthetic function is a transhistorical dominant), and of cultural history (painting is the dominant art-form of the Renaissance, music the dominant of the romantic period, and so on). Clearly, then, there are many dominants, and different dominants may be distinguished depending upon the level, scope, and focus of the analysis. Furthermore, one and the same text will, we can infer, yield different dominants depending upon what aspect of it we are analyzing: as an example of verse, it is dominated by one or other of the historical dominants of verse; as an example of verbal art, its aesthetic function is dominant; as a document of a particular moment in cultural history, it is dominated by its period’s dominant; as a unique text-structure, it possesses its own unique dominant; and so on. In short, different dominants emerge depending upon which questions we ask of the text, and the position from which we interrogate it.

Having defused somewhat the overly deterministic implications of Jakobson’s language, we can now see, I think, what kinds of advantages the concept of the dominant offers. Many of the most insightful and interesting treatments of postmodernist poetics have taken the form of more or less heterogeneous catalogues of features—the membra disjecta of literary scholarship, as Jakobson calls them. While such catalogues do often help us to begin ordering the protean variety of postmodernist phenomena, they also beg important questions, such as the question of why these particular features should cluster in this particular way—in other words, the question of what system might underlie the catalogue—and the question of how in the course of literary history one system has given way to another. These questions cannot be answered without the intervention of something like a concept of the dominant.

Now the bit about how postmodernist fiction is ontological (pp 11-12):

This brings me to a second general thesis, this time about postmodernist fiction: the dominant of postmodernist fiction is ontological. That is, postmodernist fiction deploys strategies which engage and foreground questions like the ones Dick Higgins calls “post-cognitive”: “Which world is this? What is to be done in it? Which of my selves is to do it?” Other typical postmodernist questions bear either on the ontology of the literary text itself or on the ontology of the world which it projects, for instance: What is a world?; What kinds of world are there, how are they constituted, and how do they differ?; What happens when different kinds of world are placed in confrontation, or when boundaries between worlds are violated?; What is the mode of existence of a text, and what is the mode of existence of the world (or worlds) it projects?; How is a projected world structured? And so on.

[...]

Furthermore, once we have identified the respective dominants of the modernist and postmodernist systems, we are in a good position to begin describing the dynamics of the change by which one system emerges from and supplants the other. There is a kind of inner logic or inner dynamics—or so the case of Absalom, Absalom! strongly suggests—governing the change of dominant from modernist to postmodernist fiction. Intractable epistemological uncertainty becomes at a certain point ontological plurality or instability: push epistemological questions far enough and they “tip over” into ontological questions. By the same token, push ontological questions far enough and they tip over into epistemological questions—the sequence is not linear and unidirectional, but bidirectional and reversible.

A philosopher might object that we cannot raise epistemological questions without immediately raising ontological questions, and vice versa, and of course he or she would be right. But even to formulate such an objection, the philosopher would have to mention one of these sets of questions before the other set—inevitably, since discourse, even a philosopher’s discourse, is linear and temporal, and one cannot say two things at the same time. Literary discourse, in effect, only specifies which set of questions ought to be asked first of a particular text, and delays the asking of the second set of questions, slowing down the process by which epistemological questions entail ontological questions and vice versa. This in a nutshell is the function of the dominant: it specifies the order in which different aspects are to be attended to, so that, although it would be perfectly possible to interrogate a postmodernist text about its epistemological implications, it is more urgent to interrogate it about its ontological implications. In postmodernist texts, in other words, epistemology is backgrounded, as the price for foregrounding ontology.