Hypoxia-inducible factor-2 (HIF-2) is a heterodimeric transcription factor formed through dimerization between an oxygen-sensitive HIF-2alpha subunit and its obligate partner subunit ARNT. Enhanced HIF-2 activity drives some cancers, whereas reduced activity causes anemia in chronic kidney disease. Therefore, modulation of HIF-2 activity via direct-binding ligands could provide many new therapeutic benefits. Here, we explored HIF-2alpha chemical ligands using combined crystallographic, biophysical, and cell-based functional studies. We found chemically unrelated antagonists to employ the same mechanism of action. Their binding displaced residue M252 from inside the HIF-2alpha PAS-B pocket toward the ARNT subunit to weaken heterodimerization. We also identified first-in-class HIF-2alpha agonists and found that they significantly displaced pocket residue Y281. Its dramatic side chain movement increases heterodimerization stability and transcriptional activity. Our findings show that despite binding to the same HIF-2alpha PAS-B pocket, ligands can manifest as inhibitors versus activators by mobilizing different pocket residues to allosterically alter HIF-2alpha-ARNT heterodimerization.