What do you do when the party you DM/GM for is really, and I mean REALLY good at combat?

Tarrasque

Try out a monster that stops the characters like a Mummy for example. (That’s kinda low powered I know)

Optional: give them a (homebrew) negative if they kill a certain creature

When combat's done say "roll for perception" even if nothing's there. if someone gets a nat 20+, then mention signs of a high-rank monster nearby.

Why are they good at combat? As the DM figure out your players strengths. Both their characters as well as their own. What tactics, spells, items do they usually use, or what pattern? From there- create a challenge. Create scenarios using the terrain or location of the fight to either remove or reduce advantages they’d usually have. Use enemies that maybe alone wouldn’t be a challenge, but who when stacked up compliment each other so that their abilities make certain actions that would normally save the party harm one enemy type while creating vulnerability to to other enemies. Another way? Splitting them up can work too, especially if you make each member face a challenge that would be specifically difficult just for their character. There is also giving them a battle they can’t fight. If you give them an enemy they can’t simply kill- of which there are many ways to do:

Perhaps the person they fight is the very same person they need to save, or they require them to complete the quest. Wether it’s brain washing or some other means of control. Perhaps it’s one of their own: by giving the party reason (motivational or through use of magic or control) to find them selves on opposite sides and this having to fight each other or one of their members who has been “turned.” Perhaps their quest is the very thing that could save the other party member from that fate and bring them back- if they are alive of course. You can use politics, armies- one enemy may not be enough to scare your party, but if there is a political back drop in which killing that person, or even assuaging them directly could start a war or bring an entire army down on your players.... they may have to think of other ways.

Using terrain is simple. By denying them the high ground, using or limiting choke points so that either enemies can face them en masse, or players are restricted in movement, possibly even squeezed down to where they are single file and can’t reasily back each other up and must face enemies as they come- picture a hallway between two joined halves of a cave. Once inside, enemies can flank either side but only those party members on the front and back of the row can easily fight, and will bear the full assault. It’s too narrow to switch places as well, to really dodge or use tactics. In a fight underwater your magic user may be unable to cast the verbal components of spells. In a stalagmite/tite ridden cavern filled with some type of gas, many spells that invoke flame, or lightening could ignite the gas, as could certain weapons or sparks from steel and armor. Ranged attacks don’t have clear line of sight either.

We all hate those games... but the occasional hostage situation or escort mission- be it a person or a fragile and cumbersome artifact- can challenge a party who is specialized for all out combat as a unit and not used to having to protect another entity (just... if it’s a person they need to be relatively fragile, you don’t want to make your party MORE op- unless they will be facing godlike enemies, but don’t play the character like they are a total liability that is walking around trying to die either.) by using various enemy types for successive encounters, plenty of special resistances or immunities can be found. It’s easy to slay undead all day, to have specialized spells and equipment for things like holy or silver, so throw a few undead, then throw in some enemies with opposite or totally different traits on the next encounter. That way your crew can’t over specialize but also can’t just carry the biggest dps available because the next monster may need precision.

Lastly, here are a couple from DM’s I’ve loved:
1. Modify the rules. Print copies of appendices for the party to have available as suited- but make V.1.5,2.0 etc type monsters who have specialized traits. Don’t go ham- don’t create tailor made nightmares that only exist to cancel their powers: be smart. A skeleton enemy who logically suffers little to no damage from small head arrows or finer melee weapons- regular bludgeon weapons like a staff do slightly reduced damage (no flesh, must break bone,) and regular damage from large clubs, swords, axes, and special larger/heavier arrows which must be made/bought, take more weight to carry, cost more, and have a range/accuracy/speed penalty. Use monsters who can use unit tactics and plan battles. Stiff like that.

2. This is sort of a rare “treat” and a last resort. An adversary. An old DM used another friend. One who didn’t want a full and regular commitment, but wanted to play some. This friend and the DM worked together. They made him a character. A “mercenary” of sorts. It can be an agent of darkness, a hired sword by one of the parties current or past enemies with a grudge, a bounty hunter or vigilante (spending on party alignment), it could be a person in one of your characters back stories brought to life, or just a wandering sword who challenged anyone that they find who is said to be strong. A million reasons. But this character- in essence is built specifically to counter your parties main strengths, and exists to challenge the players. You may or may not give them command of a small force, army, or group of monsters. They are used for occasional challenges either at critical junctures or when things are going slow/smooth and you want the party to sweat.

Literally they metagame combat super hard, down to movement. I have modified rules, and they've complained when I've done so, and when I do throw monsters at them that don't conform or can exploited the usual way, they complain too. My main problem is that I've in essence, given them a finger, and they took the whole hand when it came down to it. I run a pathfinder campaign, and one player in particular is problem. and it's even worse since he's a friend of mine. He plays games like XCOM Caves of Qud a lot of number and stat based games that require you to meta game super hard to be any good at living for prolonged periods of time. So much so he sort of pressures the rest of the party to build or develop according to his character's needs or weaknesses, and as he levels up, he doesn't have those weaknesses anymore, so much so he rolls three attacks, one bite, and two swords, at +8/+8 with double damage woith a dumb spell combination. If I don't tweak normal monsters, he just kills it.

Like for example, I'm pretty sure he thinks I want to change how the crit system in Pathfinder works because I want to nerf his character. I've found it dumb that you can roll a crit on any number other than a Nat 20, so I tired to change it, but it was met with mostly just him complaining about it. I'm still working on house ruling it, And when a monster he came up against that he couldn't get flanking bonuses on, he acted like he had nothing to bring to the fight. And when they went into some sewers to chase this horrible monster hiding in the sewers, and it got the drop on them and dragged one of them into a deep pool of filthy water, his first line of action was to try to rules lawyer me into why is couldn't have dragged one of the party members into the water.

But thank you all for the advice, it is really, really, really helpful.

No worries. Now we have some more context. I’ve been in informal games like that, with people like you’re friend. Some of them were very fun, others not. These are games where the player(s) just want to be billy badass. They aren’t looking for real danger and get upset if anything bad happens to their characters, even if it’s minor. They’d rather argue over why they are somehow immune or what rules allow it than to undertake a side quest to fix it, or deal with playing at a handicap. Those games tend to be more about leveling, treasure, and interactions- being able to do whatever the player likes and being the sort of absolute power of the world. In my experience anyway. Your friend(s) might differ. Having been through that though, I can say this:

As the DM, you have a lot of responsibility. You administrate so others can enjoy the game, but being the DM can and should be enjoyable for you too. If you can enjoy that sort of campaign (we had a DM That loved and hated games like that,) then why not? Who cares? Just let them go about and have your fun through the RP of different NPC’s and through griefing with insane situations and silly pranks or treasures. But... maybe this isn’t the way you want to go. If you want to run a serious campaign, you’re putting in work to research, design, and set up. Sometimes your players may “beat” you or skip whole set ups you’ve concocted- but at the end of the day you are the DM. When you make a call they need to respect it. You are the rules, the hand of fate, if they roll a 20 it is in your power to call that a critical failure if you want to. It’s built in to the rules that a DM has absolute discretion, and can change or modify rules, encounters, monsters, anything.

So- a good DM won’t get a god complex, or start throwing around blind authority, disregarding rules or changing too much mid game- however as the one who runs the campaign you balance the books. Sometimes that’s compromise. Like changing certain rules on how things function, but in exchange allowing the realization of stats or perks a character wouldn’t have taken had the rule change been in effect. You can use in game means to do this too. An old DM of mine would do things like create 1 player side quests and make players go on “mini quests” for an item or NPC who could change the thing that needed changed. They might call in a new deity or other event to alter certain “naturally laws” or have an enemy create an item (say one which hindered communication with the ethereal and thus made it harder or impossible to cast certain spells etc.) the quest you just completed could in fact end up to be the trap. The thing you retrieved or saved was actually an item that ends up causing grief.

These scenarios- you can let the player lead continue normally for a while. Create your own mod table for events in question. Start randomly causing the failures when they act, increasing frequency until it’s constant. They’ll likely notice. Ask for a review. That’s when you tell them no. It’s not the rules needing review but part of the story. If they want to find out why they must play to figure it out. They can’t really fight that one. Then their quest becomes to figure it out, and when they find out- the quest is to do what they need to do (destroy, retrieve, blah blah) the thing that is nerfing them. You haven’t done anything. It’s all within the rules, and they are the ones with the power and burden to fix it back.