Each player gets to interact with the Nemesis and progress their own Nemesis encounter. 

For example with 4 players: 

Player 1 banishes theirs → nothing to fight, another player can interact with the pile of armor now 

Player 2 empowers → everyone fights its, now the original location where the pile of armor was is interactable for player 3/4 still. 

Player 3 empowers → everyone fights 

Player 4 also banishes The Nemesis encounter for that area is now ‘fully used’