Contract

(default keyboard shortcut {)

The Contract command (Selection > Modify Selection > Contract) is the exact opposite of the Expand command, except that any elements that are adjacent to unselected elements are deselected.