The simplest and most direct way to permanently delete files or folders is by using the Shift + Delete keyboard shortcut.

You can check out the list of items being removed below, with Microsoft noting that they aren't disappearing entirely - though they'll only be accessible via modifying registry settings. This means ...