Currently there's an option "Remove all" if you do a long tap on a download. I think the option is difficult to find and it isn't clear enough (does it remove the history? or does it remove the files I downloaded?)
I am baffled that there is actually NO way to clear the list of downloads (without physically deleting the files): - Clear history from the menu has downloads, but DELETES them (I bet many people (including me) unknowingly deleted all their files by clearing history). - No visible option in the download manager. - In the download manager, when you long tap on an item, you have in the context menu "Delete all", which physically deletes the files also.