I remember recently reading, that at least in the case of things like rare books (which are considered extremely fragile) wearing gloves may actually cause more damage than handling them with well washed hands. Here is the article

Per the Library of Congress:

"Should I wear gloves when handling rare books? Before handling any collection item, thoroughly wash and dry hands. Contrary to widespread belief, gloves are not necessarily recommended to handle rare or valuable books. Gloves (nitrile or vinyl) are always recommended if there is reason to suspect a health hazard (e.g., mold, arsenic). Clean gloves (nitrile, vinyl, or lint-free cotton) are also recommended when handling photograph albums/photographs or books with metal or ivory parts. Aside from those specific situations, it is generally preferable to handle your books with clean hands, washed with soap and thoroughly dried, rather than with gloves. Why? See "Misperceptions About White Gloves ," pp. 4-16 from International Preservation News [PDF: 1.08 MB / 52 pp.]"