Can I offer free delivery for some items but not others?

Yes, you can set up a Shipping Class for any products you want to offer free delivery for.
WooCommerce Frequently Asked Questions