Do the "right" thing even though it will hurt someone I love?

To make a long story short, my husband & his grandpa have a virtually nonexistent relationship because my H feels like his Gpa hasn't made enough of an effort to show that he cares about him (H makes NO effort) & that his Gpa doesn't think he's "good enough". So, they rarely talk....even though we know from my H's dad that Gpa has been really really hurt by this. We now have kids, Gma's in a home almost dead, Gpa is getting frail (hardly the intimidating guy from my H's childhood.). We've fought about whether or not we should visit Gpa & let him meet the 2 of our 3 kids that he hasn't met yet - his great grandkids -one of whom is the first girl in 7 generations! We're going to my H's HS reunion close to where his Gpa lives. I told H I'm not dragging our 3 kids 7 hours to sit in a hotel room with a sitter we've never met to go to his reunion, if we're not going to visit grandpa too. I set up lunch w/ Gpa. H is furious & very hurt. Who's right? What now? Opinions please!