There is insulation and then there is insulation. What type did you use? Some stuff is several times better than the lesser stuff. The other big noise issue is air gaps. Is the top sealed down really well with no air gaps? Last one is your air intake. Where is that situated?

Dont know much about boats, but come to vintage/classic muscle car/ hot rods
 Seen a lot of stuff reaearched and done my own cars..
There are v2 types of noise supressant areas
1/ reverberation .. ie a side panel like a door panel/ side of the trunk(back lower 1/4 panel) under hood panels.
2/ Insulation.. ie fire wall , floors, between trunk and cabine, floors.
 Reverb: thats just adding a small panel which absorbs and stops the panel vibrating.. eg on a door panel 2 peices suck above and below the center approx 8 to 10" high x 20 to 30 " long in the middle of each section.

Insulation: that is full coverage

After several yrs looking at modern cars, old cars , before and after of many alternatives guys have used when rebuilding... and quite a few types of alternatives... I settled on the Foose stuff.
But to do the ave classic muscle car was going to cost around $2500...More looking around I found a silver coated self adhesive tar based stuff used by builders to seal window frames .. stuff like that. It is identical to the expensive Foose stuff, except required 2 layers and to do the whole car cost about $90.. Also comes in rolls different widths so install cutting etcv far simpler.
Product name  is thermo coat (???) get it miter 10.

Previous had the original heat sound insulation, still in new new condition..and replaced when car was stripped out before I  started to reassembled. The 1st think thwe missus noted how quite it was inside now ( keep in mind this is a late 60s American V8 muscle)  AND that the floor under the passengers feet remained cool. Previously used to get quite hot even with shoes on. Twin 2 1/2" exhaust pipes with the balance pipe under that area.

If I was going to do say the engine box in a boat, I would use the same stuff as it is robust, long life, insulated heat and sound, cheap, not thick, and if did not work out, fixing any other stuff would not mean having to strip it out.

 With more research, if had a alloy boat, I would seriously consider adding reverb panels to the inside of gunnels, hull cabin panels to knock back a lot of chine slap type sounds.