In a large bowl, whisk together the sweetened condensed milk, whole milk, and instant vanilla pudding mix, making sure to break up any lumps. Refrigerate the mixture for about 5 minutes until it sets.
Meanwhile, in another large bowl, use a handheld mixer on medium-high speed to beat the heavy cream and vanilla extract until stiff peaks form, about 2 to 3 minutes. Set aside half of the whipped cream for serving. Gently fold the remaining whipped cream into the pudding mixture.
Cover the bottom of a 3-quart trifle dish with a layer of vanilla wafers. Spread one-third of the pudding mixture over the wafers. Add another layer of wafers, then an even layer of banana slices. Continue layering pudding, wafers, and bananas until the dish is filled, finishing with a final layer of pudding. Reserve the remaining wafers for serving.
Refrigerate the assembled pudding for at least 3 hours or up to overnight.
Stir the granulated sugar into the reserved whipped cream. Spoon dollops of this sweetened whipped cream over the top of the pudding. Crumble the reserved vanilla wafers and sprinkle them evenly over the whipped cream.