Pour the milk into a sauce pot and heat it over medium high heat until it just comes to a simmer. In the meantime, add the egg yolks, cornstarch, granulated sugar, and salt into a bowl. Whisk the ingredients together until it turns into a pale yellow color. Then mix in the ube halaya.
When the milk is simmering, remove it from the heat. Scoop a ladleful of the hot milk then slowly pour it into the egg mixture while whisking quickly to prevent the eggs from cooking. Continue ladling in the milk while whisking until half of the milk has been used.
Pour the mixture back into the pot with the remaining milk and whisk. Return the pot back to the stove and heat over medium heat. Whisk constantly until the cream starts bubbling and thickens into a thick cream. Remove the pot from heat then mix in the ube extract and vanilla extract. Whisk in the unsalted butter until fully incorporated.
If your pastry cream is lumpy, you can pass it through a fine mesh sieve. Otherwise, transfer the pastry cream into a bowl and cover with plastic wrap. Make sure the plastic wrap makes contact with the surface of the pastry cream to prevent a skin from forming. Place it in the fridge to cool completely.
When the pastry cream has cooled, use a spatula to mix the cream to loosen it. Set it aside. Add the heavy cream and powdered sugar into the bowl of a stand mixer. Use the whisk attachment and mix on high speed until you get soft peaks. Add about 2½ cups of the whipped cream into the ube pastry cream. Using a rubber spatula, gently fold the whipped cream into the ube pastry cream until combined.
Add a layer of the vanilla wafers into a 2½ - 3 quart baking dish. It should be just enough to cover the bottom of the dish in a single layer. Add on a layer of the sliced bananas. Then spread on a layer of the ube pastry cream so that that bananas are fully covered. Then top with a layer of whipped cream.
Repeat this process one more time to create another set of layers on top. Crush any remaining vanilla wafers into pieces and sprinkle them on top. For best consistency, let the banana pudding set over night. Enoy!