Mix butter and sugar with an electric mixer on medium-high for 2-3 minutes. Add the egg and mix on med-high for 2-3 minutes.
Stream in the buttermilk and oil on low speed. Increase the mixer to med-high for 5-6 minutes until the batter is completely homogeneous.
On low speed, add the bananas and mix for 45-60 seconds.
Still on low speed, add flour, baking powder, baking soda, and salt and mix for 45-60 seconds until the batter just comes together.
Spread the batter in a quarter sheet pan lined with parchment paper and bake for 25-30 minutes. The cake is done when it bounces back when you gently poke it. The center should no longer be jiggly. Cool on a wire rack.
Cream Cheese Frosting
Mix butter and cream cheese until completely mixed. Add vanilla and beat to incorporate. Slowly add powdered sugar.
Toss graham crumbs, milk powder, sugar and salt in a medium bowl to evenly distribute.
Whisk butter and heavy cream together. Add to the dry ingredients and toss to evenly distribute. It should turn into a bunch of small clusters. Store in an airtight container. It will keep for 1 week at room temperature or 1 month in the fridge or freezer.
Melt sugar over low heat until it turns an amber brown color. Pour over walnuts on a silpat lined cookie sheet and let cool. Once cool, break into pieces and pulse in a food processor.
Line a sheet pan with parchment or silpat. Using a 6-inch cake ring, cut 2 circles from the cooled cake. Clean the cake ring and place on sheet pan. Line the inside of the cake ring with a strip of acetate.
Use the cake scraps to fill in the cake ring to create the bottom layer of cake. Using a pastry brush, soak the cake layer with milk.
Divide frosting into thirds. Spread one third of the frosting onto the cake. Top with graham cracker crumbs and walnut brittle.
Place another cut round of cake on top. Repeat frosting, crust and brittle layer. Top with final round (use the best looking one) and spread remaining frosting on top.
Freeze for 12 hours. Remove from freezer 2-3 hours before you want to enjoy it.