Preheat the oven to gas 4, 180°C, fan 160°C. Lightly butter and flour a 2 litre loaf tin and shake out any excess flour.
Using an electric hand whisk, beat the butter and sugar in a bowl for about 5 minutes until light and fluffy. Gradually add the eggs, beating all the time. Sift in the flour and bicarbonate of soda, then gradually add the milk, until smooth.
Fold the mashed bananas into the mixture with the chopped fudge (save a small handful for decorating the cake, if you like). Spoon the mixture into the prepared loaf tin and smooth the top with a spatula.
Bake for 1 hour or until a skewer inserted in the middle comes out clean. After 30 minutes of cooking, take the loaf out of the oven and cover with foil (to stop the top from getting too brown) then return to the oven for the remaining 30 minutes. Remove the loaf from the oven and leave to cool in the tin for 5-10 minutes, before turning out onto a wire rack to cool completely.
Once cool, serve plain or decorate with some whipped cream, extra sliced banas and toffee fudge pieces, if you like.