Something is being gnawed on for sure! That aluminum that looks like paint in the oil is a sign of a failed converter, although given all you have described and how much metal is in the pan my bet is something not right around the low/reverse roller clutch or rear planet.
Plan on a new converter and oil cooler. Take every piston apart and put in new seals. Wash that case real well as the ground up aluminum powder gets into the pores of the case. You want to get it all out or else you will end up a sticking governor after a day or two of running.... Ask me how I know! Hot ATF cleans real well.
BTW did you measure the end play when you originally built the unit?