You definitely have something wearing out. The only thing that really moves in the disc braking system is the piston inside the caliper. The piston side applies the pressure to the brake pad pushing it into disc causing a slight movement (sideways only) of the caliper which causes the other brake pad to contact the rotor. Maybe I'm not understanding what you are considering the caliper or where the grooves are getting rubbed into it. I can only imagine that the rotor contacting the caliper would result in a lot of noise.
Air in the brakes would cause loss of pedal and braking ability. The brake pads are almost running against the rotor constantly, the piston in the caliber will be pushed back into the caliper by the pads, as the pads wear the piston will stay out further and further until the pads are completely worn. This is why the brake fluid level will drop in the master cylinder.
Given the last statement of the bracket assembly also rubbing, I would tend to lean towards a bearing issue allowing excessive movement of the hub assembly, but that would have to one really worn out bearing.
When everything is installed correctly, the pads should sit approximately centered in the braking surface (machined part) of the rotor. There will be a thin line near the outer edge where the pads do not contact and there will be an area of the rotor near the hub that the pads also will not contact. There should be nothing loose enough to be moved by hand. What I would consider to be the top of the caliper spans over the rotor and has a pretty good sized hole to inspect the brakes through, that part of the rotor should be a pretty good distance from the rotor itself. The hole is big enough to see the thickness of both brake pads.
If you can, try to post a picture of what you are seeing.