Answer by Balaji Viswanathan:
Union Territories in India are those regions that are too small to be a state and too unique to be merged as a part of an another state.
Too far: Take the case of Andaman & Nicobar islands. It is too distant from any Indian state and it is not feasible to be managed by an another state. However, it is not big enough to be a state of its own. Thus, the central government manages that territory by providing a local representative who will represent the region's interests. Same with Lakshadweep islands.
Have a different colonial heritage: Pondicherry was a French colony, while Daman & Diu, Dadra & Nagar Haveli were all Portuguese colonies. For centuries, these were managed with a different language, different culture and different system. They didn't want to join the nearby states that were British colonies and with very different systems.
Special status: Chandigarh is a major city that was claimed by both Punjab and Haryana when these states were divided in 1966. Since, neither state wanted to give up, the central government made it an UT with a neutral ownership. In case of Delhi, it is a capital region and thus has a special status like Washington DC & many such capitals.
Union Territories to States:
Many of the present states were Union Territories once. Eventually, the local politicians convinced the Central government to upgrade their UT to a state with limited powers. Himachal Pradesh got the upgrade in 1971, Manipur, Meghalaya & Tripura in 1972, Goa, Mizoram & Arunachal in 1987.