You said the sins of those who fulfill their duty of Pilgrimage will be forgiven and they will be like born anew. What about the sins of those who cannot afford the Pilgrimage?
Firstly, not us but the following hadith tells this information:
“Whoever performs Hajj for pleasing Allah, and does not have sexual relations with his wife, and does not do any sins; he will return as if he were born anew.” (Bukhari, Hajj/Pilgrimage, 4)
In accordance with other hadiths it is understood that rightful dues are except these sins.
The people who cannot afford the Pilgrimage may still repent for all their sins, ask for forgiveness from Allah and become like those born anew. A person can even repent for the rightful dues after their recompense to the aggrieved party and may be forgiven by Allah. Thus, not only those sins will be forgiven, but also they will be transformed into good deeds.
The doors of repentance are open to everybody till death.