Though Washington never lived in the White House, he did manage the planning and construction of the House. Washing retired in 1797 from the Presidency, three years before it was livable.
In 1800, John Adams became the first President to live in the White House. He wrote to his wife Abigail that is now enscribed in the State Dining Room; "I pray Heaven to bestow the best of blessings on this house and all who shall hereafter inhabit it. May none but wise and honest men ever rule under this roof."
In 1801, Thomas Jefferson, the 3rd President of the United States, took over residence and did his own work on the mansion. In 1814, when President James Madison was gone, the British attacked the mansion and set it ablaze. But not before Dolly Madison pack up items to save including the famous George Washington portrait that hangs in the East Room.
Throughout it's life, the White House was called "The Executive Mansion" and sometime called "The Presidential Palace." by some including Mrs. Madison. But in the 1900s when Theodore Roosevelt took up residence, he officially named the mansion it's current title "The White House" and extended the mansion with the West Wing.
Though the oval office existed during the early days of the West Wing, it wasn't until Franklin D. Roosevelt was President that the oval office was located in it's present position.