Jiyong filled up his lonely days by spending more time with his loved one