The 35-year-old pop star was a long-time friend of the former 'Love Island' host - who committed suicide at her London home on February 15 - and Olly has admitted he's still struggling to come to terms with her passing.
After 'Loose Women' presenter Andrea McLean shared her condolences, a visibly emotional Olly said on the ITV show: "It's been tough, even you saying them words it doesn't really make sense to me yet - I'm still really coming to terms with it. It was a massive shock, I was on holiday at the time.
"The last couple of weeks has been ... it's so hard to even talk about it, to be honest."
Olly hasn't turned to counselling in order to cope with the tragedy.
But he has looked towards his close friends and family since Caroline's death was announced.
Olly confessed: "It's been tough. I never thought I'd never see her again. I love her to bits and I'm going to miss her."
In a separate interview with on Virgin Radio with Chris Evans, the pop star revealed he was on holiday in Bali with his girlfriend Amelia Tank when he received the heart-breaking news.
Olly - who presented 'The X Factor' alongside Caroline in 2015 - described the TV star as an "amazing person and great".
He recalled: "I got the phone call about 2:30 in the morning, so I was actually asleep at the time.
"And yeah, I just looked at my phone and just kind of in shock and to be honest, it has not really definitively sunk in properly.
"You know, she was an amazing person and great. Yeah, just amazing in every way. Just a shame that it happened."