That is part of the burdens placed upon them and the shackles set around their necks due to their standing at the "outward of the outward" of forms, and their separation from the esoteric inner powers. It is as I have said in verse:

The Hajj in the law of guidance is but a form

Expressing the existence of hidden meanings.

The path of guidance for the Knowers is his path,

And his companionship for the guided is the best companionship.

Its Safa is the heart’s purity (safaa) from its turbidity,

And its Marwa contains the perfection of manliness (muruwa).

Its Sacred Monument (Mash‘ar) is hidden from all but those who feel (sha‘ir)

What my intuition has proposed in the ecstasy of love.

Its Kaaba is a Mim; by the light of its whiteness

My insight prepared itself for the visions of beauty.