They will be ready for a reduction in heat by now, in fact as they're over three weeks old they should be nearly off heat, but the problem is that with your setup there's no way to gradually acclimatise them, it's all or nothing.
If they're well feathered and indoors in the brooder, in a room where the temperature is around 18C, you may have to persist in turning off the heat in the warm part if the day and just let them get used to it. An alternative might be to fit a less powerful bulb. If you're using a 60W, maybe a 40W? What is the actual range of brooder temp with the lamp either on or off? You definitely don't want to chill them, but are they gathering on the far side from the lamp when its on, suggesting they might be finding it a bit hot? If not, then probably they're OK. Chicks behaviour is the best guide to what to do.
Sorry I can't help you this time, as I've always used an overhead lamp which is more flexible as you can move it up or down to gradually change the temperature. How well feathered are they? How long have you left them with the lamp off? Do they settle back to feeding etc after a while huddled together?