The Gantt chart is simply a different view on the branches in your map. As a result, when you switch your map into Gantt mode all your branches are given dates and durations. You cannot remove the dates and durations from the branches in your map and yet have them appear as tasks in your Gantt chart.
Branches in your map and tasks in your Gantt chart are one and the same.
If you feel your map is becoming cluttered with icons telling you there are dates assigned to branches you can hide those. To do that:
Once you’ve done that you’ll notice the task icons will disappear from the branches in your map, but the properties themselves will be maintained in the background.