When you click near the middle of a text annotation, it will go into "edit" mode. If you click near the border (watch for a cursor change, you can select the annotation, and handles will show up for repositioning and resizing:
You can press 'Delete' when the annotation is selected to remove it.
When dealing with annotations and axes, it's helpful to know that an annotation will be anchored to the axes and will move automatically with axis changes if the annotation is created in a graph frame. If you move an annotation into or out of a frame, you can optionally reanchor it from the right-click menu:
This may help to avoid having to reposition annotations when the graph size or axes change.