I only have handwritten notes, does the GDPR still apply to me?

Yes. Even not automated, processed data is subject to the GDPR. Once you have sorted the data in one way or another, they are subject to the GDPR.

That means that the data subjects have the right of access. The files have to be secured appropriately and the data that is no longer needed has to be disposed of.

Appropriately means here that e.g. personnel files with the religious affiliation or the union membership should be locked.