Trouble remembering childhood events can be a sign of trauma and abuse, sexual or non-sexual. Please remember that non-sexual abuse can be just as damaging as sexual abuse.
If you are in a position to I recommend seeking therapy. A qualified and empathetic therapist can help you worth through memories of trauma and figure out whether there is more that you don’t remember.
And just as a reminder, anon, yelling at you and punishing you for ‘being different’ is abuse all by itself, and you have every right to feel traumatized by it.