Thank you for your feedback. In my experience, classic pre-processing steps are not needed when using BERT.

In fact, in classification tasks I often achieved lower performances by pre-processing textual input (removing stopwords, punctuation, stemming, ...) when using BERT-like models.

Of course, some sort of pre-processing is always needed, depending on the data source (e.g. remove urls, html tags, repeated headers/footers, ...).

But standard pre-processing techniques such as lemmization were designed for algorithms that did not take semantics into account. On the contrary, BERT also uses stopwords, punctuation, ... through the attention mechanism.

Nevertheless, you may also test the effect of pre-processing and verify the best approach for your specific use case.