So there's this nice thing called Let's Encrypt, I'm sure you're already aware. If you want to get free certificate for your website in an automated way, it's definitely great. However in my case I would have to make sure to renew the certificate before it expires, which would defy the whole point of using static website on S3 - that is, unless I want to change something, I don't have to touch it. I could perhaps set up a daemon somewhere that would take care of it, but then again I would have to make sure that the service actually runs.
Fortunately, there is a way to get the same results with just AWS services.