Don't load all posts at once - paginate or use some sort of lazy loading/infinite loading (where more posts are loaded as you keep scrolling.)
Watch image sizes - I've seen countless sites upload multi megabyte images expecting WordPress or an image optimizer will fix the problem (hint: they won't). Resize images to the size needed before uploading (think KB vs MB) and even optimize those beforehand as well (here's a decent tool for doing that: https://tinyjpg.com)
Consider WebP over JPG for even smaller size images.
If you're hosting videos I'd advise hosting them offsite - use Youtube, Vimeo or a CDN.
Self-hosted videos will slaughter the IO on your site.
Consider using a CDN to take the weight off your site.
Consider caching (big subject, but hosted caching is better than plugin, plugin is better than nothing).
If your theme is hand-crafted, check your post loop isn't doing anything it doesn't need to be doing.
Optimize your database - do you have a ton of post revisions in your DB?
Yyou can use a plugin such as wp-sweep to check or get rid of them (backup backup backup first).
You can also limit post revisions in wp-config using: define('WP_POST_REVISIONS', x); where x is the # of revisions you want to keep.
Lastly, check your site for malware (e.g. Sucuri, Wordfence) if there's a worm in there it will drag your site down.