File this under the “Faceplant” section. Because, when I worked out what the F was going on, that’s exactly what I did. Head, meet desk.
I maintain a couple of fun domains that basically show a single static page (one, two), both hosted on AWS. At present, one has four files: the default HTML page, the CSS file for it, a logo image, and a favicon. The other also has a background image. Not even any JavaScript: shock, horror. Anyway, for a while now, I’ve been meaning to chuck away that CSS and start over because (essentially) both sites look pretty yucky when viewed in portrait mode on a phone. Since they’re really simple, there was no way I’d buy a theme for them, and so it was time to roll up my sleeves and devise some CSS. Except…
…when I brought up the second one, neither image was being loaded. Firefox’s developer tools console showed that both image GETs were returning error 403, “Access denied/forbidden”. What? It was certainly working before, although I probably hadn’t visited it for a couple of months, so what changed? The first site (same HTML, same logo image, slightly different CSS) worked just fine, no issues. I tried viewing the broken site locally from VS Code: no problem. So… something is going wrong with AWS? Time to go down the rabbit hole.
First stop: S3, the storage for these sites. Maybe the files had somehow had their access rights revoked. Nope, all looked good. The storage as a whole had public read rights and the files were the same. I tried reading them by using the long AWS URLs and it all worked, yet the URL via the domain didn’t. Hmm, not that then.
Over to Cloudfront. Maybe I’d changed something but not told Cloudfront to invalidate its cache. I did so, and … no change. Hmm.
Oh, I know, maybe I screwed up the “Make My Site Really Secure” code in Lambda@Edge. Perhaps the Content Security Policy code there that allows images from the site (but nowhere else) got messed up. Let’s go over there and check. I cross-checked the code against all the other static sites I run on AWS, and … no difference. I reset the trigger just to be safe and sure and nothing changed: the images still weren’t being loaded, still getting the Access Denied error.
At this point I even made sure that the image file names in the HTML/CSS were copied directly from those in S3, because, you know, S3 is case-sensitive for file and folder names, when…
…suddenly…
…I stopped. I quickly renamed the files to all lowercase, edited the HTML/CSS to do the same, and, BOOM, the page came up properly. I’d suddenly remembered that problem I’d had a couple of years ago with secondboyet.com: the original HTML pages had mixed-case names, but sometimes the links from (say) Google results were lowercase which would trigger a 404. At that time I’d just replicated the whole site with lower-case file names, so it didn’t matter if your link URL was the original mixed-case or lowercase.
However, a couple of months ago I tidied it all up. I wrote some Lambda@Edge code to convert all filenames in incoming requests to lower-case. Easy-peasy, halved the storage being used and things still just worked. But, it was a right pain in the neck: the app/data store I used to generate the site wanted to use mixed-case, and I didn’t want to go through the whole thing and convert every link and filename to lowercase. At which point, I realized that the 404 errors I’d been logging on the site were all to do with HTML page names, and not with image names, CSS file names and the like. So I changed the code to check the filename on an incoming request. If it was for a .html file, convert the name to lowercase, otherwise leave it alone. Bingo!
I checked the trigger for the code and there were two. One for secondboyet.com and one for whoisthisjulian.com. Big sigh. I removed the second trigger and my original site worked again. All I can imagine is that I wasn’t paying attention when I set up the trigger for the original code, and just selected the initially displayed Cloudfront codename. Which, because they’re sorted alphabetically by that codename (you don’t see the domain name behind it), meant whoisthisjulian.com. I can well imagine (although I don’t remember) adding the trigger, modifying the code for the HTML check, and then properly adding the trigger for the correct Cloudfront codename. Oooh, it works! And not realizing that I still had a trigger for the other code version to the other site.
So, as I said: the whole experience is to be filed under “Faceplant”. Needless to say, I’ve now tidied up my Lambda@Edge functions and triggers. Now for some new CSS.
Loading links to posts on similar topics...
1 Response
#1 Luiz Marques said...
10-Oct-20 12:05 PMSomething very similar happened to me yesterday. I just noticed one obscure demo on my site wasn't working. Took me about an hour to realize it hadn't worked since I moved this site from a Windows server to a Linux server, and the filename case was always wrong in a link, but the Windows server just didn't care.
Leave a response
Note: some MarkDown is allowed, but HTML is not. Expand to show what's available.
_emphasis_**strong**[text](url)`IEnumerable`* an item1. an item> Now is the time...Preview of response