An outage (in more ways than one)

Written By: - Date published: 1:50 pm, July 11th, 2021 - 22 comments
Categories: admin, The Standard - Tags:

First off – my apologies for letting the site drop out over the past days. We had several outages, computer and human.

Sometime on Wednesday (I think), the site started to die after a operating system update and during the same day (I think),  a automatic wordpress plugin update.

The system update was that (somehow) the database server got rolled back to an older version. This was the cause of the increasing levels of sluggishness in the site. The PHP was literally timing out on the database accesses because the some of the old default settings weren’t even close to optimal at even our base levels of traffic. Logged errors provided no easy diagnosis because it didn’t indicate what subsystem was causing the timeouts.

At roughly the same time an unwanted automatic update of WordPress plugins happened – something that was meant to have been a manual process that I pretest in a staging test site. This had the effect of effectively disabling what little access was there to the back end administration of WordPress by blowing up the javascript processing.

That was the computer outage.

Unfortunately I’d also had a bug from Monday onwards. Some kind of virus (probably) that pushed my lymph nodes into painful action, put me to sleep a lot, dropped my appetite to nil, and by Wednesday were making me vomit any solid food and in the last few days has eased off into diarrhea. It also pushed me into a bird brain lack of focus.

That was the human outage(s).

It wasn’t covid-19 as I have had my third (maybe fourth) unfortunate experience in getting tested – it never gets any better. I’ve lost 3kgs in a week, and I’m still on water, barley sugars, soup and soft scrambled eggs until I stop puking up solid food.

This week hasn’t been good times for me  exactly in a good condition to try to solve this. However I had a  crack at last night and found the data problem. This morning I managed to fix the other issues by detaching backend plugin to the point I could reactivate  the functionality.

Let me know if there are any obvious problems. But for now I’m going to sip soup and have a bit of a lie down…. I’d suggest avoiding this human bug. It rather takes the joy out of life.

lprent: Opps – that got published as my testing alter ego AncientGeek, who as a mere whatever privilege level he is on, could change the author of the post

22 comments on “An outage (in more ways than one) ”

  • Comments are now closed

  1. lprent 1
    11 July 2021 at 2:34 pm

    If anyone is wondering (like I did) why the comments are bit sparse on the right hand side, it is because there is a 5 day cutout as well as a maximum number of comments.

    These were the comments that some detirmined people managed to push through despite the outrage.

    • weka 1.1
      11 July 2021 at 2:58 pm

      Good to see the Replies list is still there (logged in).

      • lprent 1.1.1
        11 July 2021 at 3:28 pm

        Replies just works off a number of replies that match the logged in or the current ’email’ secret in use on the cookies (the latter is there for those who aren’t logged in) – which are the ‘id’ of the user.

        The all comments gets expensive to pull up in database processing time. Limiting by time allowed the database to efficiently optimise the query. But comments change all of the time. So as an optimisation, as each comment is made it simply flags a change, that causes the page refresh of the author to trigger an update of the cached list from a database query. This means that everyone else who isn’t making comments doesn’t have to query for the list from the database.

        It is reasonably efficient and is one of the reason why leaving a comment takes longer than just switching a page. Everyone else just gets the most recent cached list of comments.

        I was willing to expend more database resources on replies because it only gets calculated when someone with an ‘id’ asks for a page and it isn’t cached. The first load, and whenever someone replies to them. When the person making the comment, the id of who they are replying to is looked up in the cache of replies. If it is there, then that cached list of replies to deleted. Causing the replies list for that id to be recalculated on their next page load.

        But being replied to is far rarer than having a comment made for everyone, so the server load isn’t irkcome.

        I tested those both logged off and logged on.

        Anyway nap time for the sysop.

  2. Ric 2
    11 July 2021 at 2:44 pm

    testing to correct a comment
    adding to comment
    If this posts all working on windows with firefox
    Thanks for your work keeping the site running

    • lprent 2.1
      11 July 2021 at 2:54 pm

      Yeah. I also changed the wysiwyg comment editor from CK editor to the more usual Javascript editor. It is the one we use in writing the posts.

    • lprent 2.2
      11 July 2021 at 2:59 pm

      I have tested with Linux Chrome/ Firefox, current Android Chrome / Firefox and whatever Samsung browser is. Windows 10 Chrome / Edge.

      Haven’t looked at the Mac side yet. I will do that on the remote website testing tools or plead with my partner to touch her toys (and that will be a difficult bearing in mind the state of my health).

  3. lprent 3
    11 July 2021 at 3:02 pm

    This editor seems to get around that issue of not being able to edit comments on Android inside the desktop (non mobile) theme.

  4. Stephen D 4
    11 July 2021 at 3:03 pm

    Thanks LPrent for all you do.

  5. The Al1en 5
    11 July 2021 at 3:33 pm

    Cancel reply doesn’t work – latest opera version, and it also makes the ‘leave a comment’ disappear from the bottom of the page, which takes a page refresh to reappear.

    • AncientGeek 5.1
      11 July 2021 at 3:38 pm

      Yep. Shows on Linux Chrome / Firefox as well. Probably everything.
      The javascript code appears to be doing a passive event handler that just ain’t working
      I’ll look at it when I wake up.

      Dang – AncientGeek – definitely a birdbrain today.

      • The Al1en 5.1.1
        11 July 2021 at 3:40 pm

        Though bizarrely, if you right click on ‘cancel reply’ and open in a new tab, that tab will be as it should.

        • AncientGeek 5.1.1.1
          11 July 2021 at 5:38 pm

          Yeah it breaks the javascript whilst setting up the action. The underlying link is still there.

          Can’t remember about Opera. But in Chrome (similar in Firefox) you can just right click the cancel, go Inspect and it will give you diag split screen defaulting to the object model at the object that will show the link at some expansion. If you go into the top menu item ‘console’ it will show you javascript errors after you open things up a bit. You can use the source link to see the problem code. Clicking Pretty Print will shift from minified code to something that is slightly more readable. That is the code that I have to track down and correct. If I had to guess, it is an old non-standard Internet Explorer action that hasn’t been isolated.

  6. McFlock 6
    11 July 2021 at 3:55 pm

    That bug sounds pretty nasty. They sure hit you like a brick sometimes – a particularly disgusting brick.

  7. Anker 7
    11 July 2021 at 4:26 pm

    What a week for you Iprent!

    so sorry you’ve been unwell. Take it easy and thanks for getting the site up and running again

  8. KJT 8
    11 July 2021 at 4:50 pm

    Appreciate the work you put in.

    I’m sure everyone else that reads the standard does also.

  9. Anne 9
    11 July 2021 at 5:06 pm

    Well lprent, there WAS a brief moment of excitement around 8pm last night. Muttonbird joined Adrian Thornton in the list of comments. Two of them instead of just one! Definitely a step in the right direction.laughing

  10. Jenny how to get there 10
    12 July 2021 at 5:56 am

    There is no doubt in my mind that you are running a valuable and important public service.
    I feel it would be churlish to simply go back into BAU commenting, and debating, and airing my views, that this site furnishes myself and others, without remarking on this remarkable act of public service that you provide.
    What more can I say, really.
    Other than saying, long may it last.

  11. Molly 11
    12 July 2021 at 6:52 am

    Appreciate your efforts in running and maintaining this site, and the stamina and doggedness in getting it back up when you were unwell.

    We could all have waited till you were back on form, but I was delighted to see The Standard back online again. Thanks.

  12. ianmac 12
    12 July 2021 at 11:37 am

    Missed the site. Hope your bug gives you a rest. Thanks so much for providing this site, even if the computer technology is way out of my understanding.

  13. Ad 13
    12 July 2021 at 2:10 pm

    Very sorry to hear how poorly you have been Lyn. Still, 3 kgs is 3 kgs.

    Thanks again for your tireless work supporting this venue.

  14. left for dead 14
    12 July 2021 at 5:20 pm

    I too wish you well and to your work keeping The Standard available.regards Alex

  15. Drowsy M. Kram 15
    13 July 2021 at 5:27 pm

    Thanks for all your good works Lyn, really appreciated – be well soon.

The server will be getting hardware changes this evening starting at 10pm NZDT.
The site will be off line for some hours.