Version 0.19.X Deployment

Hello world!

I’m sure many of you fine folks have been wondering why we have not upgraded to version 0.19.X yet.

The whole team here has been getting asked this question quite a bit, from both members of the community and other instance admins. We want everyone to know, YES, we will upgrade to version 0.19.

We do not have a hard date set for when we will be upgrading as of yet, as we have about 7 associated task items that go along with this upgrade for things we need to double-check and confirm there will be no breakage.

Another big reason why we have not upgraded is due to reports from other instance admins that the front-end GUI can have performance issues. While a large portion of folks access our site using apps on their phones, we would rather not degrade the user experience for the rest of everyone else.

Worst case in terms of UI slowness, we would ask everyone to use either Alexandrite () or Photon () to mitigate the potential performance issues.

One final point to mention to address the delay is that we usually wait 1-2 weeks after a new release before applying it to production. This was complicated by the fact that there were multiple releases right after each other, with each new release having issues we hoped would be patched in the NEXT version. A new version would come up, and we would have to evaluate the stability and performance of it, which would take about 2 weeks, and then ANOTHER version would come out, which we would need to test. This was further complicated by the fact this was during the holiday season.

Version Release Date
0.19.0 2023-12-15
0.19.1 2023-12-20
0.19.2 2024-01-10
0.19.3 2024-01-24

For us, stability is paramount, as we care very much about keeping everyone happy here.

While we can appreciate the rapid release of patches on 0.19, I hope everyone can now understand the delay. If all goes well, we will set a release day in the next week or two, most likely on a Saturday around 1800 UTC.

-jelloeater85

  • QaspR
    00·
    9 months ago

    I for one am thankful for the admins’ continued dedication to platform stability.

  • jivandabeast
    00·
    9 months ago

    Hey just FYI, I’ve been experiencing a postgres memory leak issue on my instance since the upgrade to 0.19.2. Theres a GitHub issue open about it (#4406), not sure if it’s something that will affect you but probably worth keeping in mind

    • Rooki
      00·
      9 months ago

      Hi,

      thanks for the tip, we will look into this issue and check if it affects anything on our scale.

      • jivandabeast
        00·
        9 months ago

        Yeah no problem, checked the issue – looks like there’s some movement on it so we’ll see

    • Rooki
      00·
      9 months ago9 months ago

      We are investigating another possible bug, that sorta spams other instances full of requests ( we say 5k requests per 5 seconds ) kinda like a DOS attack. We only saw this phenomena on 0.19.3 instances, and they are hitting us currently hard with many requests. We will have to block them until it gets fixed on their side.

      • jivandabeast
        00·
        9 months ago

        Hmmmm interesting, i hope im not on the ban list! I haven’t noticed any weird traffic like that

        • Rooki
          00·
          9 months ago

          Nope you are not ;)

          We noticed it that it happens on smaller instances, that probably doesnt have the perfect hardware to run the 0.19 version and crashes => retries federation and they probably dont have a backoff strategy.

          Because we noticed the site the domain was leading too wasnt responsive half of the time we tried to look if we can contact the admin.

          • jivandabeast
            00·
            9 months ago

            LETSGO

            Yeah my instance is small but my uptime is pretty good (except this db issue tbh) so that might be why

  • tal
    00·
    9 months ago9 months ago

    Speaking as someone who is on an instance (lemmy.today) that ran into a bunch of breakage from the 0.19.X releases, and which still isn’t fully resolved, and where the instance admin said that he wished that he could downgrade to 0.18.X but couldn’t due to schema changes, I strongly endorse a conservative approach. The releases have not really met the bar that one might want for stability.

    That’s especially true for lemmy.world, since it hosts a large chunk of the Fediverse communities, and if it has serious problems, there are gonna be spillover effects even on users elsewhere. I’d wait until less-critical instances have been the guinea pig for a bit on releases.

    • ItsMeSpez
      00·
      9 months ago

      Well said. I will happily forgo some new features for a while if it means that the overall experience is more consistent.

    • maness300
      00·
      9 months ago

      Seriously. Less is more.

      There’s no need to keep up with the meme of releasing updates as quickly as possible.

  • AnAustralianPhotographerEnglish
    00·
    9 months ago

    Thanks for the update. I appreciate the transparency and honesty.

    No pressure from me, I know all too well about upgrading to the newest version and finding out it has a showstopper bug for the features i was planning to use.

    Thankyou for running the server and helping build the community and fediverse.

  • urdaEnglish
    00·
    8 months ago

    Lebowski.Social has also held back with the 0.19 upgrade. We are monitoring lemmy.world for now and will make our upgrade after that most likely.

  • m3t00🌎English
    00·
    8 months ago

    no rush. most don’t notice minor changes. thanks

  • PutangInaMo
    00·
    9 months ago

    0.19 breaks jerboa and idk if/ when that is getting updated

    • steventhedev
      00·
      9 months ago

      @dessalines@lemmy.ml released v0.0.51 in mid December, but I think it only reached Google Play on Dec 18th. I’m not sure when/if it was pushed up to f-Droid.

      There are some minor issues, but largely it works fine for me on instances running 0.19

      • PutangInaMo
        00·
        9 months ago

        That’s weird I’ve gotten no notification there was an update out

        • steventhedev
          00·
          9 months ago

          Did you install from Google Play?

          Open the Play Store link on your phone - the automatic update process has been thoroughly broken for at least a year.

          If you installed from f-Droid I have no clue - I use stock android without any alternate stores set up.

  • Riddick3001
    00·
    9 months ago9 months ago

    So ,if everything goes well the first update will be Saturday 10 februari around 18.00 UTC, and then every two weeks? Bonne Chance!

  • OsrsNeedsF2P
    00·
    9 months ago

    We do not have a hard date set for when we will be upgrading as of yet, as we have about 7 associated task items that go along with this upgrade for things we need to double-check and confirm there will be no breakage.

    Are there tickets open for these tasks so we can see what you’re waiting for? Genuinely wondering

    • Rooki
      00·
      9 months ago

      We dont have a public Roadmap yet, but we will have to test it in multiple steps. And we dont want to get suprised with bugs ( like those in 0.19.0/1/2 )

  • Krafting
    00·
    9 months ago

    Thanks for all this info, I knew running the largest Lemmy instance would be challenging in term of version upgrade, and I’m happy you keep us posted on all your plans! Keep up the good work!

    • BOMBSEnglish
      00·
      9 months ago9 months ago

      Yeah, I think the transparency and inclusivity of this instance serves as a model. I love knowing that places like this can exist.

        • BOMBSEnglish
          00·
          9 months ago

          You’re welcome, and thank you too! 😊

          • Rooki
            00·
            9 months ago

            No, YOU are welcome here ;) You guys make this instance and the fediverse the best ;D

  • sunaurus
    00·
    9 months ago

    Good luck with the update! One great thing about 0.19 is that it allows users to check federation status between instances, will be awesome to get that for lemmy.world as well.

    • maegul (he/they)English
      00·
      9 months ago

      allows users to check federation status between instances

      Wait … what’s this?! Never heard of this as a user feature? Any guidance?

    • Dave
      00·
      9 months ago

      And one thing bad is that logged in users are logged out and can’t log back in, getting errors when loggin in or worse, the site showing them a server error and they think the site is down! All fixed by clearing cookies. Or in apps, logging out then back in.

      It looks like 0.19.3 may have a fix for this, which is great. IMO it’s good that the largest lemmy instance hasn’t updated to 0.19 yet, because a bunch of users may have disappeared thinking the site was dead. And you’d never know, because 0.19 counts active users in a different way.

      Hopefully this is the one, though!

    • mesamune
      00·
      9 months ago

      It would be nice to set that at the settings level and have it auto populated instead of asking each time on a post.

      • TCB13English
        00·
        9 months ago9 months ago

        That’s what I’ve been saying since there’s lemmy…

  • aksdb
    00·
    9 months ago

    I for one am glad you take your time. I jumped ship from another instance that upgraded to 0.19 back in december and was almost unusable far into january. It’s now on 0.19.3 and while federation seems to work again, it’s still quite slow and unstable.

  • avguser
    00·
    8 months ago

    If all goes well, we will set a release day in the next week or two, most likely on a Saturday around 1800 UTC.

    Any chance we get can get a status update?

    • MWatson17English
      00·
      8 months ago

      We’re going on four weeks from the initial statement. The lack of status updates is what bothers me the most. I understand delays, but please let us know. It just looks like it’s been forgotten about.

      • avguser
        00·
        8 months ago

        @Thekingoflorda@lemmy.world, we understand this is a volunteer effort, but as the community lead, would you be able to shed some light into the status of the upgrade? I don’t think we are asking or pressuring for the upgrade to happen ASAP, just some transparency would be nice. We’re two weeks past the proposed update, and almost a month beyond the last communication on the topic. Appreciate what you all do to provide the community for us, we ask because we care.

        • Thekingoflorda
          00·
          8 months ago

          You tagged the wrong admin (am not part of the tech team), but I’ll forward it to the other admins (: