Thank you all for reporting this and for your patience with us.
We are aware and are now working on this. Hang in there, hopefully we can roll it out today or tomorrow at latest.
I owe you an explanation for why this back and forth happened.
It failed, then worked, then failed again because we had to do a multi step catch up on all the code changes (CLs) over the last week that had been accumulating (for different reason due to Monitoring metric instrumentation work we are doing) and then a general roll back over the week end due to another issue with PDF workflow failing to send.
Each fix (CL) is incorporated into a build independently, test independently, then tested all together. When we have an issue that affect a major functionality and production application for customers, we first roll back everything immediately to the previous working state in order to restore the functionality while investigating the root cause (sometimes these causes are only apparently in real usage and not with generic test scenarios).
So the back and forth here is just the result of a general roll back, which had a side effect of reintroducing a bug that had a visual effect rather than breaking a feature. At 10PM last night we decided to call it off and get rest and resume working on it today.
I understand this decision might frustrate some of you. I can totally understand this.
The reason we do it, is because it is generally not a good idea to make emergency fixes in production very late in the day when folks are tired after two days working through the week end, and we donโt have the full team on hand to code review and test extensively.
We called it off and decided it is better to investigate how to get back into the right place in the morning when everyone is rested and has fresh eyes.
Again, we want to thank you for your loyalty and candid feedback helping us make this a better platform everyday for everyone.
Thierry
LinkedIn
Twitter
