AppSheet QA testing of new features (and even olde...

WillowMobileSys · 08-12-2021 07:32 AM

I wish to start a conversation concerning testing of new or enhanced features.

I have noticed over the years that when new features are rolled out, there are frequently inconsistencies in how the feature behaves that seems should have been easy to catch. An example is the Rich Text feature just recently made available to Detail and Form view. When the Detail view is placed in a Dashboard, the Rich Text behaves differently

So the question is: How much QA testing should we expect to be performed on the AppSheet side versus Community testing when the feature is rolled out?

One approach is that AppSheet performs QA testing with the goal of establishing a pristine set of coded features before rolling it out. Community then tests integration into their unique apps for any usage anomalies before the feature is rolled out for common use.
A second approach would be for AppSheet to perform just enough testing to get the feature working then roll out to the Community to perform a thorough QA testing reporting back any issues found.

It seems to me that we are kind of in between these 2 approaches which leaves gaps in testing efforts and introduces bugs that should have easily been found. We really should come to a common understanding of the testing expectations.

If the lean is more towards the second approach I think that is okay but AppSheet should explicitly state so. Community members would be more thorough in their testing. And maybe some compensation should be given to those members who return testing reports - for their efforts in helping establish a pristine platform.

If the first approach is the expectation then, I’m sorry AppSheet but you need to do a better job at more thorough testing covering all areas feature may be utilized within the app!

Marc_Dillon

I think I speak for the vast majority of users here that we want more new features, and we want them faster.

How do we get that? One way is by lessening the workload for the Appsheet Developers.

How can we lessen that workload? One way is to lessen the amount of thorough testing they have to do when releasing a new feature. Thorough testing is tough!

I 100% support the release of partially-completed and/or partially-tested features into the preview program in order to be crowd-source-tested and improved.

I also think it is important to re-iterate that of course these features aren’t complete, that’s why they are in preview! Thus there’s no reason to get so worked up about their lack of completeness.

GreenFlux

Except the entire Automations section, which is a major portion of the platform. There were still many major bugs after workflows/reports were disabled. I think the preview program is a good approach, but it wasn’t used on the Automations rollout, and was a big part of my decision to shift my focus to other platforms.

Marc_Dillon

Which means that what I said above does not apply to that particular feature rollout. Rage on…

GreenFlux

No rage here, just agreeing with both of you. The preview program is a decent solution, but I completely agree with @WillowMobileSystems 's frustrations.

WillowMobileSys

I started to say this in the original post but removed it to from getting long.

I think AppSheet is a great platform! I like their responsiveness to the Community - could be better but still much better than many others. I like how quickly we can implement common apps and features. But also like the flexibility to create more complex capabilities within our apps. I mean I have been able to create a COMPLETE software system.

My op is not meant to be a rage from here either. Just want to come to consensus on testing expectations. I don’t believe there is a common understanding on the testing effort needed/expected from the Community

Frustrated? A little. Only because some issues seem very petty, easily identified and reoccurring. For example, the Rich Text feature highlighted once again something that behaves differently in a standard view versus when placed in a Dashboard. Why do we have to keep pointing out the difference?

Steve

Until they’re release to GA–still incomplete. And replace what had been working just fine.

WillowMobileSys

Agree totally!

I too am OK with “crowd-source-tested” features. I have always assumed AppSheet should/would do a “complete” test then turn over to the Community. That doesn’t seem to be the case.

Again agree! This gets to the main purpose of the post.

It seems understood that AppSheet is relying largely on the Community to provide testing feedback. To help lighten the workload for AppSheet, its then fairly obvious that the Community should expect to do some dedicated in depth testing of any new features. I don’t think that is the understanding at the moment.

By the way, a quick shout out to @tsuji_koichi and @Takuya_Miyai for the testing efforts they provide on new rollouts. They are a good role-model for the rest of us to emulate.

Steve

To my knowledge, AppSheet doesn’t have any formal QA process. None that I’m aware of, at least.

Steve

takuya_miyai

Hi @WillowMobileSystems ,

Thank you for starting this valuable thread and for following it there.

I have similar thoughts to @Marc_Dillon .

I’m glad to see that the quality of AppSheet testing is improving, but with all the different applications being created, I think it will be difficult to have a complete test that encompasses everything down to the application layer.

I think that the current Preview program, in which App Creator can be involved, is doing a great job on this topic.
I remember that the Preview program was launched less than a year ago, so of course there are some things that need to be improved, but I think it will gradually improve.

I’ve experienced a similar theme in Salesforce development, and it’s working very well within the Salesforce ecosystem so far.

They are developing features for three regular releases per year.
They pre-release to the Sandbox environment about 4 to 6 weeks before the production release.
During this time, Salesforce developers verify that there are no problems with the apps they manage, and of course report any problems to Salesforce.
Salesforce will postpone the release of the relevant feature to the next release if it is difficult to fix before the production release.
These verifications are actively done within the Salesforce community (like this community! )

https://help.salesforce.com/s/articleView?id=000357271&type=1

I’m not sure how AppSheet’s future QA testing is planned, but I’m looking forward to building a quality improvement framework that includes App Creater in the same way.

WillowMobileSys

Just to re-iterate/clarify - I didn’t mean for this thread as a complaint and I am not trying to argue for one-way or another. I simply want it to be clear of the expectations from the Community (and AppSheet as well) with regards to testing. IMHO, there are too many small things that slip through the cracks, end up in the production environment that we as App Creators have to contend with daily. I want to have a solid, clean platform and willing to do whatever is needed to get us there. But we all need to band together to make it happen!

I whole-heartedly agree - there are ways in which developers will use features that others couldn’t or wouldn’t even fathom. Those things need tested by the App Creators themselves.

Maybe I haven’t clearly stated what types of “things” I have had in mind for this conversation. I am really focusing on the app-generic features, such as

Rich Text behaving on Detail views consistently - even on Dashboards.
Card views spilling outside the host container.
Incorrect large column widths within Nested tables of Card Views and Deck Views.
Truncated Summary field on Deck view.
etc.

Again, not a complaint, just a fact that these issues are there, slipped through due to inadequate testing.

My question - For these app-generic things, WHO should we expect to test and identify issues with these features? AppSheet or Community of BOTH?

takuya_miyai

Yes, I would like AppSheet to support this range.
Whenever a new feature is released, I create a mock from an idea to check the behavior, and I would like to eliminate as many bugs and uncomfortable behaviors as possible that can be found in a mock application.

WillowMobileSys

@Marc_Dillon What are your thoughts on this specific question?

Anyone else wish to brave a response?

Lucinda_Mason

I am not participating in the preview because I don’t have the time. My app development is just a part of my full time job. I will say that AppSheet has improved tremendously since I started about 18months ago.

But, there are still issues that get through and don’t get fixed fast enough.

For ex.
Adding a row filter condition. With the new UX, you can’t just type it in the box. It doesn’t do anything.

You have to click Create a Custom Expression. Then it puts what you typed in the box as the DESCRIPTION!

After you cut and paste the expression into the expression editor and save, the description over-writes the expression. Well, most of mine are long enough that it gets really messy.

This is just one of many glitches that should be easily fixed. Actually, I think it should have been caught in testing.

Today I found that when converting a report to a BOT, the description wasn’t copied over. That is simple mapping same data to same data. Thankfully, besides that issue, the bots I converted all worked without any other fixing. Now I have 13 more Workflows and 10 more reports to convert before they are done automatically - I don’t trust that process so I will continue to make screenshots to document each workflow/report and compare it the BOT. That is how I found the descriptions weren’t being copied over.

Just my thoughts. Remember, I am what AppSheet calls a true citizen developer. I do not design and build apps for a living. I design and build apps in order to do my work more efficiently and to make products that will help further the goals of the non-profit organization I work for. I am not in the IT department.

AppSheet QA testing of new features (and even older ones) vs Community testing