Kickstart discussions about providing test harness APIs

[emilio]

We should try to start some discussion in https://github.com/web-platform-tests/rfcs to add APIs for at least:

• Pinch-zoom.
• Dynamic toolbar size(s), so that we can test dvh / svh / lvh properly.

Maybe:

• Virtual keyboard?

Something else?

[karlcow]

@emilio

providing test harness APIs

Do you mean with WebDriver capabilities or something else?

[bramus]

Infra

@skobes WPT run on real devices?
@emilio Definitely in emulators.
@skobes Testing on real devices would make for more accurate results.
@skobes In Chrome for example, the code for the dynamic toolbars is Mobile specific.
@emilio Maybe just an (internal) API to mimic things on desktop?
@dlibby- If we run on real devices? How would we define the tests and cater for top / bottom address bars vs scrollOffsets

Priority of testing infra wishlist:

1. Mobile Browser on Real device
2. Mobile Browser in Emulator
3. Browser on desktop with ability to have it mimic mobile browser behavior

@emilio WebKit and Gecko have Native APIs to control the dynamic toolbars on Mobile (and maybe Desktop).
@skobes Don’t think Blink has this kind of APIs exposed to embedders
@emilio Suggestion to go through APIs that browsers use nowadays to change certain things
@emilio Those APIs could then be exposed to webdriver, to use in the WPT tests
@skobes Some things not possible to fake in Chromium Desktop, because some code is very platform specific
@bramus Don’t think we need to expose mobile specific APIs on desktop that allow mimicing mobile behavior
@emilio Example Code from Firefox: https://searchfox.org/mozilla-central/rev/52da19becaa3805e7f64088e91e9dade7dec43c8/mobile/android/geckoview/api.txt#610-611,613-614
@emilio Other vendors could implement similar APIs. Desktop browsers could implement these as well.
@bramus If desktop implements for example setDyamicToolbarHeight, would they mimic the mobile behavior or do a NOOP if they have no toolbar?
@emilio

---

Tests

E.g. Measure ICB/LVP/VVP/… on load vs after scroll

Noteworthy:

• scrolling down vs scroll up
• gesture lift finger up vs keep finger on glass
• slow scroll vs flick scroll
• overscroll

Also monitor events

---

Actions items

• @bramus Formulate basic lists of test, for review next meeting
• @dcrousso Can you compile a list of APIs similar to these ones from gecko?

[jgraham]

Not being able to get test results on wpt.fyi is a blocker to being able to include viewport items in any future Interop-20xx focus area, even if we have the appropriate testing APIs.

If we want the tests to run on mobile, the main blocker is availability of appropriate hardware.

For gecko in our own CI we're running wpt in the Android emulator. This is pretty easy to also support upstream (some initial work is landed), but it's blocked on not having the right kind of machine type available to run the Android emulator in the upstream CI (taskcluster or Azure). This requires KVM, which AIUI requires a more expensive bare-metal cloud instance, vs the cheaper VM-based instances we're currently using.

For running on real devices (or an iOS simulator that depends on macOS hardware), we'd need to get access to the relevant hardware. For physical devices in particular, we don't have any resources to run a device farm ourselves, so this would require some kind of service that can abstract over the actual phones.

If we can't run in upstream CI, there are basically two options:

• Get vendors to run the tests locally on their existing mobile testing setups and upload the results. But this is quite hard to synchronise and may lead to results that are not comparable for reasons which are hard to control.
• Depend on devtools-like mobile emulation in a desktop browser. As discussed above this is less representative of real use than running on an actual device (but may be relatively achievable for at least some tests).

So I think if we believe that this is an important area for progressing the platform, there's a case to be made for getting some participants to help provide the testing resources required to run mobile in wpt CI.

[gsnedders]

FWIW: I think any discussion about running tests on mobile platforms should probably happen in an RFC issue, rather than here. There's a lot of complexity here for a variety of reasons, not just limited to access to the actual hardware.

[jgraham]

Right, but I think it's worth ensuring that the people talking about testing mobile-dependant features are aware of the current state of testing on mobile, and what's required to include those features in Interop focus areas in the future. I don't think a wpt RFC would reach that audience (plus there's nothing to propose at the moment, so it's unclear what an RFC would say).

I agree it's also possible that I'm missing some considerations, especially for iOS where don't have any relevant experience. The above is probably sufficient to testing Firefox mobile on an Android emulator, but it might not cover all other platform/browser considerations.

[gsnedders]

I don't think a wpt RFC would reach that audience (plus there's nothing to propose at the moment, so it's unclear what an RFC would say).

Having an issue to at least have discussion about what would be required to test mobile browsers seems reasonable?

[jgraham]

Sure. I don't know it's the best place for such a discussion, but I also don't have a better suggestion.

[karlcow]

running tests on mobile platforms should probably happen in an RFC issue

@jgraham @gsnedders Feel free to edit directly the issue text.

[bramus]

Thank you for filing this, @karlcow.

[bramus]

I’ve compiled a list of possible things to test, grouped per API. For this, I’ve based myself upon the manual tests I performed earlier this year (testpages / findings) which were grouped per action.

1. The ICB
   • Page Load:
     • ICB should have same dimensions as the Layout Viewport
     • ICB should have same dimensions as window.innerHeight/window.innerWidth (overlay scrollbars) or be slightly less than window.innerHeight/window.innerWidth (classic scrollbars)
     • ICB should have same dimensions as the Visual Viewport
   • Scroll Down: should not affect value
   • Scroll Up: should not affect value
   • Focus Input: should not affect value
   • Pinch Zoom In: should not affect value
   • Pinch Zoom In + Pan Around: should not affect value
   • Pinch Zoom Out: should not affect value
   • Over-Pinch Zoom Out: should not affect value
   • Overscroll at top: should not affect value
2. The Layout Viewport
   • Page Load:
     • Layout Viewport should have same dimensions as the ICB
     • Layout Viewport should have same dimensions as window.innerHeight/window.innerWidth (overlay scrollbars) or be slightly less than window.innerHeight/window.innerWidth (classic scrollbars)
     • Layout Viewport should have same dimensions as the Visual Viewport
   • Scroll Down: LVP can grow, due to UA UI retracting (if any)
   • Scroll Up: LVP can shrink, due to UA UI retracting (if any)
   • Focus Input: should not affect value
   • Pinch Zoom In: should not affect value
   • Pinch Zoom In + Pan Around: should not affect value
   • Pinch Zoom Out: should not affect value
   • Over-Pinch Zoom Out: TBD
   • Overscroll at top: TBD
3. The Visual Viewport
   • Page Load:
     • Visual Viewport should have same dimensions as the ICB
     • Visual Viewport should have same dimensions as window.innerHeight/window.innerWidth (overlay scrollbars) or be slightly less than window.innerHeight/window.innerWidth (classic scrollbars)
     • Visual Viewport should have same dimensions as the Layout Viewport
   • Scroll Down:
     • Visual Viewport can grow, due to UA UI retracting (if any)
     • pageTop should follow window.scrollX
   • Scroll Up: Visual Viewport can shrink, due to UA UI retracting (if any)
   • Focus Input:
     • Visual Viewport must shrink
     • Depending on position of input, offsetTop/offsetLeft can change value
       • offsetTop/offsetLeft is clamped between 0 and (LVP size - VV size)
   • Pinch Zoom In:
     • Visual Viewport must shrink
     • scale must be greater than 1
     • Depending on position of pinch zoom (top left vs elsewhere), offsetTop/offsetLeft can change value
       • offsetTop/offsetLeft is clamped between 0 and (LVP size - VV size)
   • Pinch Zoom In + Pan Around: should not affect value
     • offsetTop/offsetLeft must change when panning around inside LVP boundaries
     • pageTop/pageLeft must not change when panning around inside LVP boundaries
   • Pinch Zoom Out: should not affect value
     • Visual Viewport must grow again
     • offsetTop/offsetLeft must become 0
     • scale must become 1
   • Over-Pinch Zoom Out: TBD
   • Overscroll at top: TBD
4. Viewport Units
   • Page Load:
     • sv*: same as ICB
     • lv*: greater or equal to ICB (if browser has dynamic UA UI elements)
     • dv* units: same as sv*
     • v* units: same as lv*
   • Scroll Down:
     • sv* units: same value as on load
     • lv* units: same value as on load
     • dv* units: same as lv*
     • v* units: same as lv*
   • Scroll Up:
     • sv* units: same value as on load
     • lv* units: same value as on load
     • dv* units: same as sv* again
     • v* units: same as lv*
   • Focus Input:
     • sv* units: same value as on load
     • lv* units: same value as on load
     • dv* units: either sv* or lv*, depending on whether UA UI is shown or not.
     • v* units: same as lv*
   • Pinch Zoom In:
     • sv* units: same value as on load
     • lv* units: same value as on load
     • dv* units: either sv* or lv*, depending on whether UA UI is shown or not.
     • v* units: same as lv*
   • Pinch Zoom In + Pan Around:
     • sv* units: same value as on load
     • lv* units: same value as on load
     • dv* units: either sv* or lv*, depending on whether UA UI is shown or not.
     • v* units: same as lv*
   • Pinch Zoom Out:
     • sv* units: same value as on load
     • lv* units: same value as on load
     • dv* units: either sv* or lv*, depending on whether UA UI is shown or not.
     • v* units: same as lv*
   • Over-Pinch Zoom Out:
     • sv* units: same value as on load
     • lv* units: same value as on load
     • dv* units: either sv* or lv*, depending on whether UA UI is shown or not.
     • v* units: same as lv*
   • Overscroll at top:
     • sv* units: same value as on load
     • lv* units: same value as on load
     • dv* units: either sv* or lv*, depending on whether UA UI is shown or not.
     • v* units: same as lv*
5. window.innerHeight
   • Page Load:
     • window.innerHeight/window.innerWidth should have same dimensions as the ICB or be slightly greater than the ICB (classic scrollbars)
     • window.innerHeight/window.innerWidth should have same dimensions as the Layout Viewport or be slightly greater than the Visual Viewport (classic scrollbars)
     • window.innerHeight/window.innerWidth should have same dimensions as the Visual Viewport or be slightly greater than the Visual Viewport (classic scrollbars)
   • Scroll Down: should not affect value
   • Scroll Up: should not affect value
   • Focus Input: should not affect value
   • Pinch Zoom In: should not affect value
   • Pinch Zoom In + Pan Around: should not affect value
   • Pinch Zoom Out: should not affect value
   • Over-Pinch Zoom Out: should not affect value
   • Overscroll at top: should not affect value

There’s some TBD’s mentioned in the list, as not all is clear right now in the case of over pinch-zooming out and overscrolling at the top. On mobile browsers, these all behave a little bit differently right now (some offsetting the entire canvas, some just offsetting the ICB+VV, some …)

[skobes-chromium]

(For discussion) it's still unclear to me exactly what we mean / want with stuff like

"Scroll Down: LVP can grow, due to UA UI retracting (if any)"

Does it mean

• the test scrolls down, and verifies that the UA UI retracts
• the test scrolls down, simulates (through a test harness API) that the UA UI retracted, and verifies that the LVP grew
• the test asks (through the test harness) whether the UA UI supports retraction, and if it does, it scrolls down, then verifies that the UA UI retracts?
• something else?

[bramus]

I would say:

1. Measure “before” value
2. Scroll Down
3. Request UA UI to retract (which would be a NOOP on certain platforms)
4. Measure “after” value
5. Compare values

A more thorough check than “LVP$after must be gte LVP$before” would be to calculate the height of the UA UI (which is equal to the difference between lvh and svh) and take that into account in the comparison. This extra check can only be included if the UA has support for Viewport Units, because otherwise you could end up with false negatives (in case the UA correctly retracts, but does not support the viewport units)

[bramus]

• Two TBDs
  • Over-Pinch Zoom Out: page should retain its current aspect-ratio
  • Overscroll at top: when LVP stays in place (which it should), Visual Viewport should stay as well (as it is contained inside the LVP)
• The tests
  • Scroll Down: Kickstart discussions about providing test harness APIs #23 (comment) makes sense
  • We need to distill which internal APIs we need to achieve this:
    • Retract UA UI
    • Expand UA UI
    • Show OSK
    • Hide OSK
    • Set Page Scale
    • Pinch-Zoom (already covered by touch actions most likely)
    • Overscrolling (can current scroll handle it?)
    • Ability to get properties of device, e.g. screen size, pixel density, size of UA UI, … (because maybe not directly web exposed)

[bramus]

I’ve followed up with Interop Tooling Team at Google on this, and they’ve added the list of requested APIs it to the 2023 backlog (pending prioritisation).

Not sure what’s next here. Do we graduate the list of suggested tests + API requirements to an RFC?

[bramus]

@jgraham Emilio told me you can help out with this. See last comment: how do we move this to WPT?