New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Uia in ms word by default #8919
Conversation
… version 16.0.9000 and above.
…roblems directly: Office 2010 options dialog comboboxes have no name, and Outlook pre 2016 autoComplete list should use MSAA.
82e0703
to
935d29b
Compare
I haven't yet looked at the code. I recall that there is at least still one outstanding issue with UIA. Table navigation is currently a bit broken, empty cells are considered out of table. Does this implementation also allow you to switch back to the object model for Office 2016 build 9000 and above? |
Oh, I think I missed that issue. I'll look into this.
This PR does not allow re-enabling object model support. I am very very
reluctant to do this now as I would much prefer to be forced to address
the actual issues. Otherwise, too many people may turn off UIA and then
complain about performance for ever.
We really need to move on as technologies such as Windows Defender
Application Gard will be available in MS Office before too long, and
there will simply be no object model support.
Having said this though, if there is a big push for the option from
people then I'll accept it.
|
@bramd: Could you think of any issues that need to be addressed beforehand? |
Was this tested on Windows 7? I recall reading about freezes when Office 2016 was used with UIA enabled on Win 7, but i have no way to test this easily at the moment. |
If true, we could limit this to Windows 10.
I very much doubt we could work around the freezes in older versions of
Windows at this stage.
|
source/_UIAHandler.py
Outdated
and not config.conf['UIA']['useInMSWordWhenAvailable'] | ||
): | ||
# We can only safely check the version of known Office apps using the Word document control. | ||
# Other uses for now we just need to assume the implementation is good. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment is not clear to me yet.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this comment could be reworded. Are you saying that "for unknown office apps, which we can not safely check the version using the Word document control, we must just assume the implementation is good"?
Are we assuming that the UIA implementation is good?
source/_UIAHandler.py
Outdated
try: | ||
versionMajor,versionMinor,versionBuild,versionPatch=[int(x) for x in appModule.productVersion.split('.')] | ||
except Exception as e: | ||
log.error("Error parsing versioninformation %s, %s"%(appModule.productVersion,e)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing space between version and information
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This space is still missing.
@@ -156,9 +164,15 @@ def event_NVDAObject_init(self,obj): | |||
obj.role=controlTypes.ROLE_LISTITEM | |||
|
|||
def chooseNVDAObjectOverlayClasses(self, obj, clsList): | |||
# Currently all our custom classes are IAccessible | |||
if isinstance(obj,UIA) and obj.UIAElement.cachedClassName in ("LeafRow","ThreadItem","ThreadHeader"): | |||
clsList.insert(0,UIAGridRow) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you might one to return in this block, otherwise the logic under it is needlessly executed (i.e. fetching the role)
hi, |
An error occurs when you activate UIA, enable Browse mode, and then press H, the key to go head to head. Here is a log of the error:
|
@dnz3d4c: I need to know what exact version of Microsoft Word this was.
And I will also require a sample document where the issue occurs. I
cannot reproduce this in documents I have.
|
It is an internal document and can not be shared. However, when time permits, we will create a reproducible document.
|
… property. It is no longer needed in recent versions of MS Word, and in fact it was causing tables to not be reported when positioned on a blank cell.
Table navigation is now fixed in this PR.
However, we may need to increase the build number from 9000 to 11000. I
think something got fixed in MS Word somewhere along the line, which
allowed me to remove some code which used to be needed, but caused the
table navigation bugs.
The implementation they have now is much more accurate though.
|
You can reproduce the results with the attached document:
|
If you open the attached document, activate browse mode, and then perform the Control + End gesture twice, you will get the following error: ERROR - scriptHandler.executeScript (12:29:10.523):
|
…n. Seen at the bottom of MS Word documents.
@dnz3d4c:
I fixed the control+end problem. Thanks.
I can also no longer reproduce the heading problem in the latest
UIAInMSWordByDefault branch commits, though I could reproduce it in NVDA
alpha builds.
I'm assuming it may have got fixed either with the table navigation fix
or the COMError fix.
Here is the latest try build of this branch:
https://ci.appveyor.com/api/buildjobs/a3wegd33abnyp915/artifacts/output%2Fnvda_snapshot_try-UIAInMSWordByDefault-16234%2Ce809aca6.exe
|
Both issues seem to have been fixed. We'll let you know if we have other issues. Thank you.
|
Yeah, this really needs some testing, also on Windows 8 if possible.
I myself don't require it, but the object model is required to support chards in Word. |
I did some tests and it seems to be working fine in tables.
I had some crashes in Outlook on NVDA alpha versions in the past, but I
have no idea if they are UIA related. Sometimes when making a list in an
email message, Outlook totally crashed, NVDA would only log COMErrors. I
think I only saw this happen with uIA enabled, but hard to reproduce.
|
Hi, If you have time, can you take a look at #8502 comments and see if this PR can lend support for auto-complete suggestions in Office 2016/2019? Thanks. |
Hi, regarding suggestion sound playback, it can be done as these fields raise UIA controller for event (that’s what my try build for #8502 does). The necessary components are there – I think I’d wait for this PR to go into master before we expand the scope of UIA.SearchField class a bit. Thanks.
|
source/_UIAHandler.py
Outdated
and not config.conf['UIA']['useInMSWordWhenAvailable'] | ||
): | ||
# We can only safely check the version of known Office apps using the Word document control. | ||
# Other uses for now we just need to assume the implementation is good. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this comment could be reworded. Are you saying that "for unknown office apps, which we can not safely check the version using the Word document control, we must just assume the implementation is good"?
Are we assuming that the UIA implementation is good?
source/_UIAHandler.py
Outdated
try: | ||
versionMajor,versionMinor,versionBuild,versionPatch=[int(x) for x in appModule.productVersion.split('.')] | ||
except Exception as e: | ||
log.error("Error parsing versioninformation %s, %s"%(appModule.productVersion,e)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This space is still missing.
source/_UIAHandler.py
Outdated
return True | ||
if ( | ||
versionMajor<16 | ||
or versionMajor==16 and versionMinor==0 and versionBuild<9000 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I would prefer to see these values (16, 0, 9000) put in a named constant.
if winUser.getClassName(hwnd) in ("WeekViewWnd","DayViewWnd"): | ||
windowClass=winUser.getClassName(hwnd) | ||
# #2816: Outlook versions before 2016 auto complete does not fire enough UIA events, IAccessible is better. | ||
if windowClass=="NetUIHWND": |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are a few usages of this "NetUIHWnd" can you put them in a named constant, perhaps outlookWindowClass
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't believe this would improve things. A window class string is as gooder constant from my point of view. Something like OutlookWindowClass does not addiquitly describe the string as this NetUIHWND class is used in more than just Outlook. It seems to be some kind of generic window class for Ribbons. But to be clear, not the only one though, there is also NetUIToolWindow. Personally I think the strings are good enough. However, I guess we could convert them in to constants I.e. NetUIHWND="NetUIHWND"
but I really don't see the point.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok. My hope was that we could describe the string better. As someone unfamiliar with the value, it doesn't tell you any of the things you just mentioned. Having a constant with comment you just wrote would be helpful in my opinion. Ideally a name that describes it well, it seems like that is difficult.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actions look fine, much clearer. I had one more comment to make a case for a constant for that window class value, but I'll leave the decision up to you.
I tend to agree with Mick on not seeing the necessity of having a constant here. |
Since I don't use Word much myself, I had a friend try it. Do you have to do any configuration beyond downloading the snapshot and choosing "continue running"? In a 200 page document, he didn't experience any noticeable improvement when the file was being edited with word 16.0.11209 (i.e, the usual multiple second lag between key presses). |
source/_UIAHandler.py
Outdated
@@ -65,6 +66,10 @@ | |||
"FoxitDocWnd", | |||
] | |||
|
|||
Version=namedtuple('Version',('major','minor','build')) | |||
# The minimum version of Microsoft Word where we can trust that UI Automation is complete enough to use | |||
minMSWordUIAVersion=Version(16,0,90000) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This curently fails. I believe you wanted to say 9000 here.
… a table in the UIA tree, causing a 10 second freeze and infinite recursion error.
This pull request does not yet exclude Windows 7. I've been able to test on one Windows 7 machine, which caused freezes. I"m not sure what build it was, though. Might it be possible to track down the minimum required Windows version based on the version of the UIA interfaces shipped with it? |
Hi,
In my case, HTML mail works, but only when reading say all,
When reading with arrows, some parts are skipped
From: Ralf Kefferpuetz <notifications@github.com>
Sent: Thursday, January 17, 2019 11:27 AM
To: nvaccess/nvda <nvda@noreply.github.com>
Cc: Subscribed <subscribed@noreply.github.com>
Subject: Re: [nvaccess/nvda] Uia in ms word by default (#8919)
alpha-1657 is breaking some HTML mails for me under Outlook 365. Can’t arrow through the messages and SayAll doesn’t work for them as well. Going back to alpha-1655 or earlier resolves that. Tried it twice to go back and forth with the alpha versions. Something is in alpha-1657 that stops me from reading some HTML mails.
From: Michael Curran <notifications@github.com <mailto:notifications@github.com> >
Sent: Dienstag, 15. Januar 2019 10:45
To: nvaccess/nvda <nvda@noreply.github.com <mailto:nvda@noreply.github.com> >
Cc: Ralf Kefferpuetz <novalis7747@live.com <mailto:novalis7747@live.com> >; Manual <manual@noreply.github.com <mailto:manual@noreply.github.com> >
Subject: Re: [nvaccess/nvda] Uia in ms word by default (#8919)
Merged #8919<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnvaccess%2Fnvda%2Fpull%2F8919 <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnvaccess%2Fnvda%2Fpull%2F8919&data=02%7C01%7C%7Cc1386f866e834c656d9608d67ace288e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636831423186417610&sdata=wYuMsSKwk7Ec4gVV1kDR6JqvWFc6QOO3aD3e676S%2FR8%3D&reserved=0> &data=02%7C01%7C%7Cc1386f866e834c656d9608d67ace288e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636831423186417610&sdata=wYuMsSKwk7Ec4gVV1kDR6JqvWFc6QOO3aD3e676S%2FR8%3D&reserved=0> into master.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnvaccess%2Fnvda%2Fpull%2F8919%23event-2074087008 <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnvaccess%2Fnvda%2Fpull%2F8919%23event-2074087008&data=02%7C01%7C%7Cc1386f866e834c656d9608d67ace288e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636831423186573862&sdata=YHzqsDmaO8eiziElqnpUNHl%2Fh%2BnT8Agap2whKPU2xIQ%3D&reserved=0> &data=02%7C01%7C%7Cc1386f866e834c656d9608d67ace288e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636831423186573862&sdata=YHzqsDmaO8eiziElqnpUNHl%2Fh%2BnT8Agap2whKPU2xIQ%3D&reserved=0>, or mute the thread<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAffwnI8tmZdhy-gUIjJBa1fIBe4yg1vuks5vDaMugaJpZM4YNwhc <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAffwnI8tmZdhy-gUIjJBa1fIBe4yg1vuks5vDaMugaJpZM4YNwhc&data=02%7C01%7C%7Cc1386f866e834c656d9608d67ace288e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636831423186573862&sdata=5VgzEZTzy9bZ6cieRpXBb%2BYzU9TbtFxLWC7%2FlNQpp6Y%3D&reserved=0> &data=02%7C01%7C%7Cc1386f866e834c656d9608d67ace288e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636831423186573862&sdata=5VgzEZTzy9bZ6cieRpXBb%2BYzU9TbtFxLWC7%2FlNQpp6Y%3D&reserved=0>.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub <#8919 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AKohkx5a2NRYNexhCUtjANL_RJpn-8Eoks5vEFAEgaJpZM4YNwhc> . <https://github.com/notifications/beacon/AKohkx6lsY1vR10jEoflaEuru59wZIChks5vEFAEgaJpZM4YNwhc.gif>
|
That’s what I’m getting when trying to arrow down one of this HTLM mails with alpha-1657
ERROR - scriptHandler.executeScript (11:38:11.313):
error executing script: <bound method Dynamic_WordDocumentEditableTextWithoutAutoSelectDetectionUIAWindowNVDAObject.script_caret_moveByLine of <NVDAObjects.Dynamic_WordDocumentEditableTextWithoutAutoSelectDetectionUIAWindowNVDAObject object at 0x04EB1E30>> with gesture u'down arrow'
Traceback (most recent call last):
File "scriptHandler.pyc", line 187, in executeScript
File "editableText.pyc", line 185, in script_caret_moveByLine
File "editableText.pyc", line 144, in _caretMovementScriptHelper
File "editableText.pyc", line 130, in _caretScriptPostMovedHelper
File "speech.pyc", line 774, in speakTextInfo
File "NVDAObjects\UIA\wordDocument.pyc", line 185, in getTextWithFields
File "NVDAObjects\UIA\__init__.pyc", line 644, in getTextWithFields
File "NVDAObjects\UIA\__init__.pyc", line 634, in _getTextWithFieldsForUIARange
File "NVDAObjects\UIA\__init__.pyc", line 453, in _getTextWithFields_text
File "UIAUtils.pyc", line 115, in iterUIARangeByUnit
COMError: (-2146233079, None, (None, None, None, 0, None))
From: Ralf K.
Sent: Donnerstag, 17. Januar 2019 11:27
To: 'nvaccess/nvda' <reply@reply.github.com>; nvaccess/nvda <nvda@noreply.github.com>
Cc: Manual <manual@noreply.github.com>
Subject: RE: [nvaccess/nvda] Uia in ms word by default (#8919)
alpha-1657 is breaking some HTML mails for me under Outlook 365. Can’t arrow through the messages and SayAll doesn’t work for them as well. Going back to alpha-1655 or earlier resolves that. Tried it twice to go back and forth with the alpha versions. Something is in alpha-1657 that stops me from reading some HTML mails.
From: Michael Curran <notifications@github.com>
Sent: Dienstag, 15. Januar 2019 10:45
To: nvaccess/nvda <nvda@noreply.github.com>
Cc: Ralf Kefferpuetz <novalis7747@live.com>; Manual <manual@noreply.github.com>
Subject: Re: [nvaccess/nvda] Uia in ms word by default (#8919)
Merged #8919<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnvaccess%2Fnvda%2Fpull%2F8919&data=02%7C01%7C%7Cc1386f866e834c656d9608d67ace288e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636831423186417610&sdata=wYuMsSKwk7Ec4gVV1kDR6JqvWFc6QOO3aD3e676S%2FR8%3D&reserved=0> into master.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnvaccess%2Fnvda%2Fpull%2F8919%23event-2074087008&data=02%7C01%7C%7Cc1386f866e834c656d9608d67ace288e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636831423186573862&sdata=YHzqsDmaO8eiziElqnpUNHl%2Fh%2BnT8Agap2whKPU2xIQ%3D&reserved=0>, or mute the thread<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAffwnI8tmZdhy-gUIjJBa1fIBe4yg1vuks5vDaMugaJpZM4YNwhc&data=02%7C01%7C%7Cc1386f866e834c656d9608d67ace288e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636831423186573862&sdata=5VgzEZTzy9bZ6cieRpXBb%2BYzU9TbtFxLWC7%2FlNQpp6Y%3D&reserved=0>.
|
The UIA implementation still seems to have issues with tables in HTML e-mails in Outlook. I encountered two e-mails so far that wouldn't read at all using navigation keys on a braille display and read unreliably using the arrow keys. Downgrading to 2018.4.1 fixed the problems for both e-mails. I'll investigate more soon. |
I'm guessing that most people here who are having trouble with Outlook html emails are finding that browse mode is not automatically enabled when reading emails? that is what I am seeing on my system. |
I am now thinking that this is still not ready for 2019.1. I want to keep all the NetUI / Outlook stuff, plus any improvements to UIA in MS Word. But, having it on by default should not happen until more things are fixed in Microsoft word itself. E.g. color issues. Layout tables in Outlook etc. |
Would be appreciated not to be implemented, because it breaks the html outlook mails. If you need example mails let me know where to forward them.
From: Michael Curran <notifications@github.com>
Sent: Mittwoch, 23. Januar 2019 06:27
To: nvaccess/nvda <nvda@noreply.github.com>
Cc: Ralf Kefferpuetz <novalis7747@live.com>; Manual <manual@noreply.github.com>
Subject: Re: [nvaccess/nvda] Uia in ms word by default (#8919)
I am now thinking that this is still not ready for 2019.1. I want to keep all the NetUI / Outlook stuff, plus any improvements to UIA in MS Word. But, having it on by default should not happen until more things are fixed in Microsoft word itself. E.g. color issues. Layout tables in Outlook etc.
Therefore for 2019.1 we should go back to relying on the hidden useInMSWordWhenAvailable UIA config option, though perhaps consider adding the Advanced tab to the settings dialog and exposing it there, so more people experiment with the feature.
At the same time, I want to do some more performance profiling of the existing object model supprt for MS Word, and see if we can tighten things up some more.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnvaccess%2Fnvda%2Fpull%2F8919%23issuecomment-456674994&data=02%7C01%7C%7Cedd87c28dbc14aa79a5008d680f35f32%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636838180088185651&sdata=97z8CoVgE%2Bmh48gIAr4bJA95jZ9j4ywwAIPQMxsSZIw%3D&reserved=0>, or mute the thread<https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAffwnBQY6jiMi_clHvl_6bC1koMSZTb9ks5vF_KWgaJpZM4YNwhc&data=02%7C01%7C%7Cedd87c28dbc14aa79a5008d680f35f32%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636838180088185651&sdata=a4I8ckEIihLntX8shAeYXSz7NzeXXhQfB1d%2BWb63SEw%3D&reserved=0>.
|
@michaelDCurran wrote:
I tend to agree.
This makes sense. It is related to #6788, though that proposal goes a bit further than a simple advanced tab.
That would really be helpful, as there many people still relying on older version of Office, particularly people who are at Office 2016 volume licenses, or Office 2013 and older. |
Are there plans to reconsider UIA in Word by default for a future Office/NVDA version? Have the previously-reported issues been resolved? |
I remember the last time I tried it table navigation still didn't work right for some tables, along with the other issue of performance degrading after 10 minutes or so, requiring a restart of NVDA to continue working for the next 10 minutes. |
Indeed, there are still some major drawbacks. May be they can be fixed
at NVDA's side, may be not.
|
Link to issue number:
Closes #8502
Closes #7409
Summary of the issue:
Microsoft Word 2016 exposes a rich UI Automation implementation. For some time now, users have been able to optionally turn this on with a hidden config value. It seems that NVDA's support for MS Word via UIA has major performance advantages over the older object model support, so NVDA should use the UIA support by default where available. However, as the UIA implementation improved throughout Office 2016's lifetime, we should only enable our support for recent builds of Office 2016.
Description of how this pull request fixes the issue:
This PR causes NVDA to support MS Word via UI Automation for Office 2016 build 9000 and higher. Users on lower builds can still force UIA support with the hidden config value.
This PR also ensures that the MS Office ribbon is accessed with UI Automation, both when expanded and collapsed, which solves some issues with focus not reporting for some ribbon controls.
As a part of this change, it was necessary to either move, or come up with an alternative fix, for a couple of previously solved issues. Namely supporting the autoComplete list in Outlook (#2816), and the comboboxes in the MS Office Options dialogs (#4056), as they use the NetUI framework just like the Ribbon.
Testing performed:
Known issues with pull request:
None.
Change log entry:
Bug fixes: