Xen 4.4 updates vs. Xen Stretch backport

[Peter Dreuw]

Hi out there,

as you might have noticed, we fixed many issues with Xen 4.4 in Jessie.
cf. https://security-tracker.debian.org/tracker/source-package/xen

With this, all current "trivial" cases are closed (ignoring the few arm
already marked no-DSA before the LTS support stepped in) These might be
easy to fix at some point but currently I don't see the real point in
spending too much time on these.

The open cases are

TEMP-0000000-20B25C = XSA-280

TEMP-0000000-319B92 = XSA-279

TEMP-0000000-EC90C0 = XSA-275

CVE-2018-3620, CVE-2018-3646 = XSA-273

CVE-2018-3665 = XSA-267

CVE-2018-3639 = XSA-263

CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 = XSA-254 - which is not in
the Debian tracker for Xen, actually...


While XSA-275 and XSA280 might be easy to apply the upstream fix,
XSA-279 does not apply to the current Xen 4.4 state. XSA-279 does only
affect after implementing the XSA-254 (Meltdown) fixes. From this
perspective. XSA-279 could be safely ignored until the back ports are done.

XSA-273 could be fixed only if microcode and kernel is fixed too.
According to the bug tracker, for the kernel this is not the case yet.
The patch relies on the code fixing spectre / meltdown issues so it had
to be postponed until these fixes have been ported. Only Intel CPU might
be vulnerable. A mitigation is possible but undesirable due to heavy
performance impacts.

XSA-267 could be fixed as there is a fixed kernel in Jessie security.
The first patch for this can be applied directly, the second one relies
on code for XSA-254 (spectre / meltdown). Mitigation is possible by cpu
pinning to VMs.

XSA-263 depends on fixing XSA-254 too. The other constraints like kernel
and microcode are fixed already. There is no other mitigation known but
fixing the code and firmware.

XSA-254 aka Spectre / Meltdown is still open for Xen but never made it
to the Debian security tracker for Xen, surprisingly. Currently, there
is no mitigation for CVE-2017-5753 (Spectre variant 1, SP1) For SP2,
Spectre CVE-2017-5715 there are the BTI fixes in upstream. For SP3, aka
Meltdown, CVE-2017-5754, running guests in PVH or HVM context. PV guests
could be run under special shim hypervisors available for Xen 4.10 and
up. There are shim back ports for Xen 4.8. Alternatively, there are the
page table isolation (PTI) patches that mitigate the Meltdown issue too.
Sadly, the PTI patches rely on the BTI patched code. There are 43 BTI
upstream patches for Xen 4.6 that need to be back ported.

These 43 patches to fix SP2 introduce the code basis for XSA-279,
XSA-273, XSA 267 and XSA-263 listed above.

The major question is: Are we traveling this road, implementing / back
porting the BTI fixes for XSA-254?

If so, the other fixes are probably not to much work. But implementing
BTI fixes is a long and unknown road. I cannot give any reliable numbers
how much work that would be. But anybody can estimate that this will be
much more than a few days to get done. There might be a shortcut for
some patches by back porting independent code chunks like I did with the
grant table code for Xen 4.1 (Wheezy) but for now, I can't oversee all
of this in total yet and I doubt that there will be a great shortcut to
be found.


Another option would be backporting the Xen
4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10 (and following) package from
Stretch to Jessie. This could be done including testing within a few
hours, maybe a little more than a working day or less if we abandon Xen
4.4.

Along with Xen 4.8 there might be some further impacts as e.g. libxen
changes, too. This might break some unpackaged software depending on this.


As changing the minor version of a package like Xen is kind of a break
in expectations people might have in LTS. Therefor, I'd like to ask for
feedback on both options and your opinion, which way to get to a solution. 

Don't get me wrong, I am not unwilling to work on a back port of these
fixes but this will not be done within a short amount of time and
honestly I cannot guarantee that there will be a 100% solution. A
Stretch back port on the other hand could be ready very soon.

Kind regards

Peter
--
Peter Dreuw
Teamleiter
Tel.: [phone]
Fax: [phone]
E-Mail: ***@credativ.de

gpg fingerprint: 33B0 82D3 D103 B594 E7D3 53C7 FBB6 3BD0 DB32 ED41
http://www.credativ.de/

**********************************************
Jetzt neu:
Elephant Shed - PostgreSQL Appliance
PostgreSQL und alles was dazugehört

Von Backup ÃŒber Monitoring bis Reporting:
https://elephant-shed.io/index.de.html
**********************************************

credativ GmbH, HRB Mönchengladbach 12080
USt-ID-Nummer: DE204566209
Trompeterallee 108, 41189 Mönchengladbach
GeschÀftsfÌhrung: Dr. Michael Meskes, Jörg Folz, Sascha Heuer

Unser Umgang mit personenbezogenen Daten unterliegt
folgenden Bestimmungen: https://www.credativ.de/datenschutz

[Moritz Muehlenhoff]

Post by Peter Dreuw
Hi out there,
Another option would be backporting the Xen
4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10 (and following) package from
Stretch to Jessie.

What would be the point? If you migrate to a complete new Xen release,
then you can just as well migrate to stretch (which will also have
proven, compatible matching versions of libvirt/Linux/qemu/ etc.

If some of the Spectre mitigations can't be backported, make a detailed
writeup of what people are missing in 4.4 and let them handle it
based on that data (update to stretch or stick with 4.4/jessie); there's
still plenty of legitimate use cases which can be run in a secure
manner with 4.4 (internal VMs with trusted users etc).

Cheers,
Moritz

[Antoine Beaupré]

Post by Moritz Muehlenhoff

Post by Peter Dreuw
Hi out there,
Another option would be backporting the Xen
4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10 (and following) package from
Stretch to Jessie.

What would be the point? If you migrate to a complete new Xen release,
then you can just as well migrate to stretch (which will also have
proven, compatible matching versions of libvirt/Linux/qemu/ etc.
If some of the Spectre mitigations can't be backported, make a detailed
writeup of what people are missing in 4.4 and let them handle it
based on that data (update to stretch or stick with 4.4/jessie); there's
still plenty of legitimate use cases which can be run in a secure
manner with 4.4 (internal VMs with trusted users etc).

I agree. It's not like Spectre is trivial to exploit either, so the
tradeoff might be acceptable for some users.

Xen upgrades are usually fairly smooth, but considering the dom0 is most
likely *only* running Xen, upgrading to stretch is probably equivalent
than upgrading to a Xen backported from stretch.

So while I usually am a proponent of aggressive backports, I think that,
in this case, we would actually be doing a disservice to our users by
forcibly backporting a version from stretch. :)

Peter, are there non-speculative vulnerabilities remaining we could look
at?

Otherwise I would just publish a DLA saying we simply stop supporting
that aspect of Xen...

A.

[Ben Hutchings]

On Wed, 2018-11-28 at 12:59 +0100, Peter Dreuw wrote:
[...]

Post by Peter Dreuw
While XSA-275 and XSA280 might be easy to apply the upstream fix,
XSA-279 does not apply to the current Xen 4.4 state. XSA-279 does only
affect after implementing the XSA-254 (Meltdown) fixes. From this
perspective. XSA-279 could be safely ignored until the back ports are done.

That makes sense.

Post by Peter Dreuw
XSA-273 could be fixed only if microcode and kernel is fixed too.
According to the bug tracker, for the kernel this is not the case yet.
The patch relies on the code fixing spectre / meltdown issues so it had
to be postponed until these fixes have been ported. Only Intel CPU might
be vulnerable. A mitigation is possible but undesirable due to heavy
performance impacts.

CVE-2018-3646 specifically relates to hypervisors using EPT. It is
open for Linux because KVM needs mitigations for it, but I don't
believe that a fix for Xen would depend on any of the changes in Linux.

I wish you would query possible Xen/Linux dependencies with me rather
than quietly deferring the Xen fixes.

Post by Peter Dreuw
XSA-267 could be fixed as there is a fixed kernel in Jessie security.
The first patch for this can be applied directly, the second one relies
on code for XSA-254 (spectre / meltdown). Mitigation is possible by cpu
pinning to VMs.

Based on a very quick look, I don't see any complex interaction with
the earlier mitigations, so it might be backportable without all of
XSA-254.

Post by Peter Dreuw
XSA-263 depends on fixing XSA-254 too. The other constraints like kernel
and microcode are fixed already. There is no other mitigation known but
fixing the code and firmware.
XSA-254 aka Spectre / Meltdown is still open for Xen but never made it
to the Debian security tracker for Xen, surprisingly.

So let's add it there.

Post by Peter Dreuw
Currently, there is no mitigation for CVE-2017-5753 (Spectre variant 1,
SP1)

Indeed, no general mitigation seems to be possible with reasonable
performance impact. However there is a static checker available that's
been used to identify likely vulnerable sites in Linux and there is an
ongoing flow of fixes based on this; is the same happening in Xen?

Post by Peter Dreuw
For SP2, Spectre CVE-2017-5715 there are the BTI fixes in upstream.

Are those likely to be backportable?

Post by Peter Dreuw
For SP3, aka
Meltdown, CVE-2017-5754, running guests in PVH or HVM context. PV guests
could be run under special shim hypervisors available for Xen 4.10 and
up.

But according to the XSA, running Linux PV guests in this way makes
*them* vulnerable to Meltdown (since Linux expects Xen to swap page
tables in PV mode, and it won't in this case).

Have we already advised users to use HVM/PVH for guests other than
dom0?

[...]

Post by Peter Dreuw
If so, the other fixes are probably not to much work. But implementing
BTI fixes is a long and unknown road. I cannot give any reliable numbers
how much work that would be. But anybody can estimate that this will be
much more than a few days to get done. There might be a shortcut for
some patches by back porting independent code chunks like I did with the
grant table code for Xen 4.1 (Wheezy) but for now, I can't oversee all
of this in total yet and I doubt that there will be a great shortcut to
be found.

Having spent several days on similar backports for Linux 3.2 and 3.16,
I recognise the likely difficulty and complexity of the task and I
think it still needs to be done.

(But for future releases we do seriously need to consider whether Xen
should be covered by LTS, given the amount of work needed.)

Post by Peter Dreuw
Another option would be backporting the Xen
4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10 (and following) package from
Stretch to Jessie. This could be done including testing within a few
hours, maybe a little more than a working day or less if we abandon Xen
4.4.

[...]

I don't see this as an acceptable option for LTS. We could maybe add a
xen-4.8 package if it was popular in jessie-backports, but that doesn't
excuse us from having to support 4.4.

Ben.

[Antoine Beaupré]

On 2018-12-03 20:40:08, Ben Hutchings wrote:

[...]

Post by Ben Hutchings
I don't see this as an acceptable option for LTS. We could maybe add a
xen-4.8 package if it was popular in jessie-backports, but that doesn't
excuse us from having to support 4.4.

As I was repeatedly told during my work on Enigmail / GnuPG, I will
myself remind us all that jessie-backports is unfortunately not an
option anymore. :)

A.

[Ben Hutchings]

Post by Antoine Beaupré
[...]

Post by Ben Hutchings
I don't see this as an acceptable option for LTS. We could maybe add a
xen-4.8 package if it was popular in jessie-backports, but that doesn't
excuse us from having to support 4.4.

As I was repeatedly told during my work on Enigmail / GnuPG, I will
myself remind us all that jessie-backports is unfortunately not an
option anymore. :)

I know; I meant that if xen was a popular package in jessie-backports,
then the newer version could be considered as an addition to jessie—as
I've done with linux-4.9.

But in fact xen isn't in jessie-backports at all, so there's no reason
to think many jessie users have a newer version of it.

Ben.

[Holger Levsen]

Hi Peter and everyone,

first of all, thank you all for contributing to this thread!

Post by Ben Hutchings

Post by Peter Dreuw
If so, the other fixes are probably not to much work. But implementing
BTI fixes is a long and unknown road. I cannot give any reliable numbers
how much work that would be. But anybody can estimate that this will be
much more than a few days to get done. There might be a shortcut for
some patches by back porting independent code chunks like I did with the
grant table code for Xen 4.1 (Wheezy) but for now, I can't oversee all
of this in total yet and I doubt that there will be a great shortcut to
be found.

Having spent several days on similar backports for Linux 3.2 and 3.16,
I recognise the likely difficulty and complexity of the task and I
think it still needs to be done.

yes, we should fix what's (sensibly) possible to fix in xen 4.4.

So Peter, please go ahead and backport as much as you can, while updating
us (me or this list) on estimates as you get a better understanding of the work
required.

I assume it might also be a good idea if'd summarize the state
of the various (CVE) issues in NOTEs in data/dla-needed.txt in
security-tracker.git so that it's clearly visible in one location what
the status of backporting these fixes is. That information is also in
the mails of this thread, but that's not easy to find.

You can safely spend up to 4 or 5 (8h) days on this as we have some
backlog of undispatched hours accumulated and this is a good use for that.

(in related news, if you know someone who'd be interested to work on
LTS, please tell them to contact me.)

Post by Ben Hutchings
(But for future releases we do seriously need to consider whether Xen
should be covered by LTS, given the amount of work needed.)

can we discuss this now or should we postpone this to the beginning of
the Stretch LTS circle?

Post by Ben Hutchings

Post by Peter Dreuw
Another option would be backporting the Xen
4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10 (and following) package from
Stretch to Jessie. This could be done including testing within a few
hours, maybe a little more than a working day or less if we abandon Xen
4.4.

I don't see this as an acceptable option for LTS. We could maybe add a
xen-4.8 package if it was popular in jessie-backports, but that doesn't
excuse us from having to support 4.4.

agreed.