mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v2] media: uvcvideo: Fix pixel format change for Elgato Cam Link 4K
@ 2021-04-28 23:03 Benjamin Drung
  0 siblings, 0 replies; 7+ messages in thread
From: Benjamin Drung @ 2021-04-28 23:03 UTC (permalink / raw)
  To: Laurent Pinchart, Mauro Carvalho Chehab, linux-media, linux-kernel
  Cc: Adam Goode, Benjamin Drung, stable

The Elgato Cam Link 4K HDMI video capture card reports to support three
different pixel formats, where the first format depends on the connected
HDMI device.

```
$ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'NV12' (Y/CbCr 4:2:0)
		Size: Discrete 3840x2160
			Interval: Discrete 0.033s (29.970 fps)
	[1]: 'NV12' (Y/CbCr 4:2:0)
		Size: Discrete 3840x2160
			Interval: Discrete 0.033s (29.970 fps)
	[2]: 'YU12' (Planar YUV 4:2:0)
		Size: Discrete 3840x2160
			Interval: Discrete 0.033s (29.970 fps)
```

Changing the pixel format to anything besides the first pixel format
does not work:

```
v4l2-ctl -d /dev/video0 --try-fmt-video pixelformat=YU12
Format Video Capture:
	Width/Height      : 3840/2160
	Pixel Format      : 'NV12' (Y/CbCr 4:2:0)
	Field             : None
	Bytes per Line    : 3840
	Size Image        : 12441600
	Colorspace        : sRGB
	Transfer Function : Rec. 709
	YCbCr/HSV Encoding: Rec. 709
	Quantization      : Default (maps to Limited Range)
	Flags             :
```

User space applications like VLC might show an error message on the
terminal in that case:

```
libv4l2: error set_fmt gave us a different result than try_fmt!
```

Depending on the error handling of the user space applications, they
might display a distorted video, because they use the wrong pixel format
for decoding the stream.

The Elgato Cam Link 4K responds to the USB video probe
VS_PROBE_CONTROL/VS_COMMIT_CONTROL with a malformed data structure: The
second byte contains bFormatIndex (instead of being the second byte of
bmHint). The first byte is always zero. The third byte is always 1.

The firmware bug was reported to Elgato on 2020-12-01 and it was
forwarded by the support team to the developers as feature request.
There is no firmware update available since then. The latest firmware
for Elgato Cam Link 4K as of 2021-03-23 has MCU 20.02.19 and FPGA 67.

Therefore add a quirk to correct the malformed data structure.

The quirk was successfully tested with VLC, OBS, and Chromium using
different pixel formats (YUYV, NV12, YU12), resolutions (3840x2160,
1920x1080), and frame rates (29.970 and 59.940 fps).

Cc: stable@vger.kernel.org
Signed-off-by: Benjamin Drung <bdrung@posteo.de>
---
 drivers/media/usb/uvc/uvc_driver.c | 13 +++++++++++++
 drivers/media/usb/uvc/uvc_video.c  | 21 +++++++++++++++++++++
 drivers/media/usb/uvc/uvcvideo.h   |  1 +
 3 files changed, 35 insertions(+)

I am resending this patch since I got no response in the last three
weeks.

diff --git a/drivers/media/usb/uvc/uvc_driver.c b/drivers/media/usb/uvc/uvc_driver.c
index 9a791d8ef200..6ce58950d78b 100644
--- a/drivers/media/usb/uvc/uvc_driver.c
+++ b/drivers/media/usb/uvc/uvc_driver.c
@@ -3164,6 +3164,19 @@ static const struct usb_device_id uvc_ids[] = {
 	  .bInterfaceSubClass	= 1,
 	  .bInterfaceProtocol	= 0,
 	  .driver_info		= UVC_INFO_META(V4L2_META_FMT_D4XX) },
+	/*
+	 * Elgato Cam Link 4K
+	 * Latest firmware as of 2021-03-23 needs this quirk.
+	 * MCU: 20.02.19, FPGA: 67
+	 */
+	{ .match_flags		= USB_DEVICE_ID_MATCH_DEVICE
+				| USB_DEVICE_ID_MATCH_INT_INFO,
+	  .idVendor		= 0x0fd9,
+	  .idProduct		= 0x0066,
+	  .bInterfaceClass	= USB_CLASS_VIDEO,
+	  .bInterfaceSubClass	= 1,
+	  .bInterfaceProtocol	= 0,
+	  .driver_info		= UVC_INFO_QUIRK(UVC_QUIRK_FIX_FORMAT_INDEX) },
 	/* Generic USB Video Class */
 	{ USB_INTERFACE_INFO(USB_CLASS_VIDEO, 1, UVC_PC_PROTOCOL_UNDEFINED) },
 	{ USB_INTERFACE_INFO(USB_CLASS_VIDEO, 1, UVC_PC_PROTOCOL_15) },
diff --git a/drivers/media/usb/uvc/uvc_video.c b/drivers/media/usb/uvc/uvc_video.c
index f2f565281e63..06a538d1008b 100644
--- a/drivers/media/usb/uvc/uvc_video.c
+++ b/drivers/media/usb/uvc/uvc_video.c
@@ -128,6 +128,27 @@ static void uvc_fixup_video_ctrl(struct uvc_streaming *stream,
 	struct uvc_frame *frame = NULL;
 	unsigned int i;
 
+	/*
+	 * The response of the Elgato Cam Link 4K is incorrect: The second byte
+	 * contains bFormatIndex (instead of being the second byte of bmHint).
+	 * The first byte is always zero. The third byte is always 1.
+	 *
+	 * The UVC 1.5 class specification defines the first five bits in the
+	 * bmHint bitfield. The remaining bits are reserved and should be zero.
+	 * Therefore a valid bmHint will be less than 32.
+	 */
+	if (stream->dev->quirks & UVC_QUIRK_FIX_FORMAT_INDEX && ctrl->bmHint > 255) {
+		__u8 corrected_format_index;
+
+		corrected_format_index = ctrl->bmHint >> 8;
+		uvc_dbg(stream->dev, CONTROL,
+			"Correct USB video probe response from {bmHint: 0x%04x, bFormatIndex: 0x%02x} to {bmHint: 0x%04x, bFormatIndex: 0x%02x}.\n",
+			ctrl->bmHint, ctrl->bFormatIndex,
+			ctrl->bFormatIndex, corrected_format_index);
+		ctrl->bmHint = ctrl->bFormatIndex;
+		ctrl->bFormatIndex = corrected_format_index;
+	}
+
 	for (i = 0; i < stream->nformats; ++i) {
 		if (stream->format[i].index == ctrl->bFormatIndex) {
 			format = &stream->format[i];
diff --git a/drivers/media/usb/uvc/uvcvideo.h b/drivers/media/usb/uvc/uvcvideo.h
index 97df5ecd66c9..bf401d5ba27d 100644
--- a/drivers/media/usb/uvc/uvcvideo.h
+++ b/drivers/media/usb/uvc/uvcvideo.h
@@ -209,6 +209,7 @@
 #define UVC_QUIRK_RESTORE_CTRLS_ON_INIT	0x00000400
 #define UVC_QUIRK_FORCE_Y8		0x00000800
 #define UVC_QUIRK_FORCE_BPP		0x00001000
+#define UVC_QUIRK_FIX_FORMAT_INDEX	0x00002000
 
 /* Format flags */
 #define UVC_FMT_FLAG_COMPRESSED		0x00000001
-- 
2.27.0


^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: PROBLEM: Broken pixel format for Elgato Cam Link 4K
@ 2020-11-23 15:02 Adam Goode
  2021-04-06 18:52 ` [PATCH v2] media: uvcvideo: Fix pixel format change " Benjamin Drung
  2021-06-04 17:19 ` Benjamin Drung
  0 siblings, 2 replies; 7+ messages in thread
From: Adam Goode @ 2020-11-23 15:02 UTC (permalink / raw)
  To: Benjamin Drung
  Cc: Laurent Pinchart, Mauro Carvalho Chehab, linux-media, linux-kernel

On Sun, Nov 22, 2020 at 2:34 PM Benjamin Drung
<benjamin.drung@cloud.ionos.com> wrote:
>
> Am Freitag, den 20.11.2020, 16:45 -0500 schrieb Adam Goode:
> > On Fri, Nov 20, 2020 at 1:52 PM Benjamin Drung
> > <benjamin.drung@cloud.ionos.com> wrote:
> > >
> > > Hi,
> > >
> > > I own an Elgato Cam Link 4K which is a very popular USB HDMI
> > > capture
> > > device (number one capture card by click rates on Geizhals [1]).
> > > The
> > > problem is that the video feed is distorted when using the
> > > /dev/videoX
> > > device in the browser (tested on Firefox and Chromium) for video
> > > conferencing (tested with Jitsi Meet and Google Meet). The same
> > > distortion is present when opening `v4l2:///dev/video0` with VLC.
> > >
> > > The Elgato Cam Link 4K reports to have three different pixel
> > > formats:
> > >
> > > ```
> > > $ v4l2-ctl -d /dev/video0 --list-formats-ext
> > > ioctl: VIDIOC_ENUM_FMT
> > >         Type: Video Capture
> > >
> > >         [0]: 'NV12' (Y/CbCr 4:2:0)
> > >                 Size: Discrete 3840x2160
> > >                         Interval: Discrete 0.040s (25.000 fps)
> > >         [1]: 'NV12' (Y/CbCr 4:2:0)
> > >                 Size: Discrete 3840x2160
> > >                         Interval: Discrete 0.040s (25.000 fps)
> > >         [2]: 'YU12' (Planar YUV 4:2:0)
> > >                 Size: Discrete 3840x2160
> > >                         Interval: Discrete 0.040s (25.000 fps)
> > > ```
> > >
> > > When specifying the video format 'YU12' to VLC, the video is
> > > distorted
> > > the same way as using the default video format. When specifying
> > > 'NV12'
> > > to VLC, the video feed is displayed correctly:
> > >
> > > ```
> > > vlc v4l2:///dev/video0 --v4l2-chroma=NV12
> > > ```
> > >
> > > In OBS, the video feed is always displayed correctly. All video
> > > formats
> > > 'Y/CbCr 4:2:0', 'Planar YUV 4:2:0', 'BGR3 (Emulated)', and 'YV12
> > > (Emulated)' combined with the color ranges 'Default', 'Partial',
> > > and
> > > 'Full' produce the same correct output.
> > >
> > > With Linux >= 5.9 this behavior in OBS changes: The video format
> > > 'Y/CbCr 4:2:0' displays the video correctly. Switching to 'Planar
> > > YUV
> > > 4:2:0', 'BGR3 (Emulated)', or 'YV12 (Emulated)' shows the video
> > > distorted and OBS shows this error message:
> > >
> > > ```
> > > info: v4l2-input: Pixelformat: NV12
> > > [...]
> > > libv4l2: error set_fmt gave us a different result than try_fmt!
> > > info: v4l2-input: Resolution: 3840x2160
> > > info: v4l2-input: Pixelformat: NV12
> > > ```
> > >
> > > Changing the video format back does not have an effect until I also
> > > change the color range (does seem to be relevant what to select
> > > there).
> > >
> > > Workaround
> > > ----------
> > >
> > > You can create a v4l2loopback device and use ffmpeg to stream from
> > > the
> > > Cam Link 4K to the loopback device:
> > >
> > > ```
> > > ffmpeg -f v4l2 -input_format yuv420p -video_size 3840x2160 \
> > >   -i "$camlink" -codec copy -f v4l2 "$loopdev"
> > > ```
> > >
> > > This workaround works, but is cumbersome and burns CPU cycles.
> > >
> > > Other reports
> > > -------------
> > >
> > > Searching the web for "Cam Link 4K Linux" reveals many similar
> > > reports
> > > like this. Noteworthy is blog post [3] from Mike Walters who
> > > patched
> > > the Cam Link 4K firmware to report the correct video format. I am
> > > willing to debug this issue and do test, but I don't want to flash
> > > the
> > > firmware to not break the warrenty (bisides I lack the hardware for
> > > flashing).
> > >
> > > Environment
> > > -----------
> > >
> > > This problem is present in Ubuntu 20.04 with linux 5.4.0-54.60 and
> > > Ubuntu 20.10 with linux 5.8.0-29.31. I also tested the mainline
> > > kernels
> > > builds 5.9.8-050908.202011101634 and 5.10.0-051000rc4.202011152030
> > > from
> > > Ubuntu [2].
> > >
> > > The Cam Link 4K shows follow entries in dmesg:
> > >
> > > ```
> > > [    1.575753] usb 2-3: new SuperSpeed Gen 1 USB device number 2
> > > using xhci_hcd
> > > [    1.596664] usb 2-3: LPM exit latency is zeroed, disabling LPM.
> > > [    1.598557] usb 2-3: New USB device found, idVendor=0fd9,
> > > idProduct=0066, bcdDevice= 0.00
> > > [    1.598558] usb 2-3: New USB device strings: Mfr=1, Product=2,
> > > SerialNumber=4
> > > [    1.598559] usb 2-3: Product: Cam Link 4K
> > > [    1.598560] usb 2-3: Manufacturer: Elgato
> > > ```
> > >
> > > I have another problems with 5.9.8-050908.202011101634 and 5.10.0-
> > > 051000rc4.202011152030: Chromium fail to access the video device of
> > > Cam
> > > Link 4K and the notebook integrated webcam has a too low
> > > brightness.
> > >
> > > [1] https://geizhals.de/?cat=vidext
> > > [2] https://kernel.ubuntu.com/~kernel-ppa/mainline/
> > > [3]
> > > https://assortedhackery.com/patching-cam-link-to-play-nicer-on-linux/
> > >
> > > --
> > > Benjamin Drung
> > > Debian & Ubuntu Developer
> > >
> >
> >
> > Hi,
> >
> > I am running on Fedora 32 which has the fix I wrote for the buggy
> > elgato firmware. The bug in the firmware makes it impossible to
> > properly select a non-0 pixel format when following the UVC
> > negotiation protocol. This is because the firmware returns the pixel
> > format in the wrong byte of the packet. The driver was following the
> > UVC protocol but did not send the pixel format back to the v4l2
> > subsystem. It does that now.
> >
> > I'm not surprised that other bugs are emerging now. Ultimately the
> > firmware is buggy and announces pixel formats that it then rejects.
> > If
> > I flip through the settings in OBS, I do manage to wedge the
> > interface. But most of the programs I've seen that use v4l2 are buggy
> > in this way. A reliable one is the qv4l2 test program. I've also had
> > no problems with Chromium.
> >
> > That reverse engineering is interesting! But I think it hides the
> > real
> > problem, where the pixel format negotiation on the firmware writes
> > into the wrong byte of the packet.
>
> Hi Adam,
>
> you are talking about commit ec2c23f628802317f73fab5255cc62a776bc7930
> and 8a652a17e3c005dcdae31b6c8fdf14382a29cbbe that are part of
> v5.10-rc1?
>
> What do you suggest to fix the issue? Should I contact Elgato
> requesting a firmware update? Can the kernel ship a quirk for the Cam
> Link 4K to workaround the firmware bug? I would be happy to try out
> patches.
>

I would file a ticket. I did so some time ago, but there have been no
firmware updates for many months.

> I tested the qv4l2 with Linux 5.10.0-051000rc4.202011152030. When using
> the default settings, it displays the video correctly, but with a
> vertical green one-pixel-wide line. The terminal prints "error" four
> times. Sadly there is no debug mode to figure out where that error
> comes from.

I'm not sure where this line is coming from. Can you post a link to a
screenshot or capture? I have never encountered this issue. Can you
reproduce on a Windows or Mac computer?

>
> Selecting "YU12 (Planar YUV 4:2:0)" print the following error message
> and switches back to the previous format:
>
> libv4l2: error set_fmt gave us a different result than try_fmt!
>

Yes, this is the kernel coping with the buggy firmware. These errors
are expected. It is also expected that YU12 does not work (the device
advertises it, then rejects it when selected). I believe Windows and
Mac (you can use OBS to try) do the same thing.

A quirk is possible (I hacked something up with libuvc when I was
debugging this) but it's not something I have time to look at right
now.


Adam

> --
> Benjamin Drung
> Debian & Ubuntu Developer
>

^ permalink raw reply	[flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-06-05 23:10 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-28 23:03 [PATCH v2] media: uvcvideo: Fix pixel format change for Elgato Cam Link 4K Benjamin Drung
  -- strict thread matches above, loose matches on Subject: below --
2020-11-23 15:02 PROBLEM: Broken pixel format " Adam Goode
2021-04-06 18:52 ` [PATCH v2] media: uvcvideo: Fix pixel format change " Benjamin Drung
2021-06-04 17:19 ` Benjamin Drung
2021-06-04 22:21   ` Laurent Pinchart
2021-06-05  8:19     ` Benjamin Drung
2021-06-05 21:51       ` Laurent Pinchart
2021-06-05 23:10         ` Benjamin Drung

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®