UDF: buggy? libdvdread vs. udf fs driver

UDF: buggy? libdvdread vs. udf fs driver

[Herbert Rosmanith]

good afternoon,

background:
==========

  I'm trying to burn a video-dvd using udf. for testing, I used
  (1) a loopback image (2) a DVD-Rewriteable (3) a write-only dvd.
  all tests showed the same behaviour: "it does not work" ;-)

  using udf, I'm not able to play video-dvds, neither (1) via software
  ("ogle") nor (2) via a dvd-player connected to a TV.

  on the other hand, using iso9660 (via mkisofs -dvd-video) , ogle and
  the dvd-player work and display my small testmovie.

what exactly does not work:
==========================

  the dvd-player always complains about "no disc inserted", just to
  mention, anyway, the dvd-player is not a practicable debugging device ;-)

  the better debugging source is libdvdread. ogle complains that it
  cannot find "VIDEO_TS.IFO", allthough it _is_ there. I used the "printf"
  -debugger on libdvdread and found that it behaves differently when
  using a commercial DVD, opposed to the loopback-image I formatted with
  mkudffs.

how to reproduce the "bug":
==========================

  well, I'm not 100% sure if the problem exists because I don't understand
  how to use all these software (with the right options, in particular when
  using mkudffs) or if there's really a bug.

  (1) I created an udf-formated loopback image without extended file entries.
     "Why no extened file entries?", I hear you ask. Answer: because libdvdread
     doesnt like them, I'm afraid. Have a look in
     libdvdread/dvdread/dvd_udf.c, look for "UDFMapICB", somewhere around
     line 500. The do { } while loop checks for TagID 261 and ignores all
     other Tags. TagID 261 means "TAG_IDENT_FE", and when using "EFE"s, then
     UDFMapICB() will always see TagID = 0x10a (TAG_IDENT_EFE).
     Since I'm not really sure if the problem is related to FE vs. EFE, I
     used a filesystem with EFEs one time, and one without another time: both
     file systems showed the same behaviour (="VIDEO_ITS.IFO not found")

     ok, here we go:

        bash-2.05# dd if=/dev/zero of=24M bs=1M count=24
        24+0 records in
        24+0 records out

        bash-2.05# mkudffs 24M
        start=0, blocks=16, type=RESERVED 
        start=16, blocks=3, type=VRS 
        start=19, blocks=237, type=USPACE 
        start=256, blocks=1, type=ANCHOR 
        start=257, blocks=16, type=PVDS 
        start=273, blocks=1, type=LVID 
        start=274, blocks=11757, type=PSPACE 
        start=12031, blocks=1, type=ANCHOR 
        start=12032, blocks=239, type=USPACE 
        start=12271, blocks=16, type=RVDS 
        start=12287, blocks=1, type=ANCHOR 

    or:

        bash-2.05# mkudffs --noefe 24M

    I also used various other options like --bridgea --media-type etc.,
    but it didn't help.

  (2) next I created the dvd-structure using "dvdauthor":

        bash-2.05# mount -o loop 24M /mnt
        bash-2.05# dvdauthor -o /mnt -x foo.xml 
        DVDAuthor::dvdauthor, version 0.6.11.
        ...
        INFO: dvdauthor creating table of contents
        INFO: Scanning /mnt/VIDEO_TS/VTS_01_0.IFO
        ...

     "foo.xml" just contains:
        bash-2.05# cat foo.xml 
        <dvdauthor>
            <vmgm />
            <titleset>
                <titles>
                    <pgc>
                        <vob file="foo.mpg" />
                    </pgc>
                </titles>
            </titleset>
        </dvdauthor>

     "foo.mpg" is just a sequence of random pictures converted from /dev/urandom
     with mplex, mpeg2enc, et al.


  (3) to be sure the data has been written to the image, I unmount it (shouldn't
     be neccessary, right?), and then ogle will fail and say:

        bash-2.05# ogle 24M 
        libdvdread: Can't open file VIDEO_TS.IFO.
        ERROR[ogle_nav]: faild to read VIDEO_TS.IFO
        DVDSetDVDRoot:: Root not set

     but:
        bash-2.05# find /mnt -name VIDEO_TS.IFO
        /mnt/VIDEO_TS/VIDEO_TS.IFO
        bash-2.05# ls -li /mnt/VIDEO_TS/VIDEO_TS.IFO 
           1097 -rw-r--r--   1 root     root         6144 Jun  5 14:14
           /mnt/VIDEO_TS/VIDEO_TS.IFO

     in fact, the file *is* there. or, to be a bit more precise: libdvdread cannot 
     find the file, while the udf-fs driver does. of course it does - it should
     be able to find files it has created itself (via dvdauthor in (2)).

the printf debugger:
===================

  Since I'm still not used to using fullscreen-debuggers, I've added some
  printf()s to libdvdread/dvdread/udf.c and saw that a particular length-field
  is always zero when using the image, and non-zero using a commercial dvd.

  here's what I get:

  commercial DVD:
        # ogle /dev/hdc
        > UDFFindFile filename=/VIDEO_TS/VIDEO_TS.IFO
          UDFFindFile 841
        > UDFMapICB
        > UDFFileEntry
          ad->Length=0
          L_EA=132 L_AD=8
          before switch: ad->Length=0
          flags=0230
          after switch: ad->Length=292
        < UDFFileEntry, ad->Length=292

  howebrew loopback image:
        bash-2.05# ogle ~/src/24M 
        libdvdread: Using libdvdcss version 1.2.9 for DVD access
        > UDFFindFile filename=/VIDEO_TS/VIDEO_TS.IFO
          UDFFindFile 841
        > UDFMapICB
        > UDFFileEntry
          ad->Length=0
          L_EA=0 L_AD=188
          before switch: ad->Length=0
          flags=0003
          after switch: ad->Length=0
        < UDFFileEntry, ad->Length=0
          527: return 1
          872: File.length=0 token=VIDEO_TS
        > UDFScanDir (1) Dir.Length=0 (2) FileName=VIDEO_TS
          UDFScanDir:592
          enter while: Dir.Length=0
        X UDFFindFile 880

  "the X UDFFindFile" means that libdvdread did not find the file.
  you'll probably notice that Dir.Length remains zero always, which is the
  reason that UDFFindFile fails.

  one (big?) difference is that "L_EA=132 L_AD=8" in the first case
  and "L_EA=0 L_AD=188". I don't know if this should be that way. I've just
  downloaded ecma-167 and then decided to file this report without further
  looking into the problem.

conclusion:
==========

  the udf-fs-driver and libdvdread (and my dvd-player as well) seem to have a
  different idea on how to interpret udf. I can only guess why (= I have no
  proof).

  or is the source of the problem rather myself and I'm missing a crucial
  option?

kind regards,
h.rosmanith

Re: UDF: buggy? libdvdread vs. udf fs driver

[Herbert Rosmanith]

>   "the X UDFFindFile" means that libdvdread did not find the file.
>   you'll probably notice that Dir.Length remains zero always, which is the
>   reason that UDFFindFile fails.

ok, at least I managed to get a Dir.Length > 0 using long or short "ad"
instead of inicb -- whatever that means. "AD" like "allocation descriptor"?
long or short ADs, or ADs in the INICB? what exactly does that mean?

so:

        mkudffs --ad=long --noefe 24M
or:     mkudffs --ad=short --noefe 24M

will get me a step further, but UDFFindFile will now fail at some other place.

kind regards,
h.rosmanith

Re: UDF: buggy? libdvdread vs. udf fs driver

[Herbert Rosmanith]

> ok, at least I managed to get a Dir.Length > 0 using long or short "ad"
> instead of inicb -- whatever that means. "AD" like "allocation descriptor"?
> long or short ADs, or ADs in the INICB? what exactly does that mean?
> 
> so:
> 
>         mkudffs --ad=long --noefe 24M
> or:     mkudffs --ad=short --noefe 24M
> 
> will get me a step further, but UDFFindFile will now fail at some other place.

back again ...

I've made an observation which I think is somewhat strange ...

(1) make udf filesystem on loopbackimage
    bash-2.05# dd if=/dev/zero of=24M bs=1M count=24
    24+0 records in
    24+0 records out
    bash-2.05# mkudffs --vid="The Entropy Movie" --udfrev=0x0102 --ad=short --noefe 24M
    start=0, blocks=16, type=RESERVED 
    start=16, blocks=3, type=VRS 
    start=19, blocks=237, type=USPACE 
    start=256, blocks=1, type=ANCHOR 
    start=257, blocks=16, type=PVDS 
    start=273, blocks=1, type=LVID 
    start=274, blocks=11757, type=PSPACE 
    start=12031, blocks=1, type=ANCHOR 
    start=12032, blocks=239, type=USPACE 
    start=12271, blocks=16, type=RVDS 
    start=12287, blocks=1, type=ANCHOR 
    bash-2.05# mount -o loop 24M /mnt
    bash-2.05# dmesg
    UDF-fs INFO UDF 0.9.8.1 (2004/29/09) Mounting volume 'The Entropy Movie', timestamp 2006/06/05 19:02 (1078)
    bash-2.05# df
    Filesystem           1K-blocks      Used Available Use% Mounted on
    /dev/hda3              1011960    700312    260240  73% /
    /dev/hda4            152640480  60730108  84156600  42% /data
    /data/root/src/24M       23514        12     23502   1% /mnt

(2) /mnt contains:
    bash-2.05# find /mnt
    /mnt
    /mnt/lost+found

(3) libdvdread will now ...
    bash-2.05# ogle 24M 
    libdvdread: Using libdvdcss version 1.2.9 for DVD access
    > UDFFindFile filename=/VIDEO_TS/VIDEO_TS.IFO
      UDFFindFile:856
    > UDFMapICB
    > UDFFileEntry
      ad->Length=0
      L_EA=0 L_AD=8
      before switch: ad->Length=0
      flags=0000
      flags=>UDFShortAD
      after switch: ad->Length=92
    < UDFFileEntry, ad->Length=92
    < UDFMapICB:530, return 1(ok)
      887: File.length=92 token=VIDEO_TS
    > UDFScanDir (1) Dir.Length=92 (2) FileName=VIDEO_TS
      UDFScanDir:595
      enter while: Dir.Length=92
      TagID=257 101
      filename=
      TagID=257 101
      filename=lost+found
    X UDFScanDir:638

    .... will now stop after arriving at the lost+found directory.
    please noteice "ad->Length=92".

(4) now, create dvd-files:
    bash-2.05# mount -o loop 24M /mnt
    bash-2.05# dvdauthor -o /mnt -x foo.xml 
    DVDAuthor::dvdauthor, version 0.6.11.
    ...
    INFO: dvdauthor creating table of contents
    INFO: Scanning /mnt/VIDEO_TS/VTS_01_0.IFO

(5) the image now contains:
    bash-2.05# find /mnt
    /mnt
    /mnt/lost+found
    /mnt/VIDEO_TS
    /mnt/VIDEO_TS/VTS_01_1.VOB
    /mnt/VIDEO_TS/VTS_01_0.IFO
    /mnt/VIDEO_TS/VTS_01_0.BUP
    /mnt/VIDEO_TS/VIDEO_TS.IFO
    /mnt/VIDEO_TS/VIDEO_TS.BUP
    /mnt/AUDIO_TS

(6) but this is what libdvdread sees:
    bash-2.05# umount  /mnt
    bash-2.05# ogle 24M 
    libdvdread: Using libdvdcss version 1.2.9 for DVD access
    > UDFFindFile filename=/VIDEO_TS/VIDEO_TS.IFO
      UDFFindFile:856
    > UDFMapICB
    > UDFFileEntry
      ad->Length=0
      L_EA=0 L_AD=8
      before switch: ad->Length=0
      flags=0000
      flags=>UDFShortAD
      after switch: ad->Length=92
    < UDFFileEntry, ad->Length=92
    < UDFMapICB:530, return 1(ok)
      887: File.length=92 token=VIDEO_TS
    > UDFScanDir (1) Dir.Length=92 (2) FileName=VIDEO_TS
      UDFScanDir:595
      enter while: Dir.Length=92
      TagID=257 101
      filename=
      TagID=257 101
      filename=lost+found
    X UDFScanDir:638

    strange - Dir.Length is still 92, same as above in (3), and
    libdvdread will stop after reading "lost+found".

(7) remove lost+found and try again:
    bash-2.05# mount -o loop 24M /mnt
    bash-2.05# rmdir /mnt/lost+found/
    bash-2.05# find /mnt  
    /mnt
    /mnt/VIDEO_TS
    /mnt/VIDEO_TS/VTS_01_1.VOB
    /mnt/VIDEO_TS/VTS_01_0.IFO
    /mnt/VIDEO_TS/VTS_01_0.BUP
    /mnt/VIDEO_TS/VIDEO_TS.IFO
    /mnt/VIDEO_TS/VIDEO_TS.BUP
    /mnt/AUDIO_TS
    bash-2.05# umount /mnt
    bash-2.05# ogle 24M 
    libdvdread: Using libdvdcss version 1.2.9 for DVD access
    > UDFFindFile filename=/VIDEO_TS/VIDEO_TS.IFO
      UDFFindFile:856
    > UDFMapICB
    > UDFFileEntry
      ad->Length=0
      L_EA=0 L_AD=8
      before switch: ad->Length=0
      flags=0000
      flags=>UDFShortAD
      after switch: ad->Length=92
    < UDFFileEntry, ad->Length=92
    < UDFMapICB:530, return 1(ok)
      887: File.length=92 token=VIDEO_TS
    > UDFScanDir (1) Dir.Length=92 (2) FileName=VIDEO_TS
      UDFScanDir:595
      enter while: Dir.Length=92
      TagID=257 101
      filename=
      TagID=257 101
      filename=lost+found
    X UDFScanDir:638

    what? it still stops after "lost+found", which just has been removed?

(8) am I using the wrong image?
    bash-2.05# pwd
    /data/root/src
    bash-2.05# ls -l 24M 
    -rw-r--r--   1 root     root     25165824 Jun  5 19:02 24M
    bash-2.05# df
    Filesystem           1K-blocks      Used Available Use% Mounted on
    /dev/hda3              1011960    698656    261896  73% /
    /dev/hda4            152640480  60730116  84156592  42% /data
    bash-2.05# mount -o loop 24M /mnt
    bash-2.05# df
    Filesystem           1K-blocks      Used Available Use% Mounted on
    /dev/hda3              1011960    698656    261896  73% /
    /dev/hda4            152640480  60730116  84156592  42% /data
    /data/root/src/24M       23514      1660     21854   8% /mnt

    no. the image libdvd sees is the same that is mounted and that
    dvdauthor is operating upon.

what's wrong here?

kind regards,
h.rosmanith

Re: UDF: buggy? libdvdread vs. udf fs driver

[Herbert Rosmanith]

>     > UDFScanDir (1) Dir.Length=92 (2) FileName=VIDEO_TS
>       UDFScanDir:595
>       enter while: Dir.Length=92
>       TagID=257 101
>       filename=
>       TagID=257 101
>       filename=lost+found
>     X UDFScanDir:638
> 
>     what? it still stops after "lost+found", which just has been removed?

okidok ... having seen this, I had the idea of modyfing mkudffs just
to see what's going to happen.

since "lost+found" is so persistent and remains allthough having been removed
before, I changed "udftools-1.0.0b3/mkudffs/mkudffs.c" not to create
"lost+found", but "VIDEO_TS" instead:

   642          //desc = udf_mkdir(disc, pspace, "\x08" "lost+found", 11, offset+1, desc);
   643          desc = udf_mkdir(disc, pspace, "\x08" "VIDEO_TS", 9, offset+1, desc);

good.

now:
    bash-2.05# mkudffs --vid="The Entropy Movie" --udfrev=0x0102 --ad=short --noefe 24M
    start=0, blocks=16, type=RESERVED 
    start=16, blocks=3, type=VRS 
    start=19, blocks=237, type=USPACE 
    start=256, blocks=1, type=ANCHOR 
    start=257, blocks=16, type=PVDS 
    start=273, blocks=1, type=LVID 
    start=274, blocks=11757, type=PSPACE 
    start=12031, blocks=1, type=ANCHOR 
    start=12032, blocks=239, type=USPACE 
    start=12271, blocks=16, type=RVDS 
    start=12287, blocks=1, type=ANCHOR 
    bash-2.05# mount -o loop 24M /mnt
    bash-2.05# find /mnt
    /mnt
    /mnt/VIDEO_TS

good, that's what I wanted. now create dvd:

    bash-2.05# dvdauthor -o /mnt -x foo.xml 
    DVDAuthor::dvdauthor, version 0.6.11.
    Build options: gnugetopt freetype
    Send bugs to <dvdauthor-users@lists.sourceforge.net>
    
    INFO: dvdauthor creating VTS
    STAT: Picking VTS 01
    STAT: Processing foo.mpg...
    ...
    INFO: dvdauthor creating table of contents
    INFO: Scanning /mnt/VIDEO_TS/VTS_01_0.IFO

all files in place?
    bash-2.05# find /mnt
    /mnt
    /mnt/VIDEO_TS
    /mnt/VIDEO_TS/VTS_01_1.VOB
    /mnt/VIDEO_TS/VTS_01_0.IFO
    /mnt/VIDEO_TS/VTS_01_0.BUP
    /mnt/VIDEO_TS/VIDEO_TS.IFO
    /mnt/VIDEO_TS/VIDEO_TS.BUP
    /mnt/AUDIO_TS

indeed.

now for libdvdread:
    bash-2.05# umount /mnt
    bash-2.05# ogle 24M 
    libdvdread: Using libdvdcss version 1.2.9 for DVD access
    > UDFFindFile filename=/VIDEO_TS/VIDEO_TS.IFO
      UDFFindFile:856
    > UDFMapICB
    > UDFFileEntry
      ad->Length=0
      L_EA=0 L_AD=8
      before switch: ad->Length=0
      flags=0000
      flags=>UDFShortAD
      after switch: ad->Length=88
    < UDFFileEntry, ad->Length=88
    < UDFMapICB:530, return 1(ok)
      887: File.length=88 token=VIDEO_TS
    > UDFScanDir (1) Dir.Length=88 (2) FileName=VIDEO_TS
      UDFScanDir:595
      enter while: Dir.Length=88
      TagID=257 101
      filename=
      TagID=257 101
      filename=VIDEO_TS
      ^^^^^^^^^^^^^^^^^

libdvdread now finds VIDEO_TS, which it failed to before.

but:

    > UDFMapICB
    > UDFFileEntry
      ad->Length=0
      L_EA=0 L_AD=8
      before switch: ad->Length=0
      flags=0000
      flags=>UDFShortAD
      after switch: ad->Length=40
    < UDFFileEntry, ad->Length=40
    < UDFMapICB:530, return 1(ok)
      887: File.length=40 token=VIDEO_TS.IFO
    > UDFScanDir (1) Dir.Length=40 (2) FileName=VIDEO_TS.IFO
      UDFScanDir:595
      enter while: Dir.Length=40
      TagID=257 101
      filename=
    > UDFMapICB
    < UDFMapICB:514, return 1(ok)
    X UDFScanDir:638

 ... it fails to find "VIDEO_TS.IFO", allthough:

    bash-2.05# mount -o loop 24M /mnt
    bash-2.05# ls -l /mnt/VIDEO_TS/VIDEO_TS.IFO 
    -rw-r--r--   1 root     root         6144 Jun  5 19:33 /mnt/VIDEO_TS/VIDEO_TS.IFO

... the udf-fs driver _does_ see that file.

kind regards,
h.rosmanith

Re: UDF: buggy? libdvdread vs. udf fs driver

[Herbert Rosmanith]

by the way, just to let you know ...

my attempt to create files via wrudf from the udftools
failed, too.

when operating on a diskimage, wrudf will exit with
"readPacket: read failed: bad address"

a closer look reveals that wrudf tries to read() into
a buffer-address of NIL.

yes, I'm using the latest udftools.

kind regards,
h.rosmanith

Re: UDF: buggy? libdvdread vs. udf fs driver

[Herbert Rosmanith]

> a closer look reveals that wrudf tries to read() into
> a buffer-address of NIL.

and the winner is .... 

initIO() !

it will malloc() a 64kb buffer. but only if medium==CDRW.
if the medium is a diskimage, the buffer-pointer will
remain 0x0000000 (or "NIL").

kind regards,
h.rosmanith

Re: UDF: buggy? libdvdread vs. udf fs driver

[Herbert Rosmanith]

> it will malloc() a 64kb buffer. but only if medium==CDRW.
> if the medium is a diskimage, the buffer-pointer will
> remain 0x0000000 (or "NIL").

oh, to be more precise: initIO() reads:
  if (device_type==DISK_IMAGE)
    return 0;

the malloc() loop is below the if statement.
I'll try to fix it w/o breaking other things :-/

kind regards,
h.rosmanith