Re: designing fast, streaming disk i/o with mmap: help wanted

From: Roger Larsson (roger.larsson@norran.net)
Date: Fri Mar 31 2000 - 20:22:24 EST

  • Next message: Ricky Beam: "Re: Clarification & more info (was Re: Running 2.3.99-pre3 headless doesn't create tty<n> devices???"

    Assuming the worst case.

    all files are fragmented and spread all over the disk.

    then you would not like to use the normal read ahead (n-blocks)
    but rather read ahead (only continuos blocks, at most n-blocks)

    The more fragmented you are the more likely you will read all
    the continuos blocks. And will not attempt to continue reading at
    a completely different location.

    Assume layout (letter=track, number=block):

    layout: A1 A2 B1 B2 A3 C1 B3 C2 C3
    read: 1 1+
                     2 2+
                              3
    -- first block from all channels read ---
    -- all second blocks but C read too ---
                                    4 4+
    -- need third block, C3 read already ---
                           5
                                 6

    Assuming you always try to read tree blocks
    you get a worse result:

    layout: A1 A2 B1 B2 A3 C1 B3 C2 C3
               1 1+
                           2
                     3 3+
                                 4
                              5
    --- first blocks from all channels read ---
                                    6 6+
    --- all blocks read, but are they still around? ---

    Would it be viable to limit read ahead to continuos blocks?

    /RogerL

    Paul Barton-Davis wrote:
    >
    > >By the way, BeOS streaming movies is ridiculously impressive.
    >
    > Streaming playback of movies is nothing compared to the task of
    > multichannel recording. You have to simultaneously read/write N (with
    > N typically >= 24) files at the same time with up to 9MB/sec in each
    > direction (32bit/96kHz recording) per file, to allow for instantaneous
    > "punch out" (transition from recording to playback). Doing this from a
    > single disk is possible but its not trivial. Obviously, any level of
    > RAID technology would be good.
    >
    > because of the number of files, unless you have N disks available, you
    > are almost always going to have to face seek deadlines because you
    > need to read from two different files within an extremely short space
    > of time, and so there is no filesystem design that will actually solve
    > the problem for you. needless to say, some filesystems will make the
    > job easier, as long as you make certain assumptions that i don't want
    > to make :)
    >
    > interleaving the blocks of each channel's files, for example, is one
    > trick that can do wonders. until, that is, you decide to change the
    > block size you work with during playback/recording, and then it
    > becomes worse than it was with just a regular old unfragmented ext2
    > fs. etc.
    >
    > almost all the multimedia filesystems that i've seen have been focused
    > on audio/video with relatively low numbers of "parallel" channels
    > and/or have assumed interleaved data. this is of little use for HDR
    > applications, because of the overhead of {de,re}interleaving whenever
    > you need to access the data for a single channel. i was originally
    > intrigued by the Clockwise FS, for example, until i found out that its
    > just a layer on top of ext2.
    >
    > ultimately, i would rather assume that the underlying fs is not going
    > to help me, do the best job i can with the existing tools, and then if
    > an fs design is available to make things even faster, all the better.
    >
    > --p
    >
    > -
    > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    > the body of a message to majordomo@vger.rutgers.edu
    > Please read the FAQ at http://www.tux.org/lkml/

    --
    Home page:
      http://www.norran.net/nra02596/
    

    - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/



    This archive was generated by hypermail 2b29 : Fri Mar 31 2000 - 21:33:12 EST