1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
  2. Donation with Paypal!!!

    Go to your paypal account and send directly donation to [email protected]

    1 month - 10 $ - Standart VIP

    6 months - 20 $- Standart VIP

    1 year - 30 $- Standart VIP

    2 years - 50 $- Standart VIP

    Gold member for life - 150 $- Standart VIP

    High Vip (Standart VIP include) group please send PM or email to [email protected] for info

    After Donation please send email to [email protected]

  3. Donation Ways 2020


    Paysend
  4. Telegram
Dismiss Notice

Donation with Paypal!!!

Go to your paypal account and send directly donation to [email protected]

1 month - 10 $ - Standart VIP

6 months - 20 $- Standart VIP

1 year - 30 $- Standart VIP

2 years - 50 $- Standart VIP

Gold member for life - 150 $- Standart VIP

High Vip (Standart VIP include) group please send PM or email to [email protected] for info

After Donation please send email to [email protected]

Dismiss Notice
For open hidden message no need write thanks, thank etc. Enough is click to like button on right side of thread.

Write in a hugefile backward

Discussion in 'Delphi Programming' started by darkvadr, Apr 9, 2015.

  1. darkvadr
    Offline

    darkvadr DF Member

    Some years ago I was asked to write huge files backward.
    When I say huge, we are talking 400 GB and more (Terrabytes).
    Sometimes with bHIDE-THANKSs, somtimes just one
    Byte.

    So, let's try with a 50 GB file
    50 * 1024 * 1024 * 1024 = 53687091200

    The usual trick to create such a file is 

    procedure CreateBigFile
    var
      FS: TFileStream;
    begin
      FS := TFileStream.Create('C:\BIGFILE.TMP', fmCreate);
      try
        FS.Size := 53687091200; // 50 * 1024 * 1024 * 1024; // ça c'est 50Go informatique !
      finally
        FS.Free();
      end;

    By sending the pointer outside the file, the SetEndOfFile API is called
    and the file is created, with the good size.

    But heres come the problem...
    If you write from the begining of the file to the end,
    everything run smoothly.

    Try to write one octet at the end of the 
    file in order to write byte to byte backward.

    Oooops the mft is updated and now you have to effectively
    trace all the 50 GB the first round, all the 50 GB - 1,
    the second round and so on.

    At the old good time of dos, we could easely bypass that
    by using direct ATA commands to the hdd controler.

    Since NT kernel, you need a driver to access ring 0 via
    the Native API. I did write one for windows XP.
    And believe me it's a pain in the ass as you work
    in obscurity and the blue screen is the punishment
    for a bug. And I accidently descoverded that you can override
    some HDD firmware.

    Then vista , 7 ... Legacy drivers not allowed, driver not signed,
    new framework, new hdd... pffiiiiuuuu

    Back to delphi and Ring 3


    And a bit very badly or still undocumented secrets of the MFT
    you can pass via the standard ring 3 API an IOCTLCODE
    that have a direct impact in the way kerneland will handle
    your file. We ask the kernel to use the SPARSE argument to handle
    all operations on this particular file.

    Beside fragmentation and microsoft tricks, a standard file once
    created is let say like that :

    [1 BYTE OF DATA][50 - 2 GB of random unallocated seen as 0x00 mess][1 BYTE OF DATA]


    In sparse mode. you have chunks

    [1 BYTE OF DATA][1 BYTE OF DATA] and the mft take account of that.

    You can now write a huge file backward with quite few time loss comparing
    to forward mode.

    Coooool

    There's some warnings.
    The file is seen by the system as the size you choosed let say
    400 GB. BUT it effectively take no place. And here comes a flaw
    from microsoft. To accept or not the creation of the file,
    the kernel check the physical place of this creation.
    This physical place for an empty file of 400 GB
    if I remember well is something like 10 MB.

    You can make the system accept to create a file that is waaayyyy
    larger than you HDD. Let say 100 T.

    it will take no time as usual.

    BUT the system still unaware that your file is in sparse mode.
    Only your program knows.

    Let say, you want to delete this file using the trashbin.
    The trashbin is unable to erase in sparse mode, then the mft
    epically failed to recognize it's a sparse file 
    (but it have all the stuff to do it) and instead attempt to physically allocate the 

    reserved space without verifying the size,
    because after all...
    The writing has been aproved before.
    Why the deleting should be forbiden ? mmmmmm
    and to delete something, it must exists... If not, then create it.
    Kay?
    Genious inside. :p

    So beaware in your experiments with udocumented things.
    Could be a pain in the ass.

    procedure CreateBigFile;
    const
      NTFS_IOCTLCODE_SET_SPARSE=$900C4;//SET_SPARSE control code
    var
      h             : Thandle; //Handle on the file
      Filesize      : LARGE_INTEGER; // struct INT64
      Filename      : string;
      Buffer        : DWord; //proof of concept payload
      BytesReturned : DWORD; 
    begin
            Filename := 'C:\BIGFILE.TMP'; 
            Buffer   := &68656c6c; //hell in hex   
            //Old good create file 
            h := CreateFile(pchar(Filename),
                            GENERIC_WRITE,
                            FILE_SHARE_WRITE,
                            nil
                            CREATE_ALWAYS,0,0);
           //Send the SETSPARSE control code   
           DeviceIoControl(h, 
                           NTFS_IOCTLCODE_SET_SPARSE,
                           nil,
                           0,
                           nil,
                           0,
                           BytesReturned,
                           nil);//physical space is not prereserved anymore.
     
          Filesize.QuadPart := 53687091200; //Go for 50 GB
          SetFilePointer(h,
                         Filesize.LowPart,
                         @Filesize.HighPart,
                         FILE_BEGIN); 
          SetEndOfFile (h);//Set the pointer at the end of file, the file is now created
                           //In sparse mode  
          fileseek (h,-4,1);//move 4 BYTES backward
          FileWrite(h,buffer,4);//and write our 'hell' buffer 
          setfilepointer(h,0,nil,FILE_BEGIN); //and back to the start of the file ;)
          CloseHandle(h);//done
          beep;//So experiment with and without the SETSPARSE ioctl
     
    //open your file under winex and tada GB of zero hell at the end :p
    end;

    Once a file in sparse mode, you should access it only via the MS API and
    systematically sending the IOCTL before each operation.

    Until your job is finished
    Then it can be accessed normally with any soft... That can handle a 
    500 GB file. 

    To read for more infos:
    File system forensic analysis ISBN 0-321-26817-2
    Real forensic analysis ISBN 0-321-2406
     
  2. darkvadr
    Offline

    darkvadr DF Member

    RE:

    Thank you for my first "thank's" \o/

    For the "hell" string it's ansii code, my code is raw, but you can easely adapt it for unicode strings, binary streams, whatever.
    You just have to calculate the number of bytes for the file pointer.

    And the representation of the file is an over simplification, it's not physically accurate.
    But... It's enough to be usable.

    It's untested on 64 bits OS nor Windows 8 + or virtualized environement (I think it should not be a problem)
    If you make a successful experiment, I would be very happy to know it.
     

Share This Page